Wednesday 7/2/25
Big Tech companies compile insights about you derived from your interactions with their cloud-based, large language models into encyclopedic profiles, which they sell to the highest bidders. An obvious and powerful way to substantially reduce these intrusions is to redirect a substantial portion of your queries away from Big Tech's clouds.
Why not direct your queries, instead, to free, powerful, open source, small language models running on a server in your home or office? The privacy of your most confidential concerns about your health, taxes, finances, personal relationships, and other sensitive matters will be assured by running them off-network in your local home or office data center.
This note provides a step-by-step description of how the editor of this blog constructed his own personal AI data center via a process that could easily be implemented by the vast majority of his computer savvy readers ... with no code, with no code, with no code ... 😎😎😎
Context
The reader should note that the following discussion is confined to a “no code” step-by-step creation of a home or office generative AI data center that only includes a single Mac mini containing one M4 chip that runs powerful, open source, small language models.
However, Apple’s introduction of the M4 mini in late 2023 triggered the publication of a growing body of reports from tech enthusiasts that have proclaimed the potential for clusters of these inexpensive minis — stacked into small racks — to handle far more powerful large language models. Indeed, the most successful effort thus far was recently claimed by a team that created a cluster of 8 minis that could run the famous Chinese open source V3 large models at “blazing” speeds.
-- "DeepSeek-V3 on M4 Mac: Blazing Fast Inference on Apple Silicon", DigiAlps
Of course, this impressive achievement required a considerable amount of code … 😔
-- "DeepSeek-V3 on M4 Mac: Blazing Fast Inference on Apple Silicon", DigiAlps
Of course, this impressive achievement required a considerable amount of code … 😔
… Design Features …
1.. “Satisfice” = not optimize = “good enough” to substantially reduce intrusion, not eliminate it.
2.. No code, only point and click selections from menus.
3.. Easy to configure, use, and maintain.
4.. Greater privacy for (rare) searches by using DuckDuckGo as search engine, not Google, because DuckDuckGo does not create user profiles.
5. Safari as default browser, not Chrome, because it provides a wide range of privacy protections.
6.. Open source, small language models that are powerful, reliable, and impose no subscription fees.
A. Mini + Mac | B. Notes-Safari-DDGo | C. LM Studio | D. Library of Models
… Mac & Mini Hardware…
A. Mini + Mac
- Buy a Mac Mini, the "server", with M4 chip and at least 16 GB ram, 256 GB solid state storage (but extensible via external SSD).
- Deploy a Mac desktop or laptop as the "workstation"
- Buy/obtain inexpensive a monitor, mouse, keyboard for the Mini. These devices will rarely be used, mainly when configuring hardware and software; otherwise the Mini will be “headless”. All queries to the models on the Mini will be done remotely from the Mac.
- Buy a Thunderbolt cable. Warning: Be sure you buy a Thunderbolt cable, not l a USB-C cable)l. While they look identical and will physically fit the same ports, only a proper Thunderbolt cable will enable the Thunderbolt bridge connection between the two Macs.
- Connect monitor, mouse, and keyboard to the Mini.
- Designate a Mac (desktop or laptop) as the client "workstation" for the Mini.
- Connect a Mac Thunderbolt port to a Mini Thunderbolt port with the Thunderbolt cable.
… Mac-Mini Connections…
8. Log onto the Mini to set up Thunderbolt bridge with the Mac
- Apple > System Settings > Network
- Click … "Three Dots ..." on right side of screen below the "Ethernet" label
- Click "Add Service"
- Select "Thunderbolt Bridge" > "Create"
- Click "Thunderbolt Bridge" > “Details” > “TCP/IP”
- Wait until address and mask appear in the IP address and Mask boxes.
- Click "OK"
- System Settings > General > “Sharing”
- Turn on “Screen Sharing”
- Turn on “File Sharing”
10. On the Mini, in order to never lose connections by “sleeping”
- System Settings > Energy
- Turn on “Prevent automatic sleeping when display is off”
- Turn on “Wake for network access”
- Apple > System Settings > Network
- Click three dots “..." on right side of screen below "Ethernet" label.
- Click "Add Service"
- Select "Thunderbolt Bridge" > "Create"
- Click "Thunderbolt Bridge" > “Details” > “TCP/IP”.
- Wait until address and mask appear in IP address and Mask boxes.
- Click "OK".
Use Finder to check your connection. - Finder > Go > Network ... the Mac name should appear.
- Click the name to display list of folders at lowest level on the Mini.
*** 12. On the Mac start a remote session on your "headless” Mini.
1. Configure Notes app with local pages where you will securely store prompts and responses locally on the Mini, a/k/a your "vault" inaccessible to iCloud … and Apple.
- Open Finder > Go > Network
- Double-click your Mac mini’s name when it appears.
- Click the “Share Screen” button in far right upper corner.
- Click "Continue" button.
- A full copy of what's currently on the Mini screen will be displayed in a big window on the Mac.
… Mini Software …
B. Notes, Safari, & DuckDuckGo
1. Configure Notes app with local pages where you will securely store prompts and responses locally on the Mini, a/k/a your "vault" inaccessible to iCloud … and Apple.
- On the Mac start a remote session on your Mini.
(See step *** A. 12, above) - Notes > Settings > Enable On my Mac account
- Set Locked Notes to "On My Mac"
- Set a strong password for the locked notes
- Suggestion: Always include various #hashtags in your notes to make each one easier to find via Notes "Search". There are no folders in the locked notes storage.
2. Make Safari the default browser on the Mini
- Continue the remote session on the Mini
or start a new session (See step *** A. 12, above) - Safari > Settings > "Safari is not your default browser"
- Select "Safari"
3. Set DuckDuckGo as search engine for Safari
- Continue the remote session on the Mini
or start a new session (See step *** A. 12, above) - Go to the App Store
- Download DuckDuckGo to the Mini
- Safari > Settings > Search ... SearchEngine ... Select "DuckDuckGo"
C. LM Studio app includes LM Server
that hosts the models and a chatbot that provides user access to the models .
that hosts the models and a chatbot that provides user access to the models .
- On the Mac start a remote session on the Mini.
(See step *** A. 12, above) - Open Safari
Go to: https://lmstudio.ai
- Click “Download for macOS” button on the homepage.
-- Wait for the .dmg file to download
-- The file is named something like LM-Studio.dmg.
-- It will appear in your Downloads folder. - Double-click file name in the Downloads folder
- Drag LM Studio icon into the Applications folder
- Open Launchpad > Double-click LM Studio icon to launch it.
-- If macOS asks if you’re sure you want to open it, click Open.
-- Allow Full Disk Access (Optional but recommended) - *** Search and download models
-- Click on the Search tab (or “Discover” tab) in the left sidebar. This opens the model catalog
-- In the search bar, type the full name of the mode
-- Click the Download button next to it
-- Wait for download … The download will start and show progress … Once complete, the model will appear in your “My Models” tab
D. Library of Initial Models recommended by Claude
Here are my [Claude's] recommendations with the exact GGUF specifications for your Mac Mini M4 with 16GB RAM: ... [GGUF stands for “GPT-Generated Unified Format” - it’s a file format specifically designed for storing and running large language models efficiently.]
Here are my [Claude's] recommendations with the exact GGUF specifications for your Mac Mini M4 with 16GB RAM: ... [GGUF stands for “GPT-Generated Unified Format” - it’s a file format specifically designed for storing and running large language models efficiently.]
*** Best General Foundation Model:
-- mistral-7b-instruct-v0.3.Q5_K_M.gguf (~5GB)
-- Your reliable daily driver with excellent quality/performance balance.
An Alternative General Foundation Model:
-- Qwen2.5-7B-Instruct.Q5_K_M.gguf
--
*** Specialized Models Worth Exploring:
For Medical/Health Questions:
• meditron-7b.Q5_K_M.gguf (~5GB) - Built on Llama 2, trained on medical literature
• biomistral-7b.Q5_K_M.gguf (~5GB) - Mistral fine-tuned for biomedical tasks
For Coding Help/Tech Explanations:
• codellama-7b-instruct.Q5_K_M.gguf (~5GB) - Excellent for explaining code and technical concepts
• starcoder-7b.Q5_K_M.gguf (~5GB) - Good alternative for multiple programming languages
For Creative Writing/Content:
• zephyr-7b-beta.Q5_K_M.gguf (~5GB) - Llama 2-based, excellent for conversational and creative tasks
• openhermes-2.5-mistral-7b.Q5_K_M.gguf (~5GB) - Great for storytelling and creative writing
For Research/Analysis:
• nous-hermes-2-mixtral-8x7b.Q4_K_M.gguf (~26GB) - Complex reasoning, uses most of your RAM
• wizardlm-7b.Q5_K_M.gguf (~5GB) - Good for following complex instructions
For Legal/Professional Writing:
• saul-7b.Q5_K_M.gguf (~5GB) - Trained on legal documents (not legal advice)
*** Setup Strategy:
1. Start with mistral-7b-instruct-v0.3.Q5_K_M.gguf
2. Add 2-3 specialized models based on your interests
Each model name is exactly what you’ll search for when downloading!
Note: Follow the *** Search and download procedure described at the end of Section C for each model in a remote session on the Mini from the Mac, as described in step *** A. 12
__________________________________
Links to related notes on this blog:
- "The failure of Apple's leadership to comprehend the impact of the generative AI revolution on Internet search" 5/16/25
- "Apple's dilemma: Pleasing its iPhone users AND its long term investors", 6/12/25
- "Reducing your carbon footprint with a personal AI data center in your home based on Apple technology", 6/28/25
No comments:
Post a Comment
Your comments will be greatly appreciated ... Or just click the "Like" button above the comments section if you enjoyed this blog note.