Another hit from the last two weeks on Hacker News (900+ points) — the project TurboFieldfare, which runs the 26-billion-parameter Gemma 4 26B on a MacBook Air with 8 GB RAM, while consuming only about 2 GB of RAM. Sounds like magic — in reality it’s beautiful engineering, and it’s useful to everyone who runs local LLMs and hits memory limits.
How does 26B fit into 2 GB
The secret is that Gemma 4 26B is an MoE model (mixture-of-experts): out of 26 billion parameters, only about ~3.88 billion are actually used per token. The model has 32 “experts,” but for a given token you need only a few. TurboFieldfare exploits this to the max:
- in memory you keep only the general core (~1.35 GB) and FP16 KV cache;
- experts stream from SSD as needed — 24 of 32 experts reside on disk and are loaded only when the model router selects them;
- weights are compressed with 4-bit quantization MLX (group 64), routers — 8 bits;
- there is an LFU cache with 16 slots for experts: frequently used ones stay in memory, misses are read from disk with limited parallel
pread.
On each layer Metal computes attention and decides which experts are needed; the CPU checks the cache and reads the missing parts from the SSD, while Metal concurrently processes the main branch. Because of this, peak RAM usage stays around 2 GB, not tens of gigabytes as with loading the whole model.
Figures
- Storage: ~14.3 GB on disk (the
.gturbodirectory with compressed weights). The installer streams byte ranges directly from Hugging Face, without materializing the full checkpoint. - Decoding speed: on M2 (8 GB MacBook Air) — 5.1–6.3 tokens/s; on M5 Pro (24 GB) — 31–35 tokens/s.
- Requirements: Apple Silicon, macOS 26 with Metal 4, Xcode 26, Swift 6.2, ~15 GB for loading weights.
- Model: Gemma 4 26B-A4B instruction-tuned, only text; code license — Apache 2.0 (weights — under Google Gemma license).
Installation is straightforward:
git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac
First run will download ~15 GB of weights (range requests to Hugging Face) and verify hashes. There is also a GUI application with controls for temperature/top-K/top-P, and a CLI (--prompt, --max-context, --temperature 0 for determinism, --seed, --stop).
A sober assessment
Don’t be fooled by the headline “26B in 2 GB” — context matters.
- This is an experimental, niche project. The author himself says he did it “for character, not practicality”; at the time of writing the repo has ~85 stars. It’s a demonstration of technique, not a ready-for-prime-time enterprise-inference setup.
- Only macOS/Apple Silicon, and only a recent one — macOS 26, Metal 4, Swift 6.2. No Linux, Windows, or mobile. For a typical self-hosted Linux-based setup this isn’t directly applicable yet.
- “2 GB RAM” — the price of a trade-off. Experts are read from SSD per token, so much depends on storage speed and page-cache state; 5–6 tokens/s on M2 is “works,” but not fast.
- No tool-calling and multimodality, the model is fixed to a particular checkpoint.
The value here isn’t that everyone should rush to install TurboFieldfare, but the approach itself: streaming MoE experts from disk — a viable way to run large models on modest memory, and this idea is likely to be picked up by more universal runners (llama.cpp/MLX). It’s worth watching.
What to do
- You have a Mac with Apple Silicon and curiosity — build and try; this is the best way to feel MoE streaming in real life.
- If you’re on Linux/server — adopt the principle, not the project: watch when the streaming loading of experts lands in your favorite runner.
- Need an everyday local assistant here and now — more practical are dense models in the 7–14B range in llama.cpp/Ollama with reasonable speed, and keep this project as a reference for where memory optimization is headed.
Sources
On which hardware and in which runner are you running local models — and did you hit RAM ceilings? Streaming experts from SSD — is this the future of home inference or a workaround for a niche scenario?