what turboquant actually unlocks on a 24GB mac

I have a Mac Mini M4 Pro with 24GB. I wanted to run the best local coding model I could, with enough context to be useful. TurboQuant — Google’s KV cache compression — is supposed to make impossible models possible.

I spent two days testing. Some things worked. Most of my assumptions were wrong.

The problem TurboQuant solves

Every token in your conversation eats memory. How much depends on the model:

Gemma 4 models have 3-7x more KV per token than Qwen. Without compression, you burn through your memory budget before you hit useful context lengths.

TurboQuant (ICLR 2026) compresses KV cache using randomized Hadamard rotation + Lloyd-Max quantization. 3-bit values, ~2.9x compression, near-zero quality loss. It compresses the cache that grows with context — not model weights.

What I tested

Four models. Two backends. Nine Svelte 5 tests. Here’s what happened.

MLX can’t do it

Apple’s MLX is the native inference framework for Apple Silicon. It’s 9% faster than llama.cpp on raw FP16 generation.

But Gemma 4 uses sliding window attention with RotatingKVCache. MLX’s QuantizedKVCache doesn’t support that. Error: RotatingKVCache Quantization NYI.

Community TurboQuant implementations for MLX exist (PR #1067, turbomlx) but none are merged. The standalone ones add 22-41% overhead.

TheTom’s llama.cpp fork works today. 2% overhead on the TurboQuant paper’s benchmark. I used this for everything.

Nemotron-Cascade 2 doesn’t fit at any quantization

30B total, 3B active, hybrid Mamba-Attention. Looked perfect on paper.

Even IQ2_XXS — the most aggressive 2-bit quantization possible — is 16.8 GB. The Mamba SSM state tensors are f32 and can’t be compressed. With 24GB minus 4GB for macOS, the weights alone don’t fit. TurboQuant is irrelevant here because KV cache is tiny (6 attention layers). The bottleneck is model weights + Mamba state.

Hybrid Mamba-Attention models solve long context differently — but the tradeoff is an incompressible weight floor that kills them on small devices.

Qwen 3.5 27B is too slow

Best code quality in real-world agentic tests. 92.2% on SvelteBench. Gets complex tasks right on the first try.

~11 tok/s on M4 Pro. Dense 27B model, bandwidth-limited. Add Qwen’s thinking mode (500-2000 tokens of internal reasoning before each response) and every interaction takes minutes. I abandoned the SvelteBench run after repeated timeouts.

TurboQuant would extend its context from 70K to 237K. Doesn’t matter if you can’t stand waiting.

Gemma 4 26B-A4B: the one that works

MoE with 3.8B active parameters. ~68 tok/s on M4 Pro. Fast enough to iterate.

Without TurboQuant: 91K max context (FP16 KV). With TurboQuant: 250K max context (q8_0 keys + turbo3 values).

That’s the difference between “fits one file” and “fits a codebase.”

SvelteBench: local Q4_K_M vs full-precision API

I ran SvelteBench locally against Gemma 4 26B-A4B Q4_K_M with TurboQuant. 5 samples per test, 9 tests.

Core Svelte 5 — $state, $derived, {#each}, onclick — works perfectly. The quality drop is from Q4 weight quantization, not TurboQuant’s KV compression. Q5_K_S (17.5 GB) would likely close the gap, but it’s 577 MiB over the GPU budget.

The model produces correct rune syntax, modern event handlers, clean component structure. For single-component tasks planned by a stronger model, it’s reliable on the fundamentals.

What TurboQuant actually unlocks

It doesn’t make bad models good. Weight quantization degrades quality. TurboQuant can’t fix that — it compresses KV cache, not weights.

It doesn’t help every architecture. Qwen’s KV cache is already small. Nemotron’s bottleneck is Mamba state, not KV. TurboQuant is most impactful on models with large KV caches — Gemma 4’s hybrid attention with 256-dim heads across 30 layers.

What it does: turns “can see one file” into “can see a codebase.” Gemma 4 26B-A4B goes from 91K to 250K context. That’s the whole project in context, not just the file you’re editing.

The practical setup: Opus plans the architecture. The local model implements individual components at 68 tok/s. TurboQuant means the local model can hold enough context to understand where each component fits.

Setup

git clone https://github.com/TheTom/llama-cpp-turboquant
cd llama-cpp-turboquant
git fetch origin feature/turboquant-kv-cache
git checkout -b turboquant FETCH_HEAD
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(sysctl -n hw.ncpu)

MODEL=/path/to/gemma-4-26B-A4B-it-Q4_K_M.gguf

./build/bin/llama-server \
  -m $MODEL \
  --cache-type-k q8_0 \
  --cache-type-v turbo3 \
  --flash-attn on \
  -c 131072 \
  -ngl 99 \
  --parallel 1 \
  --host 0.0.0.0 --port 8080

OpenAI-compatible API at http://localhost:8080/v1/chat/completions. Point any coding tool at it.

The numbers

Machine: Mac Mini M4 Pro, 24GB. Model: Gemma 4 26B-A4B Q4_K_M (15.7 GB).

TurboQuant paper · llama.cpp fork · SvelteBench · Gemma 4 · MLX vs llama.cpp benchmarks