Every few weeks someone posts a "local AI agent on a Mac Mini" thread. The screenshot shows Ollama running. The missing footnote is how long a real coding loop takes once you account for context size, hidden reasoning tokens, and multi-turn tool calls.
The AlphaSignal signal on Qwen3 8B is still worth attention: a full coding agent at ~20 tok/s on a 10-year-old GPU is plausible when you quantize correctly and tune the runtime. You do not need a fresh RTX 5090 to experiment.
What Qwen3 8B actually is
Qwen3's dense lineup includes 0.6B through 32B parameters. There is no 7B slot; 8B is the practical small model. For local work, qwen3:8b at Q4_K_M is the usual starting point:
| Setting | Typical value |
|---|---|
| VRAM (Q4_K_M) | ~5.2 GB |
| Min GPU | 12 GB class (e.g. GTX 1080 Ti / RTX 3060) |
| Decode speed | ~20–50 tok/s on older 12 GB cards; higher on bandwidth-rich GPUs |
| Context | Depends on runtime; cap it on purpose |
On a 12 GB RTX 3060, community benchmarks often land near 50 tok/s decode for Q4_K_M variants. That is interactive enough for experimentation, not necessarily for all-day product work.
The agent loop is not the same as chat
Chat benchmarks lie to you. Agentic coding sends a fat system prompt plus tool definitions every turn. Prefill latency dominates when context balloons.
A recent public breakdown on high-end Apple silicon showed:
- Ollama auto-allocated a 262K context, wasting GB of KV cache
- Qwen3's default thinking tokens burned most of the output budget unless disabled
- Even at 67 tok/s generation, multi-step tasks stretched past three minutes
Lessons that transfer to budget GPUs:
- Cap context to what your agent actually needs (often 16K–32K, not "whatever fits").
- Disable hidden reasoning for tool-use loops unless you measure a quality win.
- Measure time-to-first-token, not just tok/s.
- Prefer smaller tool schemas and fewer round trips.
Local agents are viable for private repos, air-gapped environments, and cost caps. They are not a free clone of cloud Claude latency.
A sane local setup
ollama pull qwen3:8b # In your agent config, set context well below auto-max # Disable thinking/reasoning modes for tool loops unless you need them
| Tool | Role |
|---|---|
| Ollama | Fastest path to pull and serve Qwen3 8B |
| llama.cpp / vLLM | More control on Linux servers |
| OpenClaw / Claude Code / custom harness | Agent orchestration and tool calls |
Quantization cheat sheet for coding:
| VRAM | Quant | Model sizes |
|---|---|---|
| 8 GB | Q4_K_M | 7B–8B class |
| 12 GB | Q4_K_M / Q5_K_M | 8B–14B |
| 16 GB+ | Q5_K_M / Q8_0 | 14B–32B |
When local Qwen3 8B is the right call
Good fits:
- Prototyping agent tools on laptop hardware without API bills
- Codebases that cannot leave your network
- Batch refactors where latency is fine but privacy is not negotiable
- Teaching and workshops where students run the same small model
Still reach for cloud models when:
- You need long-horizon reasoning across huge repos
- Time-to-merge matters more than token cost
- Eval quality on your stack clearly favors larger models
I treat local 8B agents as a lab environment, not a drop-in replacement for production coding agents like Devin or cloud Claude Code on client work.
Useful references:
Bottom line
Qwen3 8B at sensible quantization runs on hardware many teams already have in a closet. The hard part is not pulling the model. It is configuring the agent loop so you are not paying 16 seconds of prefill on every tool call.
If you want help designing a local-vs-cloud split for coding agents (privacy, cost, eval gates), book a free discovery call.