A 27-billion-parameter model used to mean cloud bills, datacenter GPUs, or at best a beefy workstation. PrismML's Bonsai 27B changes the framing: the 1-bit variant lands at 3.9 GB, small enough to load inside a high-end phone's per-app memory budget.
That is not a demo quant for leaderboard screenshots. It is a multimodal model derived from Qwen3.6-27B with vision, tool use, coding, and multi-step agent workflows, released under Apache 2.0 on Hugging Face.
I ship local-agent prototypes for clients who cannot send every document to a hosted API. Bonsai 27B is the first release where I would seriously sketch a private on-device agent without laughing at the hardware sheet.
Why 3.9 GB is the number that matters
Storage size and runtime memory are different problems. A phone does not give your app all of its RAM. A 12 GB iPhone might expose roughly 6 GB to a single process. That slice has to hold weights, KV cache, activations, and your app shell.
| Variant | Deployed size | vs FP16 (~54 GB) | Performance retention (PrismML suite) |
|---|---|---|---|
| FP16 baseline | ~54 GB | 1.0x | 100% |
| Ternary g128 | ~5.9 GB | ~9x smaller | ~95% |
| 1-bit binary g128 | ~3.9 GB | ~14x smaller | ~90% |
PrismML's pitch is blunt: at ~4 GB, the 1-bit build is the first 27B-class stack that clears the phone gate with headroom for cache. Their July 2026 benchmarks cite 11 tokens/s on iPhone 17 Pro for the MLX build, which is slow for chat but workable for background agent steps that batch work.
What they did to the weights
Bonsai is not "just GGUF Q4." PrismML runs end-to-end low-bit coverage across embeddings, attention projections, MLPs, and the LM head. The 1-bit variant stores weights as binary minus-one and plus-one values with FP16 group-wise scaling (GGUF Q1_0_g128, about 1.125 effective bits per weight).
The vision tower ships as HQQ 4-bit with an optional ~0.63 GB mmproj pack loaded only when you pass images. KV cache uses near-lossless 4-bit quantization. Because the backbone is hybrid attention (roughly 75% linear / 25% full attention), only 16 of 64 layers grow a full attention cache. At the full 262K window that cache is about 4.3 GB in their spec sheet, which is why long context on phone is a tradeoff, not a free lunch.
Acceleration includes a DSpark speculative-decoding drafter layer. Draft-and-verify is exactly the kind of trick you want when base decode is in the tens of tokens per second, not hundreds.
Where I would actually use this
Good fits for applied teams:
- Field agents on locked-down devices (health, legal, finance) where outbound API calls need a business case
- Offline coding assistants on flights or factory floors with no reliable uplink
- Personal knowledge workflows where the corpus never leaves the device
- Prototype multimodal agents that read screenshots or camera frames without uploading them
Weak fits:
- Latency-sensitive voice loops (11 tok/s is not a conversation product)
- Tasks that need the freshest frontier model every week
- Teams with no appetite for MLX, llama.cpp forks, or custom kernel maintenance
The Apache 2.0 license matters as much as the size. You can ship commercial products, fine-tune, and redistribute without revenue caps. That puts Bonsai in a different bucket from research-only embedding models or gated API-only releases.
How to try it without betting a sprint
PrismML publishes multiple backends:
- MLX for Mac, iPhone, and iPad (
prism-ml/Bonsai-27B-mlx-1bit) - GGUF for llama.cpp with CUDA, Metal, or CPU (
prism-ml/Bonsai-27B-gguf) - A limited-time developer preview API if you want to benchmark before you flash weights to hardware
My practical eval order:
- Run the same 10-prompt agent harness you use for cloud models (tool calls, JSON repair, short coding edits)
- Measure tokens per second and peak RAM, not vibe quality on three cherry-picked prompts
- Test vision only if your workflow needs it (mmproj adds weight and load time)
- Compare against a 7B–8B cloud API baseline on cost and privacy, not just accuracy
If you already run Qwen3.8-27B on a 24 GB GPU, Bonsai is the complementary story: same capability class, different deployment target (pocket instead of workstation).
The bigger pattern: frontier class, edge footprint
AlphaSignal framed this digest alongside Anthropic funding Canadian research and modular RL stacks. The rhyme is real. Labs are opening the stack at both ends: bigger cloud training grants and binaries small enough to run where the data lives.
For operators, that split is useful. Not every workflow belongs on Claude or GPT-5. Some belong on device because of privacy, because of offline requirements, or because per-step API cost kills the unit economics of an agent loop.
Bonsai 27B does not replace hosted frontier models. It gives you a credible local tier where "27B-class" stopped being a joke on a phone.
If you are designing agent architecture and want help choosing where inference should live (device, VPC, or API), book a free discovery call. The hardware decision is cheaper before you bake it into every client deployment.