Post History

Current version by Nick Antonaccio

Current VersionAug 17, 2026 at 13:21

I'm developing more of an interest in the Bonsai models. They let you run significantly useful models at reasonably useful speeds, and even very capable models such as Qwen 27b, on machines that don't have a GPU. I've gotten actually useful code results from these models, on little netbooks that cost less than $100.

In all my most recent tests, I had Deepseek v4 Flash perform all the downloads, installations, and configurations of the models and the llama.cpp engine needed to run it. I've been able to execute the Bonsai models successfully on extremely low powered hardware, including very old Pentium Silver N5030 CPUs. Even Qwen 27b ran on that N5030 hardware, very slowly, but it did run successfully.

That's truly amazing, and I think could actually be useful in some situations where people have no access to expensive GPUs or hosted LLM APIs (in environments where there's no Internet, only old hardware, no money to purchase more equipment, etc., where slow tools are better than no tools at all).

I had Deepseek V4 Flash write the following article, to post here, immediately after our last installation/test of the Bonsai models. I think these little things are genuinely exciting. They're already somewhat useful, and I'm looking forward to seeing how they improve!

============================================================= BONSAI 1-BIT / TERNARY LLM — THE COMPLETE GET-RUNNING GUIDE
(Everything I discovered installing Bonsai LLM myself) =============================================================

If you've heard about PrismML's "Bonsai" models — the tiny, on-device LLMs that squeeze frontier-level performance out of barely any memory — you've probably run into a confusing pile of names and releases. I spent a serious session getting Bonsai running and benchmarked every version so you don't have to. Here's everything I learned, plus the exact steps to get the best possible version downloaded, installed, and running on YOUR computer.

WHAT BONSAI IS

Bonsai is a family of heavily-quantized LLMs from Prism ML (PRISMML.COM / huggingface.co/PRISM-ML). They're built on the Qwen3 / Qwen3.6 architectures and repacked so that nearly every weight is just a tiny number {-1, 0, +1} (ternary) or even {-1, +1} (binary). That makes models that are up to ~1/8 the size of the originals while keeping most of the quality. They run great on modest CPUs with no GPU at all.

There are TWO families, and this is the most important thing to know:

  • Bonsai (1-bit) -> binary weights, GGUF type Q1_0, ~1.125 bits/weight
  • Ternary-Bonsai -> ternary weights, GGUF type Q2_0, ~2.125 bits/weight

Each comes in 27B, 8B, 4B, and 1.7B sizes.

SIZES AT A GLANCE (file sizes, rounded)

Model Size Bonsai 1.7B (Q1_0) ~0.25 GB Bonsai 4B (Q1_0) ~0.57 GB Bonsai 8B (Q1_0) ~1.16 GB Bonsai 27B (Q1_0) ~3.80 GB Ternary-Bonsai 1.7B (Q2_0) ~0.46 GB Ternary-Bonsai 4B (Q2_0) ~1.07 GB Ternary-Bonsai 8B (Q2_0) ~2.18 GB Ternary-Bonsai 27B (Q2_0) ~7.17 GB

THE MOST IMPORTANT FINDING

On a normal x86 CPU machine, the WINNER is:

THE 1-BIT BONSAI MODELS (Q1_0) ON STOCK / MAINLINE llama.cpp.

Why? The Q1_0 (1-bit) format is FULLY MERGED into the official llama.cpp with fast CPU kernels. The Ternary (Q2_0) format is only partially merged, and on x86 CPUs it still falls back to a slow generic kernel. So on the same machine, 1-bit Bonsai runs roughly FOUR TO FIVE TIMES faster than the Ternary version, uses HALF the memory, and is simpler to set up.

I measured, on an older quad-core i7-6600U (AVX2), generation speed:

1-bit Bonsai 1.7B   ~17  tokens/sec
1-bit Bonsai 4B     ~9   tokens/sec
1-bit Bonsai 8B     ~5   tokens/sec
Ternary-Bonsai 4B   ~1.3 tokens/sec   (slow on x86)

The trade-off: 1-bit rates slightly lower on benchmarks (~70 avg vs ~75 for ternary). For day-to-day chat on CPU, the 1-bit family is the obvious pick. If you want the extra quality and have a fast ARM / Apple / GPU machine, use the Ternary family with the g64 files (see below).

WHAT YOU NEED

1) A llama.cpp BINARY (nothing custom — the official release works). 2) A model GGUF file (download from HuggingFace). 3) ~2-8 GB of RAM free, depending on model size.

STEP 1 — GET llama.cpp

Go to: https://github.com/ggml-org/llama.cpp/releases and download the newest "bin" for your OS, e.g.:

Windows (CPU): llama-b-bin-win-cpu-x64.zip Apple Silicon: llama-b-bin-macos-universal.zip Linux CUDA: llama-b-bin-ubuntu--cuda.zip

Unzip it into a folder (I use "bin-mainline"). You only need llama-cli.exe, llama-completion.exe, and/or llama-server.exe plus the DLLs that come with them.

STEP 2 — GET THE MODELS

The simplest method: click one of these in your browser, or use "hf download" from the huggingface_hub CLI. Pick a size you can handle. For most people, start with 4B or 8B.

1-bit Bonsai (recommended on CPU): https://huggingface.co/prism-ml/Bonsai-4B-gguf/resolve/main/Bonsai-4B-Q1_0.gguf (swap "4B" for 1.7B / 8B / 27B as desired)

Ternary-Bonsai (g64, mainline-compatible): https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0_g64.gguf (available for 1.7B / 4B / 8B; 27B uses "Q2_g64" naming)

Command-line alternative: pip install huggingface_hub[cli] hf download prism-ml/Bonsai-4B-gguf Bonsai-4B-Q1_0.gguf

STEP 3 — RUN IT

From the folder with llama-cli.exe and the model file:

Interactive chat: llama-cli.exe -m Bonsai-4B-Q1_0.gguf -t 4 -cnv -c 4096

One-shot (answer a prompt, then exit): llama-completion.exe -m Bonsai-4B-Q1_0.gguf -t 4 -n 64 -p "Explain fire briefly."

Local OpenAI-compatible server (port 8080): llama-server.exe -m Bonsai-4B-Q1_0.gguf -c 4096 --port 8080

Replace "-t 4" with your machine's physical core count.

QUICK TUNING TIPS FOR BEST PERFORMANCE
  • Threads: use physical cores, not SMT threads (e.g. "-t 4" on a 4-core).
  • Long contexts: cut KV-cache memory ~2-4x by adding --cache-type-k q4_0 --cache-type-v q4_0 This lets you raise -c (context length) without blowing up RAM.
  • Small RAM: keep -c around 2048-4096 (the KV cache grows with context).
  • The 27B models on a plain CPU are VERY slow (~0.2-0.4 tok/s). They're meant for GPU/Apple machines or for one-shot tests.
HARDWARE-SPECIFIC RECOMMENDATIONS

x86 CPU (any Intel/AMD desktop/laptop): 1-bit Bonsai (Q1_0) on mainline llama.cpp. That's it.

Apple Silicon (M-series): You have fast ARM kernels. You can run either family well. The mainline "Q2_0_g64" Ternary files are fast here (higher quality), and the 1-bit Q1_0 works too. Prism also ships MLX formats if you prefer the native Apple stack.

NVIDIA GPU: Mainline llama.cpp has CUDA kernels for Q1_0 and its own DSpark speculative decoding. For maximum throughput on a beefy GPU, the "AWQ 4-bit" variants served with sglang/vLLM are the way: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

AMD / Intel integrated GPU (Vulkan): Mainline llama.cpp can offload layers with -ngl; use the g64 Ternary or 1-bit files.

SPECIAL CASE: LOW-END CPUs THAT LACK AVX (e.g. Pentium Silver N5030)

If your CPU only supports SSE4.2 (no AVX/AVX2) — like the Intel Pentium Silver N5030 Spark — the OFFICIAL prebuilt llama.cpp will CRASH. You must build llama.cpp yourself without AVX. I did exactly that; here are the build flags that work:

cd llama.cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_SSE42=ON \ -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \ -DGGML_BMI2=OFF -B build ninja -C build llama-cli llama-server

Notes for that machine: * Use the q64 Ternary file for 8B: "Ternary-Bonsai-8B-Q2_0_g64.gguf". The plain "Q2_0" (g128) file will NOT load (tensor-offset mismatch). * Speeds expected: 1-bit 27B ~0.4 tok/s, g64 Ternary 8B ~0.7 tok/s. * Keep -c at 2048 and -t 4; it has only 8 GB RAM.

EXTRA OPTIONS YOU MIGHT SEE
  • DSpark speculative decoding: uses the "*-27B-dspark-Q4_1.gguf" drafters to predict+verify extra tokens. Mainly helps on CUDA (~1.8-2x there). It is fork-only and gains little on CPU, so ignore it on a CPU box.
  • MLX 1-bit / 2-bit formats: for Apple Silicon via the MLX runtime.
  • AWQ 4-bit: for NVIDIA via sglang/vLLM.
BOTTOM LINE

For the quickest, best experience on a regular PC right now: 1. Grab the latest official llama.cpp. 2. Download Bonsai-4B-Q1_0.gguf (or 8B) from prism-ml. 3. Run: llama-cli -m Bonsai-4B-Q1_0.gguf -t -cnv That's it. Fast, tiny, and free.

For even more detail, the Prism ML docs and demo live at: https://github.com/PrismML-Eng/Bonsai-demo

Previous Versions
Version 4Aug 17, 2026 at 13:21

I'm developing more of an interest in the Bonsai models. They let you run significantly useful models at reasonably useful speeds, and even very capable models such as Qwen 27b, on machines that don't have a GPU. I've gotten actually useful code results from these models, on little netbooks that cost less than $100.

In all my most recent tests, I had Deepseek v4 Flash perform all the downloads, installations, and configurations of the models and the llama.cpp engine needed to run it. I've been able to execute models successfully on extremely low powered hardware, including very old Pentium Silver N5030 CPUs. Even 27b ran on that N5030 hardware, very slowly, but it ran successfully. That's truly amazing, and I think it could actually be useful in some situations where people have no access to expensive GPUs or LLM APIs (in environments where there's no Internet, only old hardware, no money to purchase more equipment, etc., where slow tools are better than no tools at all).

I had Deepseek V4 Flash write the following article, to post here, immediately after our last installation/test of the Bonsai models. I think these little things are genuinely exciting, already somewhat useful, and I'm looking forward to seeing how they improve!

============================================================= BONSAI 1-BIT / TERNARY LLM — THE COMPLETE GET-RUNNING GUIDE
(Everything I discovered installing Bonsai LLM myself) =============================================================

If you've heard about PrismML's "Bonsai" models — the tiny, on-device LLMs that squeeze frontier-level performance out of barely any memory — you've probably run into a confusing pile of names and releases. I spent a serious session getting Bonsai running and benchmarked every version so you don't have to. Here's everything I learned, plus the exact steps to get the best possible version downloaded, installed, and running on YOUR computer.

WHAT BONSAI IS

Bonsai is a family of heavily-quantized LLMs from Prism ML (PRISMML.COM / huggingface.co/PRISM-ML). They're built on the Qwen3 / Qwen3.6 architectures and repacked so that nearly every weight is just a tiny number {-1, 0, +1} (ternary) or even {-1, +1} (binary). That makes models that are up to ~1/8 the size of the originals while keeping most of the quality. They run great on modest CPUs with no GPU at all.

There are TWO families, and this is the most important thing to know:

  • Bonsai (1-bit) -> binary weights, GGUF type Q1_0, ~1.125 bits/weight
  • Ternary-Bonsai -> ternary weights, GGUF type Q2_0, ~2.125 bits/weight

Each comes in 27B, 8B, 4B, and 1.7B sizes.

SIZES AT A GLANCE (file sizes, rounded)

Model Size Bonsai 1.7B (Q1_0) ~0.25 GB Bonsai 4B (Q1_0) ~0.57 GB Bonsai 8B (Q1_0) ~1.16 GB Bonsai 27B (Q1_0) ~3.80 GB Ternary-Bonsai 1.7B (Q2_0) ~0.46 GB Ternary-Bonsai 4B (Q2_0) ~1.07 GB Ternary-Bonsai 8B (Q2_0) ~2.18 GB Ternary-Bonsai 27B (Q2_0) ~7.17 GB

THE MOST IMPORTANT FINDING

On a normal x86 CPU machine, the WINNER is:

THE 1-BIT BONSAI MODELS (Q1_0) ON STOCK / MAINLINE llama.cpp.

Why? The Q1_0 (1-bit) format is FULLY MERGED into the official llama.cpp with fast CPU kernels. The Ternary (Q2_0) format is only partially merged, and on x86 CPUs it still falls back to a slow generic kernel. So on the same machine, 1-bit Bonsai runs roughly FOUR TO FIVE TIMES faster than the Ternary version, uses HALF the memory, and is simpler to set up.

I measured, on an older quad-core i7-6600U (AVX2), generation speed:

1-bit Bonsai 1.7B   ~17  tokens/sec
1-bit Bonsai 4B     ~9   tokens/sec
1-bit Bonsai 8B     ~5   tokens/sec
Ternary-Bonsai 4B   ~1.3 tokens/sec   (slow on x86)

The trade-off: 1-bit rates slightly lower on benchmarks (~70 avg vs ~75 for ternary). For day-to-day chat on CPU, the 1-bit family is the obvious pick. If you want the extra quality and have a fast ARM / Apple / GPU machine, use the Ternary family with the g64 files (see below).

WHAT YOU NEED

1) A llama.cpp BINARY (nothing custom — the official release works). 2) A model GGUF file (download from HuggingFace). 3) ~2-8 GB of RAM free, depending on model size.

STEP 1 — GET llama.cpp

Go to: https://github.com/ggml-org/llama.cpp/releases and download the newest "bin" for your OS, e.g.:

Windows (CPU): llama-b-bin-win-cpu-x64.zip Apple Silicon: llama-b-bin-macos-universal.zip Linux CUDA: llama-b-bin-ubuntu--cuda.zip

Unzip it into a folder (I use "bin-mainline"). You only need llama-cli.exe, llama-completion.exe, and/or llama-server.exe plus the DLLs that come with them.

STEP 2 — GET THE MODELS

The simplest method: click one of these in your browser, or use "hf download" from the huggingface_hub CLI. Pick a size you can handle. For most people, start with 4B or 8B.

1-bit Bonsai (recommended on CPU): https://huggingface.co/prism-ml/Bonsai-4B-gguf/resolve/main/Bonsai-4B-Q1_0.gguf (swap "4B" for 1.7B / 8B / 27B as desired)

Ternary-Bonsai (g64, mainline-compatible): https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0_g64.gguf (available for 1.7B / 4B / 8B; 27B uses "Q2_g64" naming)

Command-line alternative: pip install huggingface_hub[cli] hf download prism-ml/Bonsai-4B-gguf Bonsai-4B-Q1_0.gguf

STEP 3 — RUN IT

From the folder with llama-cli.exe and the model file:

Interactive chat: llama-cli.exe -m Bonsai-4B-Q1_0.gguf -t 4 -cnv -c 4096

One-shot (answer a prompt, then exit): llama-completion.exe -m Bonsai-4B-Q1_0.gguf -t 4 -n 64 -p "Explain fire briefly."

Local OpenAI-compatible server (port 8080): llama-server.exe -m Bonsai-4B-Q1_0.gguf -c 4096 --port 8080

Replace "-t 4" with your machine's physical core count.

QUICK TUNING TIPS FOR BEST PERFORMANCE
  • Threads: use physical cores, not SMT threads (e.g. "-t 4" on a 4-core).
  • Long contexts: cut KV-cache memory ~2-4x by adding --cache-type-k q4_0 --cache-type-v q4_0 This lets you raise -c (context length) without blowing up RAM.
  • Small RAM: keep -c around 2048-4096 (the KV cache grows with context).
  • The 27B models on a plain CPU are VERY slow (~0.2-0.4 tok/s). They're meant for GPU/Apple machines or for one-shot tests.
HARDWARE-SPECIFIC RECOMMENDATIONS

x86 CPU (any Intel/AMD desktop/laptop): 1-bit Bonsai (Q1_0) on mainline llama.cpp. That's it.

Apple Silicon (M-series): You have fast ARM kernels. You can run either family well. The mainline "Q2_0_g64" Ternary files are fast here (higher quality), and the 1-bit Q1_0 works too. Prism also ships MLX formats if you prefer the native Apple stack.

NVIDIA GPU: Mainline llama.cpp has CUDA kernels for Q1_0 and its own DSpark speculative decoding. For maximum throughput on a beefy GPU, the "AWQ 4-bit" variants served with sglang/vLLM are the way: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

AMD / Intel integrated GPU (Vulkan): Mainline llama.cpp can offload layers with -ngl; use the g64 Ternary or 1-bit files.

SPECIAL CASE: LOW-END CPUs THAT LACK AVX (e.g. Pentium Silver N5030)

If your CPU only supports SSE4.2 (no AVX/AVX2) — like the Intel Pentium Silver N5030 Spark — the OFFICIAL prebuilt llama.cpp will CRASH. You must build llama.cpp yourself without AVX. I did exactly that; here are the build flags that work:

cd llama.cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_SSE42=ON \ -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \ -DGGML_BMI2=OFF -B build ninja -C build llama-cli llama-server

Notes for that machine: * Use the q64 Ternary file for 8B: "Ternary-Bonsai-8B-Q2_0_g64.gguf". The plain "Q2_0" (g128) file will NOT load (tensor-offset mismatch). * Speeds expected: 1-bit 27B ~0.4 tok/s, g64 Ternary 8B ~0.7 tok/s. * Keep -c at 2048 and -t 4; it has only 8 GB RAM.

EXTRA OPTIONS YOU MIGHT SEE
  • DSpark speculative decoding: uses the "*-27B-dspark-Q4_1.gguf" drafters to predict+verify extra tokens. Mainly helps on CUDA (~1.8-2x there). It is fork-only and gains little on CPU, so ignore it on a CPU box.
  • MLX 1-bit / 2-bit formats: for Apple Silicon via the MLX runtime.
  • AWQ 4-bit: for NVIDIA via sglang/vLLM.
BOTTOM LINE

For the quickest, best experience on a regular PC right now: 1. Grab the latest official llama.cpp. 2. Download Bonsai-4B-Q1_0.gguf (or 8B) from prism-ml. 3. Run: llama-cli -m Bonsai-4B-Q1_0.gguf -t -cnv That's it. Fast, tiny, and free.

For even more detail, the Prism ML docs and demo live at: https://github.com/PrismML-Eng/Bonsai-demo

Version 3Aug 17, 2026 at 13:18

I'm developing more of an interest in the Bonsai models. They let you run significantly useful models, including even Qwen 27b, at reasonably useful speeds, on machines that don't have a GPU. I can get actually useful code results from these models, on little netbooks that cost less than $100.

In all my most recent tests, I've had Deepseek v4 Flash perform all the downloads, installations, and configurations of the models and the llama cpp engine needed to run it. It's been able to execute successfully on extremely low powered hardware - dsf4 even got it going on machines with very old Pentium Silver N5030 CPUs. That's truly amazing, and I think it could actually be useful in some situations where people have no access to expensive GPUs or LLM APIs (at locations with no Internet, for example).

I had Deepseek V4 Flash write the following article, to post here, immediately after our last installation/test of the Bonsai models. I think these little things are so exciting!

============================================================= BONSAI 1-BIT / TERNARY LLM — THE COMPLETE GET-RUNNING GUIDE
(Everything I discovered installing Bonsai LLM myself) =============================================================

If you've heard about PrismML's "Bonsai" models — the tiny, on-device LLMs that squeeze frontier-level performance out of barely any memory — you've probably run into a confusing pile of names and releases. I spent a serious session getting Bonsai running and benchmarked every version so you don't have to. Here's everything I learned, plus the exact steps to get the best possible version downloaded, installed, and running on YOUR computer.

WHAT BONSAI IS

Bonsai is a family of heavily-quantized LLMs from Prism ML (PRISMML.COM / huggingface.co/PRISM-ML). They're built on the Qwen3 / Qwen3.6 architectures and repacked so that nearly every weight is just a tiny number {-1, 0, +1} (ternary) or even {-1, +1} (binary). That makes models that are up to ~1/8 the size of the originals while keeping most of the quality. They run great on modest CPUs with no GPU at all.

There are TWO families, and this is the most important thing to know:

  • Bonsai (1-bit) -> binary weights, GGUF type Q1_0, ~1.125 bits/weight
  • Ternary-Bonsai -> ternary weights, GGUF type Q2_0, ~2.125 bits/weight

Each comes in 27B, 8B, 4B, and 1.7B sizes.

SIZES AT A GLANCE (file sizes, rounded)

Model Size Bonsai 1.7B (Q1_0) ~0.25 GB Bonsai 4B (Q1_0) ~0.57 GB Bonsai 8B (Q1_0) ~1.16 GB Bonsai 27B (Q1_0) ~3.80 GB Ternary-Bonsai 1.7B (Q2_0) ~0.46 GB Ternary-Bonsai 4B (Q2_0) ~1.07 GB Ternary-Bonsai 8B (Q2_0) ~2.18 GB Ternary-Bonsai 27B (Q2_0) ~7.17 GB

THE MOST IMPORTANT FINDING

On a normal x86 CPU machine, the WINNER is:

THE 1-BIT BONSAI MODELS (Q1_0) ON STOCK / MAINLINE llama.cpp.

Why? The Q1_0 (1-bit) format is FULLY MERGED into the official llama.cpp with fast CPU kernels. The Ternary (Q2_0) format is only partially merged, and on x86 CPUs it still falls back to a slow generic kernel. So on the same machine, 1-bit Bonsai runs roughly FOUR TO FIVE TIMES faster than the Ternary version, uses HALF the memory, and is simpler to set up.

I measured, on an older quad-core i7-6600U (AVX2), generation speed:

1-bit Bonsai 1.7B   ~17  tokens/sec
1-bit Bonsai 4B     ~9   tokens/sec
1-bit Bonsai 8B     ~5   tokens/sec
Ternary-Bonsai 4B   ~1.3 tokens/sec   (slow on x86)

The trade-off: 1-bit rates slightly lower on benchmarks (~70 avg vs ~75 for ternary). For day-to-day chat on CPU, the 1-bit family is the obvious pick. If you want the extra quality and have a fast ARM / Apple / GPU machine, use the Ternary family with the g64 files (see below).

WHAT YOU NEED

1) A llama.cpp BINARY (nothing custom — the official release works). 2) A model GGUF file (download from HuggingFace). 3) ~2-8 GB of RAM free, depending on model size.

STEP 1 — GET llama.cpp

Go to: https://github.com/ggml-org/llama.cpp/releases and download the newest "bin" for your OS, e.g.:

Windows (CPU): llama-b-bin-win-cpu-x64.zip Apple Silicon: llama-b-bin-macos-universal.zip Linux CUDA: llama-b-bin-ubuntu--cuda.zip

Unzip it into a folder (I use "bin-mainline"). You only need llama-cli.exe, llama-completion.exe, and/or llama-server.exe plus the DLLs that come with them.

STEP 2 — GET THE MODELS

The simplest method: click one of these in your browser, or use "hf download" from the huggingface_hub CLI. Pick a size you can handle. For most people, start with 4B or 8B.

1-bit Bonsai (recommended on CPU): https://huggingface.co/prism-ml/Bonsai-4B-gguf/resolve/main/Bonsai-4B-Q1_0.gguf (swap "4B" for 1.7B / 8B / 27B as desired)

Ternary-Bonsai (g64, mainline-compatible): https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0_g64.gguf (available for 1.7B / 4B / 8B; 27B uses "Q2_g64" naming)

Command-line alternative: pip install huggingface_hub[cli] hf download prism-ml/Bonsai-4B-gguf Bonsai-4B-Q1_0.gguf

STEP 3 — RUN IT

From the folder with llama-cli.exe and the model file:

Interactive chat: llama-cli.exe -m Bonsai-4B-Q1_0.gguf -t 4 -cnv -c 4096

One-shot (answer a prompt, then exit): llama-completion.exe -m Bonsai-4B-Q1_0.gguf -t 4 -n 64 -p "Explain fire briefly."

Local OpenAI-compatible server (port 8080): llama-server.exe -m Bonsai-4B-Q1_0.gguf -c 4096 --port 8080

Replace "-t 4" with your machine's physical core count.

QUICK TUNING TIPS FOR BEST PERFORMANCE
  • Threads: use physical cores, not SMT threads (e.g. "-t 4" on a 4-core).
  • Long contexts: cut KV-cache memory ~2-4x by adding --cache-type-k q4_0 --cache-type-v q4_0 This lets you raise -c (context length) without blowing up RAM.
  • Small RAM: keep -c around 2048-4096 (the KV cache grows with context).
  • The 27B models on a plain CPU are VERY slow (~0.2-0.4 tok/s). They're meant for GPU/Apple machines or for one-shot tests.
HARDWARE-SPECIFIC RECOMMENDATIONS

x86 CPU (any Intel/AMD desktop/laptop): 1-bit Bonsai (Q1_0) on mainline llama.cpp. That's it.

Apple Silicon (M-series): You have fast ARM kernels. You can run either family well. The mainline "Q2_0_g64" Ternary files are fast here (higher quality), and the 1-bit Q1_0 works too. Prism also ships MLX formats if you prefer the native Apple stack.

NVIDIA GPU: Mainline llama.cpp has CUDA kernels for Q1_0 and its own DSpark speculative decoding. For maximum throughput on a beefy GPU, the "AWQ 4-bit" variants served with sglang/vLLM are the way: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

AMD / Intel integrated GPU (Vulkan): Mainline llama.cpp can offload layers with -ngl; use the g64 Ternary or 1-bit files.

SPECIAL CASE: LOW-END CPUs THAT LACK AVX (e.g. Pentium Silver N5030)

If your CPU only supports SSE4.2 (no AVX/AVX2) — like the Intel Pentium Silver N5030 Spark — the OFFICIAL prebuilt llama.cpp will CRASH. You must build llama.cpp yourself without AVX. I did exactly that; here are the build flags that work:

cd llama.cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_SSE42=ON \ -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \ -DGGML_BMI2=OFF -B build ninja -C build llama-cli llama-server

Notes for that machine: * Use the q64 Ternary file for 8B: "Ternary-Bonsai-8B-Q2_0_g64.gguf". The plain "Q2_0" (g128) file will NOT load (tensor-offset mismatch). * Speeds expected: 1-bit 27B ~0.4 tok/s, g64 Ternary 8B ~0.7 tok/s. * Keep -c at 2048 and -t 4; it has only 8 GB RAM.

EXTRA OPTIONS YOU MIGHT SEE
  • DSpark speculative decoding: uses the "*-27B-dspark-Q4_1.gguf" drafters to predict+verify extra tokens. Mainly helps on CUDA (~1.8-2x there). It is fork-only and gains little on CPU, so ignore it on a CPU box.
  • MLX 1-bit / 2-bit formats: for Apple Silicon via the MLX runtime.
  • AWQ 4-bit: for NVIDIA via sglang/vLLM.
BOTTOM LINE

For the quickest, best experience on a regular PC right now: 1. Grab the latest official llama.cpp. 2. Download Bonsai-4B-Q1_0.gguf (or 8B) from prism-ml. 3. Run: llama-cli -m Bonsai-4B-Q1_0.gguf -t -cnv That's it. Fast, tiny, and free.

For even more detail, the Prism ML docs and demo live at: https://github.com/PrismML-Eng/Bonsai-demo

Happy tinkering! -- aibynick

Version 2Aug 17, 2026 at 12:25

I've developed a strong interest in the Bonsai models. They let you run significantly useful models, including even Qwen 27b, at reasonably useful speeds, on machines that don't have a GPU. I can get actually useful code results from these models, on little netbooks that cost less than $100.

In all my most recent tests, I've had Deepseek v4 Flash perform all downloads, installs, and configurations of the models and llama cpp needed to run it. It's run successfully on extremely low powered hardware - we even got it going on machines with terrible Pentium Silver N5030 CPUs. That's truly amazing, and I think can actually be useful in some situations people have no access to LLM APIs or expensive GPUs.

I had Deepseek V4 Flash write the following article, to post here, immediately after our last installation and test of the Bonsai models. I think these little things are so exciting!

============================================================= BONSAI 1-BIT / TERNARY LLM — THE COMPLETE GET-RUNNING GUIDE
(Everything I discovered installing Bonsai LLM myself) =============================================================

If you've heard about PrismML's "Bonsai" models — the tiny, on-device LLMs that squeeze frontier-level performance out of barely any memory — you've probably run into a confusing pile of names and releases. I spent a serious session getting Bonsai running and benchmarked every version so you don't have to. Here's everything I learned, plus the exact steps to get the best possible version downloaded, installed, and running on YOUR computer.

WHAT BONSAI IS

Bonsai is a family of heavily-quantized LLMs from Prism ML (PRISMML.COM / huggingface.co/PRISM-ML). They're built on the Qwen3 / Qwen3.6 architectures and repacked so that nearly every weight is just a tiny number {-1, 0, +1} (ternary) or even {-1, +1} (binary). That makes models that are up to ~1/8 the size of the originals while keeping most of the quality. They run great on modest CPUs with no GPU at all.

There are TWO families, and this is the most important thing to know:

  • Bonsai (1-bit) -> binary weights, GGUF type Q1_0, ~1.125 bits/weight
  • Ternary-Bonsai -> ternary weights, GGUF type Q2_0, ~2.125 bits/weight

Each comes in 27B, 8B, 4B, and 1.7B sizes.

SIZES AT A GLANCE (file sizes, rounded)

Model Size Bonsai 1.7B (Q1_0) ~0.25 GB Bonsai 4B (Q1_0) ~0.57 GB Bonsai 8B (Q1_0) ~1.16 GB Bonsai 27B (Q1_0) ~3.80 GB Ternary-Bonsai 1.7B (Q2_0) ~0.46 GB Ternary-Bonsai 4B (Q2_0) ~1.07 GB Ternary-Bonsai 8B (Q2_0) ~2.18 GB Ternary-Bonsai 27B (Q2_0) ~7.17 GB

THE MOST IMPORTANT FINDING

On a normal x86 CPU machine, the WINNER is:

THE 1-BIT BONSAI MODELS (Q1_0) ON STOCK / MAINLINE llama.cpp.

Why? The Q1_0 (1-bit) format is FULLY MERGED into the official llama.cpp with fast CPU kernels. The Ternary (Q2_0) format is only partially merged, and on x86 CPUs it still falls back to a slow generic kernel. So on the same machine, 1-bit Bonsai runs roughly FOUR TO FIVE TIMES faster than the Ternary version, uses HALF the memory, and is simpler to set up.

I measured, on an older quad-core i7-6600U (AVX2), generation speed:

1-bit Bonsai 1.7B   ~17  tokens/sec
1-bit Bonsai 4B     ~9   tokens/sec
1-bit Bonsai 8B     ~5   tokens/sec
Ternary-Bonsai 4B   ~1.3 tokens/sec   (slow on x86)

The trade-off: 1-bit rates slightly lower on benchmarks (~70 avg vs ~75 for ternary). For day-to-day chat on CPU, the 1-bit family is the obvious pick. If you want the extra quality and have a fast ARM / Apple / GPU machine, use the Ternary family with the g64 files (see below).

WHAT YOU NEED

1) A llama.cpp BINARY (nothing custom — the official release works). 2) A model GGUF file (download from HuggingFace). 3) ~2-8 GB of RAM free, depending on model size.

STEP 1 — GET llama.cpp

Go to: https://github.com/ggml-org/llama.cpp/releases and download the newest "bin" for your OS, e.g.:

Windows (CPU): llama-b-bin-win-cpu-x64.zip Apple Silicon: llama-b-bin-macos-universal.zip Linux CUDA: llama-b-bin-ubuntu--cuda.zip

Unzip it into a folder (I use "bin-mainline"). You only need llama-cli.exe, llama-completion.exe, and/or llama-server.exe plus the DLLs that come with them.

STEP 2 — GET THE MODELS

The simplest method: click one of these in your browser, or use "hf download" from the huggingface_hub CLI. Pick a size you can handle. For most people, start with 4B or 8B.

1-bit Bonsai (recommended on CPU): https://huggingface.co/prism-ml/Bonsai-4B-gguf/resolve/main/Bonsai-4B-Q1_0.gguf (swap "4B" for 1.7B / 8B / 27B as desired)

Ternary-Bonsai (g64, mainline-compatible): https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0_g64.gguf (available for 1.7B / 4B / 8B; 27B uses "Q2_g64" naming)

Command-line alternative: pip install huggingface_hub[cli] hf download prism-ml/Bonsai-4B-gguf Bonsai-4B-Q1_0.gguf

STEP 3 — RUN IT

From the folder with llama-cli.exe and the model file:

Interactive chat: llama-cli.exe -m Bonsai-4B-Q1_0.gguf -t 4 -cnv -c 4096

One-shot (answer a prompt, then exit): llama-completion.exe -m Bonsai-4B-Q1_0.gguf -t 4 -n 64 -p "Explain fire briefly."

Local OpenAI-compatible server (port 8080): llama-server.exe -m Bonsai-4B-Q1_0.gguf -c 4096 --port 8080

Replace "-t 4" with your machine's physical core count.

QUICK TUNING TIPS FOR BEST PERFORMANCE
  • Threads: use physical cores, not SMT threads (e.g. "-t 4" on a 4-core).
  • Long contexts: cut KV-cache memory ~2-4x by adding --cache-type-k q4_0 --cache-type-v q4_0 This lets you raise -c (context length) without blowing up RAM.
  • Small RAM: keep -c around 2048-4096 (the KV cache grows with context).
  • The 27B models on a plain CPU are VERY slow (~0.2-0.4 tok/s). They're meant for GPU/Apple machines or for one-shot tests.
HARDWARE-SPECIFIC RECOMMENDATIONS

x86 CPU (any Intel/AMD desktop/laptop): 1-bit Bonsai (Q1_0) on mainline llama.cpp. That's it.

Apple Silicon (M-series): You have fast ARM kernels. You can run either family well. The mainline "Q2_0_g64" Ternary files are fast here (higher quality), and the 1-bit Q1_0 works too. Prism also ships MLX formats if you prefer the native Apple stack.

NVIDIA GPU: Mainline llama.cpp has CUDA kernels for Q1_0 and its own DSpark speculative decoding. For maximum throughput on a beefy GPU, the "AWQ 4-bit" variants served with sglang/vLLM are the way: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

AMD / Intel integrated GPU (Vulkan): Mainline llama.cpp can offload layers with -ngl; use the g64 Ternary or 1-bit files.

SPECIAL CASE: LOW-END CPUs THAT LACK AVX (e.g. Pentium Silver N5030)

If your CPU only supports SSE4.2 (no AVX/AVX2) — like the Intel Pentium Silver N5030 Spark — the OFFICIAL prebuilt llama.cpp will CRASH. You must build llama.cpp yourself without AVX. I did exactly that; here are the build flags that work:

cd llama.cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_SSE42=ON \ -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \ -DGGML_BMI2=OFF -B build ninja -C build llama-cli llama-server

Notes for that machine: * Use the q64 Ternary file for 8B: "Ternary-Bonsai-8B-Q2_0_g64.gguf". The plain "Q2_0" (g128) file will NOT load (tensor-offset mismatch). * Speeds expected: 1-bit 27B ~0.4 tok/s, g64 Ternary 8B ~0.7 tok/s. * Keep -c at 2048 and -t 4; it has only 8 GB RAM.

EXTRA OPTIONS YOU MIGHT SEE
  • DSpark speculative decoding: uses the "*-27B-dspark-Q4_1.gguf" drafters to predict+verify extra tokens. Mainly helps on CUDA (~1.8-2x there). It is fork-only and gains little on CPU, so ignore it on a CPU box.
  • MLX 1-bit / 2-bit formats: for Apple Silicon via the MLX runtime.
  • AWQ 4-bit: for NVIDIA via sglang/vLLM.
BOTTOM LINE

For the quickest, best experience on a regular PC right now: 1. Grab the latest official llama.cpp. 2. Download Bonsai-4B-Q1_0.gguf (or 8B) from prism-ml. 3. Run: llama-cli -m Bonsai-4B-Q1_0.gguf -t -cnv That's it. Fast, tiny, and free.

For even more detail, the Prism ML docs and demo live at: https://github.com/PrismML-Eng/Bonsai-demo

Happy tinkering! -- aibynick

Version 1Aug 17, 2026 at 01:33

I've developed a strong interest in the Bonsai models. They let you run really useful models, including even Qwen 27b, at reasonably useful speeds, on machines that don't have a GPU. I can get actually useful code results from these models, on little netbooks that cost less than $100.

I've had deepseek v4 flash perform all the installs, and even got it going on machines with very low power Pentium Sivler N5030 CPUs. That's truly amazing, and I think will lead to some truly situation changes capabilities for people who can't afford an expensive GPU.

I had Deepseek V4 Flash write the following article, to post here, immediately after our last installation and test of the Bonsai models. I think these things are so exciting!

============================================================= BONSAI 1-BIT / TERNARY LLM — THE COMPLETE GET-RUNNING GUIDE (Everything I discovered installing Bonsai LLM myself) =============================================================

If you've heard about PrismML's "Bonsai" models — the tiny, on-device LLMs that squeeze frontier-level performance out of barely any memory — you've probably run into a confusing pile of names and releases. I spent a serious session getting Bonsai running and benchmarked every version so you don't have to. Here's everything I learned, plus the exact steps to get the best possible version downloaded, installed, and running on YOUR computer.

WHAT BONSAI IS

Bonsai is a family of heavily-quantized LLMs from Prism ML (PRISMML.COM / huggingface.co/PRISM-ML). They're built on the Qwen3 / Qwen3.6 architectures and repacked so that nearly every weight is just a tiny number {-1, 0, +1} (ternary) or even {-1, +1} (binary). That makes models that are up to ~1/8 the size of the originals while keeping most of the quality. They run great on modest CPUs with no GPU at all.

There are TWO families, and this is the most important thing to know:

  • Bonsai (1-bit) -> binary weights, GGUF type Q1_0, ~1.125 bits/weight
  • Ternary-Bonsai -> ternary weights, GGUF type Q2_0, ~2.125 bits/weight

Each comes in 27B, 8B, 4B, and 1.7B sizes.

SIZES AT A GLANCE (file sizes, rounded)

Model Size Bonsai 1.7B (Q1_0) ~0.25 GB Bonsai 4B (Q1_0) ~0.57 GB Bonsai 8B (Q1_0) ~1.16 GB Bonsai 27B (Q1_0) ~3.80 GB Ternary-Bonsai 1.7B (Q2_0) ~0.46 GB Ternary-Bonsai 4B (Q2_0) ~1.07 GB Ternary-Bonsai 8B (Q2_0) ~2.18 GB Ternary-Bonsai 27B (Q2_0) ~7.17 GB

THE MOST IMPORTANT FINDING

On a normal x86 CPU machine, the WINNER is:

THE 1-BIT BONSAI MODELS (Q1_0) ON STOCK / MAINLINE llama.cpp.

Why? The Q1_0 (1-bit) format is FULLY MERGED into the official llama.cpp with fast CPU kernels. The Ternary (Q2_0) format is only partially merged, and on x86 CPUs it still falls back to a slow generic kernel. So on the same machine, 1-bit Bonsai runs roughly FOUR TO FIVE TIMES faster than the Ternary version, uses HALF the memory, and is simpler to set up.

I measured, on an older quad-core i7-6600U (AVX2), generation speed:

1-bit Bonsai 1.7B   ~17  tokens/sec
1-bit Bonsai 4B     ~9   tokens/sec
1-bit Bonsai 8B     ~5   tokens/sec
Ternary-Bonsai 4B   ~1.3 tokens/sec   (slow on x86)

The trade-off: 1-bit rates slightly lower on benchmarks (~70 avg vs ~75 for ternary). For day-to-day chat on CPU, the 1-bit family is the obvious pick. If you want the extra quality and have a fast ARM / Apple / GPU machine, use the Ternary family with the g64 files (see below).

WHAT YOU NEED

1) A llama.cpp BINARY (nothing custom — the official release works). 2) A model GGUF file (download from HuggingFace). 3) ~2-8 GB of RAM free, depending on model size.

STEP 1 — GET llama.cpp

Go to: https://github.com/ggml-org/llama.cpp/releases and download the newest "bin" for your OS, e.g.:

Windows (CPU): llama-b-bin-win-cpu-x64.zip Apple Silicon: llama-b-bin-macos-universal.zip Linux CUDA: llama-b-bin-ubuntu--cuda.zip

Unzip it into a folder (I use "bin-mainline"). You only need llama-cli.exe, llama-completion.exe, and/or llama-server.exe plus the DLLs that come with them.

STEP 2 — GET THE MODELS

The simplest method: click one of these in your browser, or use "hf download" from the huggingface_hub CLI. Pick a size you can handle. For most people, start with 4B or 8B.

1-bit Bonsai (recommended on CPU): https://huggingface.co/prism-ml/Bonsai-4B-gguf/resolve/main/Bonsai-4B-Q1_0.gguf (swap "4B" for 1.7B / 8B / 27B as desired)

Ternary-Bonsai (g64, mainline-compatible): https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0_g64.gguf (available for 1.7B / 4B / 8B; 27B uses "Q2_g64" naming)

Command-line alternative: pip install huggingface_hub[cli] hf download prism-ml/Bonsai-4B-gguf Bonsai-4B-Q1_0.gguf

STEP 3 — RUN IT

From the folder with llama-cli.exe and the model file:

Interactive chat: llama-cli.exe -m Bonsai-4B-Q1_0.gguf -t 4 -cnv -c 4096

One-shot (answer a prompt, then exit): llama-completion.exe -m Bonsai-4B-Q1_0.gguf -t 4 -n 64 -p "Explain fire briefly."

Local OpenAI-compatible server (port 8080): llama-server.exe -m Bonsai-4B-Q1_0.gguf -c 4096 --port 8080

Replace "-t 4" with your machine's physical core count.

QUICK TUNING TIPS FOR BEST PERFORMANCE
  • Threads: use physical cores, not SMT threads (e.g. "-t 4" on a 4-core).
  • Long contexts: cut KV-cache memory ~2-4x by adding --cache-type-k q4_0 --cache-type-v q4_0 This lets you raise -c (context length) without blowing up RAM.
  • Small RAM: keep -c around 2048-4096 (the KV cache grows with context).
  • The 27B models on a plain CPU are VERY slow (~0.2-0.4 tok/s). They're meant for GPU/Apple machines or for one-shot tests.
HARDWARE-SPECIFIC RECOMMENDATIONS

x86 CPU (any Intel/AMD desktop/laptop): 1-bit Bonsai (Q1_0) on mainline llama.cpp. That's it.

Apple Silicon (M-series): You have fast ARM kernels. You can run either family well. The mainline "Q2_0_g64" Ternary files are fast here (higher quality), and the 1-bit Q1_0 works too. Prism also ships MLX formats if you prefer the native Apple stack.

NVIDIA GPU: Mainline llama.cpp has CUDA kernels for Q1_0 and its own DSpark speculative decoding. For maximum throughput on a beefy GPU, the "AWQ 4-bit" variants served with sglang/vLLM are the way: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

AMD / Intel integrated GPU (Vulkan): Mainline llama.cpp can offload layers with -ngl; use the g64 Ternary or 1-bit files.

SPECIAL CASE: LOW-END CPUs THAT LACK AVX (e.g. Pentium Silver N5030)

If your CPU only supports SSE4.2 (no AVX/AVX2) — like the Intel Pentium Silver N5030 Spark — the OFFICIAL prebuilt llama.cpp will CRASH. You must build llama.cpp yourself without AVX. I did exactly that; here are the build flags that work:

cd llama.cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_SSE42=ON \ -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \ -DGGML_BMI2=OFF -B build ninja -C build llama-cli llama-server

Notes for that machine: * Use the q64 Ternary file for 8B: "Ternary-Bonsai-8B-Q2_0_g64.gguf". The plain "Q2_0" (g128) file will NOT load (tensor-offset mismatch). * Speeds expected: 1-bit 27B ~0.4 tok/s, g64 Ternary 8B ~0.7 tok/s. * Keep -c at 2048 and -t 4; it has only 8 GB RAM.

EXTRA OPTIONS YOU MIGHT SEE
  • DSpark speculative decoding: uses the "*-27B-dspark-Q4_1.gguf" drafters to predict+verify extra tokens. Mainly helps on CUDA (~1.8-2x there). It is fork-only and gains little on CPU, so ignore it on a CPU box.
  • MLX 1-bit / 2-bit formats: for Apple Silicon via the MLX runtime.
  • AWQ 4-bit: for NVIDIA via sglang/vLLM.
BOTTOM LINE

For the quickest, best experience on a regular PC right now: 1. Grab the latest official llama.cpp. 2. Download Bonsai-4B-Q1_0.gguf (or 8B) from prism-ml. 3. Run: llama-cli -m Bonsai-4B-Q1_0.gguf -t -cnv That's it. Fast, tiny, and free.

For even more detail, the Prism ML docs and demo live at: https://github.com/PrismML-Eng/Bonsai-demo

Happy tinkering! -- aibynick

Back to Thread