Run Bonsai Locally on Windows: 1-bit Models, Vision, and MCP Tool Calls

Bonsai is a set of low-bit native models. The official demo repository provides sizes of 1.7B, 4B, 8B and 27B, and integrates chat, vision, inference strength, OpenAI style tool calls and MCP.

Project address:

https://github.com/PrismML-Eng/Bonsai-demo

“1-bit” does not mean that when the model is run, it is as simple as dividing the number of parameters by eight. The weights can be compressed very small, but inference still requires additional memory such as KV Cache, runtime, context and image encoding. When selecting a model, look at the available memory of the entire machine, not just the download file size.

Choose 1-bit or three-valued model first

The official demo offers two model families:

It is recommended to select the default Ternary-Bonsai for the first time running. Don’t default to it for better results just because “1-bit” is more eye-catching.

Preparation before Windows installation

It is recommended to prepare:

• 64-bit Windows 11;
• Git；
• PowerShell；
• Sufficient memory and disk space;
• Network with access to Hugging Face;
• Hugging Face Token required for 27B model.

The availability status of scripts and models may vary between versions. You should check the warehouse README before running, and do not permanently write down the model address in the old tutorial.

Clone project

1
2

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

Check available space first:

1

Get-PSDrive -PSProvider FileSystem

Models, caches, and binary files may occupy several GB to tens of GB. Do not start downloading directly when there is insufficient system disk space.

Select model size

By default 27B is used:

1

$env:BONSAI_MODEL = "27B"

For ordinary computers, it is recommended to start with 4B or 8B:

1

$env:BONSAI_MODEL = "4B"

Select a three-value model family:

1

$env:BONSAI_FAMILY = "ternary"

If you want to test 1-bit:

1

$env:BONSAI_FAMILY = "1bit"

Environment variables are only valid for the current PowerShell session and need to be reset after closing the window.

Configure Hugging Face Token

If the selected model repository requires authorization:

1

$env:BONSAI_TOKEN = "hf_your_token_here"

Do not write real tokens into articles, scripts, Git repositories, or terminal screenshots. You can clear it after use:

1

Remove-Item Env:BONSAI_TOKEN

If 401 or 403 appears, first confirm that you have accepted the model permission on the Hugging Face page, and then check whether the Token has read permission.

Run the installation script

Official Windows process:

1
2

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1

-Scope Process only affects the current PowerShell process. Restoring the original policy after closing the terminal is more reliable than permanently lowering the system execution policy.

The script installs dependencies based on environment variables, downloads the model, and prepares the binaries required to run. Do not repeatedly delete the entire directory when the download is interrupted. First check whether the cache and script support continued downloading.

Command line test

After the installation is complete, you can directly send the prompt word:

1

.\scripts\run_llama.ps1 -p "请用三句话解释什么是低比特大模型"

If you just want to confirm that the model can be loaded, use a short hint word and a short context first. Long contexts will significantly increase the KV Cache and may cause models that were originally launched to suddenly run out of memory.

Start the web chat interface

The official demo can start the local llama server and provide chat, visual and tool calling interfaces by default. The current Windows scripts in the warehouse may be adjusted with the version, you can check first:

1

Get-ChildItem .\scripts\*server*

After startup, usually access:

1

http://localhost:8080

If the port cannot be accessed, check whether the process is still running, whether the firewall prompt is denied, and whether 8080 is occupied by other programs.

How to use visual ability

The Bonsai demo supports sending photos, screenshots, and PDFs. Please pay attention to the following when using it in practice:

• Images will occupy additional memory;
• High-resolution screenshots do not necessarily lead to more accurate text recognition;
• If the PDF has many pages, it should be split first or only the relevant pages should be sent;
• Although private documents are inferred locally, the interface, logs and temporary file storage locations must be confirmed.

You can test it with a simple screenshot first: ask the model to list the page buttons, explain the error message, and then check whether it has fabricated invisible content.

Tool calls and MCP

The demonstration supports OpenAI style tool_calls, and can connect to MCP Server in the demonstration interface. The model makes tool calls, but it is the host program that actually executes the tools.

Before accessing file, terminal or browser MCP, you should first confirm:

1. What tools will be exposed by Server?
2. Whether to limit it to the specified working directory;
3. Whether dangerous commands require manual confirmation;
4. Whether the tool output is written to the chat log;
5. Will the model be called repeatedly when it fails?

The small size of the low-bit model does not mean that the tool call judgment must be reliable. Approvals and least privileges must be preserved when it comes to deleting files, executing shells, sending messages, or modifying external systems.

How big a model can the computer run?

The following can only be used as a conservative starting point. Actual requirements will be affected by model format, context length, running backend and visual input:

Even if 27B weights could fit into memory, speed would likely be limited by memory bandwidth and CPU instruction set. Being able to load does not mean that the interactive experience is smooth.

FAQ

PowerShell prohibits script execution

Release only for the current process:

1

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Do not directly modify the permanent execution policy of the entire computer.

Model download reports 401 or 403

Check whether the model requires permission, whether BONSAI_TOKEN exists, and whether the Token has read permission:

1

Test-Path Env:BONSAI_TOKEN

Insufficient memory when loading

Change to a smaller model:

1
2

$env:BONSAI_MODEL = "4B"
.\setup.ps1

Also shorten context, turn off visual input and other memory-hogging software. Don’t rely on the Windows page file to “hard run” a model that severely exceeds physical memory. The speed will usually be very poor.

The generation speed is very slow

Confirm the actual backend, CPU instruction set, and thread settings used. The first run may also include model loading, caching, or kernel preparation time, and first-round latency should be viewed separately from subsequent generation speeds.

MCP can connect but the tool call is unstable

First test with a tool with only one parameter, simple and read-only parameters. If the wrong parameters are often filled in for small models, the number of tools should be reduced, clear descriptions should be added, or model families and sizes should be replaced with higher quality.

How to choose between Bonsai and Ollama

Ollama is a model running and management tool, and Bonsai is a model and its presentation environment. The two are not products of the same layer.

• If you want to uniformly manage multiple common models and quickly provide APIs: give priority to Ollama;
• Want to study 1-bit/triple weights, test official visuals and toolchains: use Bonsai Demo;
• If you want to put Bonsai into an existing application: first confirm whether the current format can be loaded directly by your runtime;
• Just looking at “smaller model files” is not enough to decide the solution, also compare quality, speed, context and tool call reliability.

It is recommended to first use 4B Ternary-Bonsai to complete the closed loop of text, visual and a read-only MCP tool, and then decide whether to download 27B or switch to the 1-bit family.