LookAtMyQuant 2

At 1.2 GB our Qwen3.5-4B solves 330 of 540 math problems and Bonsai 2 Ternary solves 328

Bonsai 2 Ternary 4B [1] is Prism ML's compression of Qwen3.5-4B [2] to three values per weight. Its language weights take 1.20 GB. The original's take 8.4 GB.

Our file is the same model at 2.3 bits per weight, 1.23 GB. The compiler from LookAtMyQuant builds it from the original's weights, and 4,000 steps of training follow.

The math test is 540 generated problems with exact answers. There are twelve kinds of problem, from long addition and multiplication to determinants, modular powers and linear systems, at three difficulties. A problem counts only if the final answer is exactly right within 2,048 tokens. Models answer greedily, with thinking turned off.

Ours solves 330, Bonsai 328 and the original 359. 62 problems are solved only by ours and 60 only by Bonsai, so the test does not separate the two files.

On text, ours has a lower KL than Bonsai on 11 of 13 sources

KL divergence measures how far a model's next-token predictions are from the original's. Lower is better. On held-out text from 13 sources, ours averages 0.45 and Bonsai 0.59. On WikiText-2, ours is 0.42 and Bonsai 0.50.

Figure 2 shows each source. Bonsai is lower on chat and Hindi. Ours is lower on the rest, by the most on Chinese, Korean, Japanese and Arabic.

A smaller Qwen3.5 at more bits per weight solves 284 of 540 at best

The same space can hold a smaller model with more bits for each weight. We rounded the two smaller Qwen3.5 models the way MLX's standard files do [8]. Weights are stored in groups of 64, and each group has its own scale and offset.

Qwen3.5-2B at 5 bits is 1.29 GB and solves 284. At 4 bits it is 1.06 GB and solves 242. Qwen3.5-0.8B at 8 bits is 0.80 GB and solves 205. More bits would not help the 0.8B, because 8 bits already change its weights by only half a percent. No Qwen3.5 model fills 1.2 GB at 8 bits. That takes 1.1 billion weights, and these two models have 0.8 and 1.9 billion.

On text, their KL from the original 4B's predictions is 0.51, 0.60 and 0.75. Ours is 0.45.

Before any training, changes to the build took the file from 1 problem of 540 to 317

The build makes the 1.2 GB file from the original's weights and a sample of text. Nothing is trained. The first file at this size solved 1 problem, although its KL on WikiText-2 was 0.60 against Bonsai's 0.50. Figure 3 shows the score after each change.

Trellis codes leave a third less rounding error than one code per weight

The first file stored each weight as one of a few fixed levels. A trellis code [3] stores a row of weights as one path of bits. Each weight adds 1 to 4 bits to the path. Its value is read from a table of 4,096 values, at the position given by the path's last 12 bits. One weight's bits therefore also help to pick the values of the weights after it, and the whole row lands closer to the original.

On Gaussian weights at 2 bits, the trellis leaves 36% less squared error than the best fixed levels. In the whole model, WikiText-2 KL fell from 0.60 to 0.39 and the score went from 28 to 117.

The rest of the format follows QuIP# [4] and QTIP [3]. A random Hadamard transform rotates each matrix's inputs, which makes its weights look Gaussian. Each row has one scale. Weights are rounded 16 columns at a time, and each block's error is pushed onto the columns not yet rounded, weighted by statistics of the layer's inputs [5].

Calibrating on long worked answers, weighted toward digits, took the score from 117 to 224

Rounding needs a sample of text, because statistics of each layer's inputs decide which errors matter. With general text the file solved 1. With the original's own solutions to grade-school math problems [10] as half of the text, it solved 28.

Two more changes to the text came after the trellis codes. First, all of the text became math, and positions where the next token is a digit counted ten times as much as other positions. The score went from 117 to 147. Second, the short grade-school solutions gave way to the original's long worked solutions to competition problems [9], 64 windows of 2,048 tokens. They have many intermediate results, like the answers the test needs. The score went from 147 to 224.

Correcting half of the upstream error beat correcting all of it

Layers are rounded in order, so each layer receives inputs that already carry the error of the layers before it. A layer can be fitted to cancel that error [6]. Cancelling half of it rather than all of it took the score from 224 to 265 and WikiText-2 KL from 0.85 to 0.66.

Giving more bits to the rows that matter for digits took the score from 265 to 300

The rows of one matrix differ widely in how much they matter for predicting digits. In the matrices that read a linear-attention layer's input, a tenth of the rows carry 79% of that importance. We measured every row's importance on the original model. Important rows then got 3 or 4 bits and unimportant rows 1 or 2, at the same file size.

Moving the top eighth of each matrix's rows up one bit and the bottom eighth down gave 292. Choosing every row's bits across the whole model at once gave 300. Two more builds of the first kind gave 261 and 266.

Refitting the trellis's value tables with e-means updates added 17 problems

A trellis code needs its table of 4,096 values. We had fitted it with six passes of Lloyd's algorithm, the usual k-means, over one set of Gaussian samples. e-means [7] replaces those full passes with small batches and exponential moving averages. We fitted the table that way, with fresh samples in every batch, a decaying sum and count for each of the 4,096 values, and a weight on each new batch that falls from 0.5 to 0.01.

On Gaussian weights the new table leaves 4% to 9% less error at 1 to 4 bits per weight. The file is the same size and decodes the same way. With new tables for the 2-bit and 3-bit codes, which hold 83% of the weights, the score went from 300 to 311. With new tables for every code it went to 317. KL on the 13 sources was 2% lower with the first and 4% lower with the second.

Training added 13 problems and cut KL on text by two thirds

A trellis code cannot be fine-tuned the usual way

Changing one stored bit of a trellis code changes several weights at once, so a weight cannot take a small step. Ordinary fine-tuning of a built file can move only the row scales and the value tables, which are stored as plain numbers.

We train real-valued weights and hold them near their codes

The training runs on a copy of the model whose weights are ordinary real numbers. The copy starts exactly on the stored model. Figure 4 shows the steps.

The loss compares the copy's predictions with the original's 64 most likely tokens at 128 positions of each 512-token window. Half of each batch is worked math solutions. The other half is general text, science, code, instructions and tool calls. A step uses 8 windows. Adam's damping term is set to each matrix's typical gradient size, so weights with tiny gradients do not take full-size steps.

This training took the score from 317 to 321 and KL on the 13 sources from 1.41 to 0.79. The codes hardly moved. Only 0.1% of a layer's codes change when it is rounded again, because a weight that sits near its code rounds back to the same code.

Choosing every layer's codes again during training cut KL on text by another 43%

So we choose every layer's codes again, once, between steps 1,000 and 1,750. Each layer is rounded from the original's weights. The rounding uses statistics of the layer's inputs in the partly trained model, kept as a running average over the training batches. That average covers eight times as much text as the build's. 97% of the codes change, and training continues from the new ones.

We ran both kinds of training from two builds. With the codes chosen again, the files solve 330 and 324 problems and their KL on the 13 sources is 0.45 and 0.45. Without, they solve 321 and 314 and their KL is 0.79 and 0.80.

Table 2. Two builds, each untrained, trained, and trained with the codes chosen again.

Calibration and training cost about $34 of GPU time in all

Table 3 lists every step that makes the file once, at Modal's on-demand prices [11] for the GPU, CPU and memory each step used. Calibration is every step before training, and costs $13. Training costs $21. The steps take 5.4 GPU-hours in all. Calibration reads 131,000 tokens of text, and training reads 16 million.

Table 3. The time and cost of each step.

Limitations

• Each file is one run. Builds of one recipe have differed by 31 of 540 problems, so a gap of a few problems means nothing.
• We used this math test throughout the work, which favors our files and not Bonsai. On a second set of 216 problems of the same kinds, which we used less, our file solves 125 and Bonsai 138.
• The calibration and training text is mostly math. Beyond KL on text, we have not measured coding, tool use or instruction following.
• Our language weights are 1.23 GB packed, 2% more than Bonsai's 1.20 GB. That is the packed size of a build with the same layout. New value tables and training change the values a file stores and leave its layout alone, and we did not pack the trained file again.
• We score the weights that the stored codes decode to, at full precision. We have not timed this file on a device.

References

1. Bonsai 2 Ternary 4B. Prism ML. huggingface.co/prism-ml/Ternary-Bonsai-2-4B-mlx-2bit
2. Qwen3.5-4B. Qwen team. huggingface.co/Qwen/Qwen3.5-4B
3. QTIP. Tseng, Sun, Hou and De Sa, QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024. arXiv:2406.11235
4. QuIP#. Tseng et al., QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024. arXiv:2402.04396
5. GPTQ. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. arXiv:2210.17323
6. QEP. Arai and Ichikawa, Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization. NeurIPS 2025. arXiv:2504.09629
7. e-means. Bessa, Mazaré, Szilvasy, Touvron, Picard, Douze and Jégou, e-means: outperforming k-means (in one-tenth the epochs). 2026. arXiv:2611.09501, e-means.github.io
8. MLX. Apple machine learning research. github.com/ml-explore/mlx
9. MATH. Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021. arXiv:2103.03874
10. GSM8K. Cobbe et al., Training Verifiers to Solve Math Word Problems. 2021. arXiv:2110.14168
11. Modal. Pricing, read on 10 November 2026. modal.com/pricing

Citation

@misc{boson2026lookatmyquant2,
  title  = {LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB},
  author = {{Shin Megami Boson}},
  year   = {2026},
  month  = nov
}