eaddario/IQ2_NL-IQ3_NL-Test-Models

Experimental IQ2_NL & IQ3_NL Tests

Please note that these models cannot be used with standard llama.cpp. See quantize: add IQ2_NL and IQ3_NL types (CPU)

Quantization test results for NVIDIA-Nemotron-Nano-9B-v2, comparing standard llama.cpp quantization against a fork that adds two new non-linear block types: IQ2_NL and IQ3_NL.

The test model uses the nemotron_h hybrid Mamba-2 / attention architecture, whose tensor dimensions are divisible by 32 but not by 256. Every K-quant (Q2_K … Q6_K) and every I-quant (IQ1_M, IQ2_XXS, …) in llama.cpp operates on 256-element super-blocks. When a tensor's row length is not a multiple of 256, the quantizer cannot use the requested type and falls back to a 32-element block type.

NVIDIA-Nemotron-Nano-9B-v2 triggers this on almost every tensor in the model:

Tensor group	ncols	÷ 32	÷ 256
token_embd, output, attn_[q,k,v], ffn_up, ssm_in	4480	✅ 140	❌ 17.50
ffn_down	15680	✅ 490	❌ 61.25
attn_output	5120	✅ 160	✅ 20
ssm_out	10240	✅ 320	✅ 40

The consequence, taken directly from the quantization logs:

warning: blk.31.ffn_up.weight - ncols 4480 not divisible by 256 (required for type q2_K) -> falling back to q4_0
llama_model_quantize_impl: WARNING: 90 of 341 tensor(s) required fallback quantization

90 of 341 tensors, or rather 90 of the 122 quantizable tensors (73.8%) bypass the requested type entirely. And because the lowest-bpw 32-block type available in standard llama.cpp is Q4_0/IQ4_NL at 4.50 bpw, that fallback establishes a hard floor. A Q2_K quantization of this model does not produce a 2-bit model; it lands at ~4.50 bpw, and a IQ1_M lands at ~4.36 bpw. The sub-4 bpw region is unreachable.

The fork adds IQ2_NL and IQ3_NL non-linear quantization types built on the same layout as the existing IQ4_NL, extending the family downward.

Type	Block	Bits/weight	Bytes per 32 weights	Payload + scale	Origin
IQ2_NL	32	2.50	10	8 + 2	fork
IQ3_NL	32	3.50	14	12 + 2	fork
IQ4_NL	32	4.50	18	16 + 2	standard

Because these are 32-block types, nemotron_h's 4480 and 15680 geometries are no longer a barrier. Fallback count collapses from 90 → 1:

Build	Requested type	Fallbacks	Resulting BPW
standard	Q2_K	90 / 341	4.5000
standard	Q3_K	90 / 341	4.8400
fork	IQ2_NL	1 / 341	2.8046
fork	IQ3_NL	1 / 341	3.6722

The single remaining fallback is output.weight, requested as Q5_K but quantized to Q5_1 (6.00 bpw, 32-block).

1. Benefits

a) 32-block divisibility restores the low-bit type options

The 256-element super-block is a high-quality throughput optimisation and is an excellent option for architectures with power-of-two-friendly geometries, but a hard wall for everything else. nemotron_h and many recent architectures are the second case: neither d_model = 4480 nor d_ffn = 15680 is a whole number of super-blocks (17.5 and 61.25 respectively), so they fall back to other, usually sub-optimal, types.

IQ2_NL and IQ3_NL sidestep this entirely by inheriting IQ4_NL's 32-element block:

• Fallbacks drop from 90 / 341 to 1 / 341.
• The requested type is honoured on 120 & 121 of 122 quantizable tensors, where the naive Q2_K and Q3_K recipes honour only 27.
• The delivered bit-rate finally tracks the requested bit-rate: IQ2_NL asks for 2.50 bpw and delivers 2.80 bpw overall (the overhead being FP32 norms, Q5_1 output, and IQ3_NL embeddings), where Q2_K asks for 2.6258 bpw and delivers 4.50.

The new types generalize well beyond Nemotron. Any architecture with non-power-of-two hidden dimensions like hybrid SSM/Mamba models, pruned or distilled checkpoints, models with unusual GQA head counts, and most width-searched Neural Architecture Search (NAS) derivatives hit the same wall, and the fix is the same.

b) bypassing the 4.5 bpw floor imposed by IQ4_NL

The complete set of 32-block quantization types in llama.cpp is:

Type	Bits/weight
IQ4_NL	4.50
Q4_0	4.50
Q4_1	5.00
Q5_0	5.50
Q5_1	6.00
Q8_0	8.50

The minimum is 4.50 bpw. For a model where the 256-block types are unavailable, that value is an absolute floor on the entire model. Every "aggressive" quant type collapses onto it: IQ1_M → 4.36 bpw, Q2_K → 4.50 bpw, Q3_K → 4.84 bpw. Three nominally distinct types spanning 1.75–3.44 bpw all land within 0.5 bpw of each other, at more than double the requested budget.

IQ2_NL (2.50) and IQ3_NL (3.50) extend the non-linear family below that floor:

Metric	Standard floor (IQ1_M)	IQ3_NL	IQ2_NL
BPW	4.3600	3.6722	2.8046
Size	4.52 GiB	3.81 GiB (−15.8%)	2.91 GiB (−35.7%)
Mean PPL	8.6505	8.3177	11.6323
Mean KLD	0.129217	0.078364 (−39.4%)	0.432011
Top-1 agree	83.879%	86.045%	71.111%

IQ3_NL wins on size and on every quality metric, but not on speed. It is smaller than IQ1_M on disk and simultaneously better on perplexity, KL divergence, token-probability RMS error, and top-1 agreement, but generates tokens ~35% slower (tg128 35.53 vs 54.70 t/s, pp1024+tg1024 66.38 vs 98.43 t/s). For a 9B model this moves the practical deployment target from "needs 5 GiB" to "fits in 4 GiB" and with better quality, provided that throughput cost is acceptable.

IQ2_NL is a valuable capability extension. At 2.80 bpw it produces a model 17.6% the size of F16, an option that previously did not exist at any quality, but the fidelity cost is substantial: PPL +50.28%, mean KLD 0.432, and top-1 agreement of only 71.1%. Roughly three tokens in ten change rank against the F16 reference. This is characteristic of ~2.5 bpw quantization generally rather than a defect of the type, and it is the expected shape of the curve; but it means IQ2_NL should be understood as a last-resort type for hard VRAM ceilings, not a default. IQ3_NL is where the fork's practical value sits.

Standard IQ1_M is outperformed once a coherent 2.5 bpw fallback exists.

Key Takeaways

• IQ3_NL outperforms the entire sub-5-bpw range. At 3.81 GiB / 3.67 bpw it is 15.8% smaller than the smallest model standard llama.cpp can build (IQ1_M, 4.52 GiB / 4.36 bpw) while scoring materially better on every metric: PPL 8.3177 vs 8.6505, mean KLD 0.0784 vs 0.1292 (−39.4%), top-1 agreement 86.05% vs 83.88%.
• IQ2_NL removes the 4.5 bpw floor. At 2.91 GiB / 2.80 bpw it is 35.7% smaller than anything the standard types can produce for this model.
• IQ2_NL also outperforms IQ1_M. It is 7.8% smaller and 6.8% lower KLD than the fork's own IQ1_M.
• Nothing at Q4_K and above changes. Those four quantized models are bit-for-bit identical across the two test runs; the quantization recipes never select the new types at those levels.

2. Disadvantages

1. Codebook efficiency. At 2.50 bpw, IQ2_NL sits at the same bit-rate as IQ2_S and above IQ2_XS (2.31) and IQ2_XXS (2.06), which spend their bits on lattice codebooks rather than a per-32 scale. On well-aligned models the super-block I-quants should still win at the low end. The NL family's value is availability, not peak efficiency.
2. Kernel maturity. Quantization wall-clock is ~2.7× the standard path (41.8 s vs 15.8 s for the 2-bit type on this test). More significantly, IQ3_NL inference is materially slower: tg128 35.53 t/s against 51.16 for standard Q3_K (−30.6%) and 54.70 for standard IQ1_M (−35.0%); pp1024+tg1024 66.38 against 92.00 and 98.43 (−27.8% and −32.6%). Prompt processing is unaffected (pp512 600.65 vs 600.77), which points at the token-generation dot-product path specifically. The regression does not generalise to the new types as a class: IQ2_NL posts the fastest tg128 in the benchmark table at 57.86 t/s (+2.3% over standard Q2_K) and the fork's IQ1_M is +2.1% over its standard counterpart, both while being substantially smaller. That isolates the cost to IQ3_NL's kernel rather than to 32-element blocks in general.
3. The mid-to-high types are untouched. Q4_K through Q8_0 are bit identical between the two runs. Those recipes still fall back to Q5_0/Q5_1/Q8_0, inflating Q4_K to 5.87 bpw and Q6_K to 8.22 bpw. Routing those fallbacks to IQ4_NL would shrink Q4_K on this architecture with no change to the new types.
4. output.weight remains a fallback. The recipe requests Q5_K and gets Q5_1 at 6.00 bpw for a 1120 MiB tensor (~420 MiB quantized), or ~14% of the entire IQ2_NL model. An IQ5_NL, or simply requesting IQ4_NL instead, would recover a meaningful fraction of that.
5. Single architecture, single model. Every conclusion here is drawn from one 9B hybrid-SSM checkpoint against one calibration corpus (imatrix). The 256-divisibility argument generalises well but the specific metrics results do not.

3. Comparative Performance Tables

Lower is better for every metric except Top-1.

Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build. Rows 4–7 are identical between the two test runs.

S·Type	S·BPW	S·GiB	S·PPL	S·PPL Δ%	S·KLD	F·Type	F·BPW	F·GiB	F·PPL	F·PPL Δ%	F·KLD
IQ1_M	4.3600	4.52	8.6505	+11.76%	0.129217	IQ1_M	3.0416	3.15	11.8810	+53.50%	0.463744
Q2_K	4.5000	4.66	8.0878	+4.49%	0.052257	IQ2_NL	2.8046	2.91	11.6323	+50.28%	0.432011
Q3_K	4.8400	5.01	7.8840	+1.86%	0.025244	IQ3_NL	3.6722	3.81	8.3177	+7.46%	0.078364
Q4_K	5.8664	6.08	7.8034	+0.82%	0.006809	Q4_K	5.8664	6.08	7.8034	+0.82%	0.006809
Q5_K	6.3562	6.58	7.7752	+0.45%	0.003110	Q5_K	6.3562	6.58	7.7752	+0.45%	0.003110
Q6_K	8.2158	8.51	7.7488	+0.11%	0.000528	Q6_K	8.2158	8.51	7.7488	+0.11%	0.000528
Q8_0	8.5058	8.81	7.7471	+0.09%	0.000388	Q8_0	8.5058	8.81	7.7471	+0.09%	0.000388

As row 3 shows, IQ3_NL is 24.0% smaller than the naive Q3_K it replaces, but that naive Q3_K is a 4.84 bpw model wearing a 3-bit label.

Extended Metrics

Impl	Type	BPW	Size (GiB)	% of F16	Fallbacks	Mean PPL(Q)	Mean KLD	99% KLD	Δp RMS	Top-1 agree
-	F16	16.0040	16.57	100.0%	-	7.740272 ± 0.051484	-	-	-	-
naive	IQ1_M	4.3600	4.52	27.3%	90	8.650547 ± 0.058683	0.129217 ± 0.000514	0.904445	9.762%	83.879%
naive	Q2_K	4.5000	4.66	28.1%	90	8.087846 ± 0.053982	0.052257 ± 0.000208	0.354226	6.166%	89.151%
naive	Q3_K	4.8400	5.01	30.2%	90	7.883991 ± 0.052571	0.025244 ± 0.000102	0.167221	4.232%	92.203%
fork	IQ1_M	3.0416	3.15	19.0%	90	11.881048 ± 0.087265	0.463744 ± 0.001581	3.016266	18.739%	70.879%
fork	IQ2_NL	2.8046	2.91	17.6%	1	11.632334 ± 0.084906	0.432011 ± 0.001450	2.716625	18.216%	71.111%
fork	IQ3_NL	3.6722	3.81	23.0%	1	8.317650 ± 0.056391	0.078364 ± 0.000294	0.501990	7.755%	86.045%
both	Q4_K	5.8664	6.08	36.7%	90	7.803435 ± 0.052058	0.006809 ± 0.000026	0.044414	2.247%	95.761%
both	Q5_K	6.3562	6.58	39.7%	90	7.775161 ± 0.051827	0.003110 ± 0.000012	0.019825	1.500%	97.040%
both	Q6_K	8.2158	8.51	51.4%	90	7.748767 ± 0.051608	0.000528 ± 0.000002	0.002775	0.677%	98.633%
both	Q8_0	8.5058	8.81	53.2%	0	7.747085 ± 0.051616	0.000388 ± 0.000001	0.002175	0.598%	98.809%

Prompt Processing + Text Generation Benchmarks

Impl	Model	Size	Params	Backend	Threads	Test	T/S
standard	nemotron_h 9B IQ1_M - 1.75 bpw	4.51 GiB	8.89 B	MTL,BLAS	12	pp512	630.60 ± 0.78
standard	nemotron_h 9B IQ1_M - 1.75 bpw	4.51 GiB	8.89 B	MTL,BLAS	12	tg128	54.70 ± 0.08
standard	nemotron_h 9B IQ1_M - 1.75 bpw	4.51 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	98.43 ± 0.93
fork	nemotron_h 9B IQ1_M - 1.75 bpw	3.15 GiB	8.89 B	MTL,BLAS	12	pp512	625.29 ± 6.73
fork	nemotron_h 9B IQ1_M - 1.75 bpw	3.15 GiB	8.89 B	MTL,BLAS	12	tg128	55.86 ± 0.15
fork	nemotron_h 9B IQ1_M - 1.75 bpw	3.15 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	99.04 ± 0.92
standard	nemotron_h 9B Q2_K - Medium	4.66 GiB	8.89 B	MTL,BLAS	12	pp512	630.75 ± 5.27
standard	nemotron_h 9B Q2_K - Medium	4.66 GiB	8.89 B	MTL,BLAS	12	tg128	56.57 ± 0.26
standard	nemotron_h 9B Q2_K - Medium	4.66 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	102.88 ± 0.51
fork	nemotron_h 9B IQ2_NL - 2.5 bpw	2.90 GiB	8.89 B	MTL,BLAS	12	pp512	616.08 ± 10.77
fork	nemotron_h 9B IQ2_NL - 2.5 bpw	2.90 GiB	8.89 B	MTL,BLAS	12	tg128	57.86 ± 0.61
fork	nemotron_h 9B IQ2_NL - 2.5 bpw	2.90 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	98.73 ± 1.88
standard	nemotron_h 9B Q3_K - Medium	5.00 GiB	8.89 B	MTL,BLAS	12	pp512	600.77 ± 10.07
standard	nemotron_h 9B Q3_K - Medium	5.00 GiB	8.89 B	MTL,BLAS	12	tg128	51.16 ± 1.08
standard	nemotron_h 9B Q3_K - Medium	5.00 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	92.00 ± 0.61
fork	nemotron_h 9B IQ3_NL - 3.5 bpw	3.80 GiB	8.89 B	MTL,BLAS	12	pp512	600.65 ± 21.27
fork	nemotron_h 9B IQ3_NL - 3.5 bpw	3.80 GiB	8.89 B	MTL,BLAS	12	tg128	35.53 ± 0.50
fork	nemotron_h 9B IQ3_NL - 3.5 bpw	3.80 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	66.38 ± 0.58
standard	nemotron_h 9B Q4_K - Medium	6.07 GiB	8.89 B	MTL,BLAS	12	pp512	604.89 ± 0.46
standard	nemotron_h 9B Q4_K - Medium	6.07 GiB	8.89 B	MTL,BLAS	12	tg128	47.16 ± 0.25
standard	nemotron_h 9B Q4_K - Medium	6.07 GiB	8.89 B	MTL,BLAS	12	pp1024+tg1024	83.79 ± 0.76

Actual File Structure

What each recipe actually wrote to disk, across the 122 quantizable tensors:

Impl	Requested	Realised composition
naive	IQ1_M	78 × IQ4_NL / 27 × IQ1_M / 12 × Q4_0 / 4 × IQ2_XXS / 1 × Q8_0
naive	Q2_K	90 × Q4_0 / 27 × Q2_K / 4 × Q3_K / 1 × Q8_0
naive	Q3_K	61 × Q4_0 / 27 × Q3_K / 24 × Q5_0 / 5 × Q5_1 / 4 × Q4_K / 1 × Q8_0
naive	Q4_K	75 × Q5_0 / 31 × Q4_K / 16 × Q8_0
naive	Q5_K	75 × Q5_1 / 31 × Q5_K / 16 × Q8_0
naive	Q6_K	91 × Q8_0 / 31 × Q6_K
fork	IQ1_M	78 × IQ2_NL / 27 × IQ1_M / 12 × Q4_0 / 4 × IQ2_XXS / 1 × Q8_0
fork	IQ2_NL	120 × IQ2_NL / 1 × IQ3_NL (token_embd) / 1 × Q5_1 (output)
fork	IQ3_NL	121 × IQ3_NL / 1 × Q5_1 (output)

The Q2_K row is the clearest illustration of the problem: 74% of the tensors in a "2-bit" model are stored at 4.5 bpw. The Q4_K and Q5_K rows show the fallback resolves upward to Q5_0 (5.50 bpw) and Q5_1 (6.00 bpw) against a Q4_K/Q5_K nominal of 4.50/5.50 bpw, inflating those builds by roughly 1.4 and 0.9 bpw respectively.

4. Logs and Test Results

Metric	Source
BPW, quant size, fallback counts	*/logs/*.log
PPL, KLD, Δp, top-1	*/scores/*.ppx
Tensor geometry & model metadata	*/scores/*.md
Per-tensor activation statistics	*/scores/*.itx