Per-tensor layout maps for GGUF quantization

As we all know, when it comes to creating llama.cpp quantizations, there's a lot of unknowns and variability. The one thing a lot of us can agree on is that the upstream code is a little long in the tooth - that's not a slight at the developers or anyone there, but it's a tricky area to maintain and critical for every quantization that goes out the door.

The Backstory

A couple years ago, around the time MoE models started getting more mainstream, many found that the existing algorithm wasn't doing a good job handling them. The introduction of more than 8 experts (llama-quant.cpp only has special cases for mixtral at exactly 8 experts), and the realization that the shexp tensors were pretty sensitive despite being incredibly small, led to a need for extra work to come in. You may know some of these methods, and I maintain a copy of llama-quant.cpp with my own changes for better handling these MoE models. This has worked pretty well for a while, but as more models emerge and I see more tensor layout methods perform better and better, it was clear I needed to take another swing at it.

The Problem

The main issue with both upstream's code and my changes is that they're always going to be model-agnostic. Yes, they're shape dependent, and use clever methods to check how deep in a model we are, like use_more_bits which checks if the current layer is in the first 1/8ᵗʰ, last 1/8ᵗʰ, or every third layer in the middle, in order to do what some would call "dynamic" assignment of higher quant values, but it can only go so far. It also checks for attention layers, bumping some more than others, crushing some tensors, all in an effort to round out to a target bits per weight by assigning extra information to where it matters most as determined by some tests several years ago.

Data gathering

Enter what I like to call "Claude Code running for days unleashed on my Framework Desktop", or more succinctly, my quantization experiments.

The basic question is this: does a method exist of learning the relative sensitivity of a given tensor in a model, and if it does, can it be generalized? I worked with Claude on a framework for testing, it wrote up some scripts, and I ran them for the next ~96 hours producing and evaluating over 1000 quants across various sizes to gather as much information as possible.

I'll take a second at this time to shout out Framework who sent me one of their AMD AI Max+ 395 128GB desktops, they didn't have to do this, and they asked nothing in return, but since I've been using it for a bit now and it was extremely pivotal to have an idle powerful machine to run these tests on, I figured this was a good time to throw them a huge thank you

The first step was to attempt a wide sweep of "degrade-one" runs, where all tensors are set to q8_0 except for one which is set to q2_k, and "upgrade-one" runs, where all tensors are set to q2_k except for one which is set to q8_0. These tests were performed against Qwen3.5-0.8B and Qwen3.5-4B (more on that later). Upgrade-one turned out to be less useful than anticipated, most runs were just noise, so the degrade-one data drove most of the following data.

Followers of Thireus may recognize this method from their GGUF-Tool-Suite, while I've never looked into it deeply I know it similarly tries to crush individual tensors in order to yield their theoretical "sensitivity".

There were additional validation tests run against several other models to ensure that trends from Qwen carried across to other families, including Gemma 4, Granite 4.2 (3B/8B/30B), Laguna, Ling, Muse, and Ornith. These were not used extensively, but just to ensure that there was nothing special about how Qwen behaved with depth and tensor type.

What the data found

First, a quick note on measuring, since every number from here on uses it: KLD is the KL divergence between a quant's token probabilities and the bf16 model's, measured with `llama-perplexity --kl-divergence` on wikitext-2-raw at a context of 512. Lower is better. The main sensitivity sweep ran on a lower number of chunks so it could cover the 1000+ configs, and every head-to-head against my old heuristic later in the post is 100 chunks of that set, run again at 300 chunks when the gap was too close.

1. token_embd dominates the sensitivity scale (champions of "q8_0 embeddings or bust" rejoice, you have been vindicated), costing about 8x the worst single weight tensor at 0.8B and 16x at 4B. That said, it gets a large majority of its performance back from q4_k, so on smaller bitrate models it's not necessarily going to be always at q8_0, but it's always going to receive a healthy bump. The old q8_0 embedding versions are gone, replaced with a suggestion to just grab the next highest size, because that'll benefit you more.
2. Depth is a U shape of sensitivity. This is probably the least surprising overall finding, use_more_bits that I mentioned earlier (and several other places in llama-quant.cpp) explicitly bump early and late because even several years ago that was the case.
3. Per bit, small attention projections are the most sensitive. In general, attn_v, attn_output, ffn_up, and ssm_out are extreme standouts on sensitivity. Interestingly, ffn_gate is basically never worth any extra bits
4. Relative damage measured from pure runs for each type, documented in the prior.json table

The Solution

Okay, so we have all this great data, now what? I had 2 main goals: the first, of course, have the best possible layout for performance per bit. The second, try to bring back the meaning behind the names like Q4_K_M, Q3_K_S, IQ2_XXS etc etc.. These kind of got quite bloated somewhere along the way. If a file that starts with Q3_K_ is mostly non-Q3_K tensor types and sits above 5 bits per weight, something has gone wrong..

With Claude's help, the solver was created with the goal of taking a model shape, the prior.json, some basic data about ggml block sizes, and the target quant type as input, and produce the "optimal" output.

The solver is allowed to crush weights in addition to bumping, but interestingly, K quants almost never benefit from a crush, likely because the differences in K-quant types are just too large. IQ-quants benefit from a more granular slope of change, so there are times where a weight will be crushed to save some bits for a bump, but the damage of going down a K-quant level is too extreme to justify.

What we end up with

Like I mentioned, I wanted in part to bring some meaning back to the quant-names, so we have a couple of rules about how many of the model's body's tensors can be bumped: _S must be 90% the named tensor type, _M 70%, and _L is allowed to go up to 50% bump. You can find these tiers in my posted generate.py. This means if you see Q3_K_S, you know it will be 90% Q3_K tensors inside. This was in part also to avoid an observed "bpw collapse", where models targeting different outcomes all ended up basically the same size. This also means that some file sizes will change, for example Q3_K_M on Qwen3.5-4B gets 15% smaller, but since the bits are being more appropriately spent, this should mean that if you look for the same file size, you should be getting better performance than you were before. In some tests for example, Q4_K_M ended up having a slightly worse overall KLD, but came in just over 5% smaller, meaning per bit it was a better model.

I also decided to force K-quants to not use IQ-tensors, this is largely for compatibility, but in the future I may consider mixed types. IQ-quants are not restricted in this way and can use any types that the solver wishes.

Another thing you'll see in generate.py is a definition of "TINY_PARAMS", this is to force all extremely small tensors to F32 for speed and quality.

But does it generalize?

Okay, great, so we now know the answer: there does exist a method of learning the relative sensitivity of a given tensor in a model. But now came the big question: does it generalize? Because if it doesn't, then sure we have the best possible Qwen3.5-0.8B and Qwen3.5-4B quants at each type, but I can't be running days of compute each time a model comes out. So off to the tests we went: Qwen-3.6-35B-A3B Gemma 4 E4B, granite-4.2-8b, Ling 3.0 tiny and flash, MiniCPM5-2B, Muse Glimmer, and DeepSeek-V2-Lite (for an MLA model) all went through the wringer.

For each of them, we used the solver/generator to create a recipe, quantized with it, then quantized with my own llama-quant.cpp fork, and measured the head-to-head KLD. It should be noted: the solver didn't win. There were a few tweaks to do, granite's and MiniCPM's dense all-attention layout (I'd only been measuring on the hybrid attention Qwen models), and DeepSeek-V2-Lite all failed on the first pass. Thankfully, all of those were solved with a few tweaks to the generator and its prior (the body prior needed a per-class table, and the embedding rule needed to look at the table's share of the file) and overall we now seemed to have a generalizable solution!

It's worth noting that below ~3 bits, dense models basically come out as a tie to the old method, so this isn't always a huge win across the board, but it is consistent:

In fact, both my MiniCPM5-2B-GGUF and Gryphe_Pantheon-Reasoning-26B-A4B-1.1-V2-GGUF releases use this method.

Now you may be saying, "wait, if it failed for some models, sure those are fixed, but how do you know it won't fail for others?" and the answer is: I don't. But, what I do know, is how to test for it.

The "canary" test

Going forward, every new model shape (keyed on architecture name and an ordered list of tensor names and dimensions, as well as the generator version so that updates to the method are re-evaluated) goes through a test first. We quantize to Q4_K_M, Q3_K_M, and IQ2_XS (or whatever the smallest target quant is going to be) using both the map and my changes in llama.cpp. We then calculate the KLD of all 6 of those models, and plot them on a line of "KLD per bit". If any of the mapped quants fall above the curve (by more than noise), we reject the map and fallback to my own heuristic.

These canary numbers will be posted alongside the model for you to read in the Per-tensor layouts section, like here: https://huggingface.co/bartowski/MiniCPM5-2B-GGUF#per-tensor-layouts

On my pipeline, it looks like this:

I am keying off of the shape because tunes should behave identically to the base model when it comes to sensitivity, much like imatrix does. If this ever turns out to not be the case, I'll re-evaluate.

While testing, MiniCPM5-2B actually failed my canary tests (mentioned above), and my pipeline correctly fell back to my original heuristic instead of pushing on.

What ships

The main thing you'll notice different is that the _L variants now carry different meaning than they used to. Before, Q2_K_L, Q3_K_XL, Q4_K_L, Q5_K_L, and Q6_K_L all just meant "whatever the previous quant was, but with q8_0 as the embedding and output". Now, Q4_K_L and Q6_K_L remain these as "budgets" of tensor types, and gone are Q2_K_L, Q3_K_XL, and Q5_K_L. I believe this is the superior method: if you were going to reach for a higher tier embedding, instead reach for just the next rung up the ladder. It also gives access to a "7.2-7.4 bpw" with Q6_K_L and avoids weird situations where you'd see Q3_K_XL being larger than Q5_K_M because the embeddings were massive (looking at you, gemma-4-E4B...)

Besides that, hopefully you'll notice marginal improvements across the board for all quants going forward, and a more even distribution of file sizes! And remember, file sizes may have changed, if before you could fit Q4_K_M maybe now you can fit Q4_K_L, take a look and experiment, and help each other out!

What's next?

First, I'll run the full map against Qwen3.8-27B, gather KLD stats, then run KLD against my original upload. If the changes are large, I'll push them with the relevant info. Other old models will likely not be getting similar treatment unless demand is there.

Second, there are a couple gaps: Qwen3.8-Flash-Next has a giant PLE n-gram table which my script is not prepared to handle. It shouldn't be terribly difficult, but wasn't a priority. It will be the next step of my investigation (especially in light of DeepSeek-V4.1-Flash also having an n-gram table...). Hy4-preview also has a strange gated MLA (q_lora_rank 2048 and kv_lora_rank 512) with a sparse attention, which I'm also not prepared for. That said, the model is so large I wasn't prepared to make it at all.. But maybe if I can sneak in a few quantization creation speedups PRs I can consider models of that size in the future!

Limits of this work

• KLD was run on wikitext only, and while I believe that to be fine, it may be worth a pass with another dataset to triple check.
• Prior was only fit using 2 small Qwen models (and later granite), but as seen in testing seems to have generalized.
• Above ~Q5 the bumps are almost purely noise, I make the assumption that the logic still carries, but it's difficult to prove.
• Finally, this has not yet been tested wide (I've only released two models with it so far) but since canary testing will continue, I'm confident in the quality of models going forward.

Final words

I guess all I really have to say here is thanks for reading and as always thank you so much to the llama.cpp developers and community, without whom I'd basically have nothing to do!