ManniX PRO

You need to think more "out of the box" and have a boarder vision.
The evaluation harness is limited and exercise a single use case.

The only number that matters is the reduction at P, Step 0.
If you consider also S per step, it's not going to tell you anything really valuable.

This is a fixed, mocked eval harness that runs a static number of steps.
It has its own purpose which is exercising and validating the API and the engine.
Doesn't really reflect a real use case; maybe there are similar ones but it's an edge case.

Every agent in the harness have a very small prefix (P=1216) which is most often not true.
Unless you are using agents to do some very basic and trivial things, hardly a real use case (most probably you don't need an agent for this kind of stuff).

In a real use case the P is what matters most of the time; in my claude-hooks consultants skill the P is gigantic: 60-200K to describe the project, provide the tools, the AST, list of the files to inspect, what was done, test results, etc etc.

Then the agent will start its work and you have no idea how many steps will take, how long will be the output, how much thinking will be needed. Ever spawned agent will have, in non synthetic tasks, most likely a very different behavior. So you can't compute and forecast a ratio of the savings due to the shared P and involve S and the steps.

The capacity lever is at P: let's frame a realistic scenario, 128k of "P" plus max 128k for "S".
If you have space for 1M tokens on your GPU and you need to run agents without PolyKV you can run 4 of them.
With PolyKV you can run 7 agents and they will be faster overall plus the context compaction gotcha.

The context compaction is another big win: without PolyKV every agent needs to be instructed to preserve the P and they all need to process its own P. They could easily mess it up and remove critical information or just make mistakes.
With PolyKV the P is static and it's not reprocessed by the agents, doesn't have a chance to get messed up.

You are really wrong when you say that the fleet size is not a lever: it's exactly the opposite.
The floor tps scheduler is a killer feature just like the shared P.
That's why I uploaded so many reports with different models and different floor tps.
Each combo model/GPU needs the right floor tps otherwise it starts crapping out and the wall time will increase, the quality will decrease. That's the value of the test harness as it is configured now.

You can't quite precisely predict the load of every agent, if you choose a static value that's it. You are bound to it.
If you are running multiples instances of any other inference engine without the floor tps scheduler you have either to under or over subscribe the number of agents.
Unless you put on top a complex custom orchestrator which needs to be managed.

Since opencoti-llamafile with the PolyKV API knows the actual load on the GPU it can auto-balance with just the right tps floor value for the model, independently of any other instance running or not running in parallel.
If there's no other load can scale up the workers up or above the available slots, your decision.
In case there's load the floor tps scheduler will restrict the amount of agents that can be spawned automatically.
I still need to improve concurrency and avoid over subscription at start but the logic is simple and should be clear.

Abou the chart: I will probably replace the graph of tps at step as it's redundant to the tps with context and less granular.
But as I said a "cell reduction" graph as you describe I don't think that really make sense.
Have to think a bit more about it on how to frame it.

You can start wiring it now and test yourself.
The new c4 release provide a very rich and sophisticated API for PolyKV, including a floor-tps scheduler that allows automatic workers allocation based on engine capacity.
Should be properly documented but it's just been done and I didn't have time to double check.

There is also an evaluation benchmark test harness, package-courier, and a few benchmark runs.
You can use the test harness as a reference implementation of the API in Langchain/Langgraph.

You can simulate multi-turn tool usage with it and track the gen tps over turns, there's even a specific graph in the report.
Didn't have time yet to test it with more than 10 turns, context per-agent is small at n=10 and the tps increase over turns due to that (you need really a massive context to see the tps decrease and opencoti-llamafile is slower with small context sizes).
If you test it let me know how it goes and if the documentation is clear enough!

The ratio grows with the number of agents since the shared prefix is stored once instead of N times: 6.9× is the measured KV-cell reduction at N=8 agents (with a ~875-token shared prefix and short private suffixes).

Projected onto a fixed KV buffer, that geometry fits about 306 agents pooled vs about 9 naive before exhaustion. The more sub-agents attached to a pool, the closer you get to the prefix/suffix ceiling.
The suffixes always diverge, so there's no memory saving there — the batched-decode gains (~+14% aggregate at N=8) come from the smaller cache footprint, not from suffix sharing.

llama.cpp's built-in cache reuse (--cache-reuse) is a different axis: a slot reusing its own previous prompt across turns. It doesn't share cells across concurrent sequences — that's exactly what PolyKV adds, and the two compose.

I would wait before wiring it in, the mechanics are there but they are raw.
I'm adding a proper API control and Langchain/Langgraph extension, actually a very sophisticated one, which will make it easier to use it and way more powerful.

About your question: the prefix is immutable.

If you want to change the prefix you will have to spawn a new pool and new sub-agents.
There's no problem adding a new mcp server or tool def in the suffix but it will live inside that sub-agent session only.
(Which anyway makes sense as if it was something needed by all sub-agents it would be part of the prefix)

PolyKV is exactly this, sharing an immutable prefix and run a pool of sub-agents that shares it.
This allows parallelizing tasks for a single agent/session very efficiently.

Really depends on the specific use case; the headline is that with DCA you can extend any context and overcome the model limitation.
It's not only useful to get to 1M context, which can get damn slow even with a MoE on an RTX6000, but also to get up to an acceptable range with a model trained for a small context.

There's also the KV-window which it's truly novel, not to be under-estimated considering the crazy GPU and memory prices. A modern system with full PCIe 5.0 x16 and DDR5 can hide behind computation up to 3-4GB, depending on the model attention mechanism, and keep almost full gen tps speed.
Offloading model layers to RAM, the classic llama.cpp method, tanks the decoding speed immediately, no matter if is by a small amount or not.

If you are working with agentic framework coding SharedKVPool is the headline.
The auto-tier is a boot selection algorithm, doesn't have much to do with them; it's an helper (pretty sophisticated with probing and micro-bench at boot) to automatically enable the KV-window in conservative mode and/or select a K/V type and strategy that fits the available VRAM.

The prefix with PolyKV is what is shared between the sessions; it doesn't change. Same quantization for everyone, if it changes it's done only once.
The sessions in the same pool share the same prefix and only the suffix changes, that's what diverges and always.
The win is by definition in prefill and memory usage but also on gen tps (less overhead) and multi-turn re-query.

Made a benchmark and updated the docs:
https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/USAGE.md#34-polykv-sharedkvpool-multi-agent-shared-prefix

It's really a game-changer and a killer-feature for agentic frameworks that can drive it, both in capacity and performances.