inclusionAI

/

Ling-3.0-flash

like 352

Follow

inclusionAI 2.69k

https://huggingface.co/gopi87/Lingflashfixclaudecode/tree/main

i have manually fixed the issues with claude code

Thank you so much for your contribution! Since we don't currently provide official support for llama.cpp, testing under this framework hasn't been as thorough as we'd like. We truly appreciate your efforts in identifying and fixing the issue — we'll make sure to merge your fix when we release the corresponding version in the future. Thanks again for your help!

@gopi87 thank you!

I'm curious, how did you manage to use AI (Claude in this case) to fix a Jinja template without it stopping itself when outputting one of the stop tokens itself?

Sometimes I'll use Qwen to try to fix a Jinja template but when it comes across lets say </think> it stops reasoning because it thinks that token should stop its own chat.

@gopi87 thank you!

I'm curious, how did you manage to use AI (Claude in this case) to fix a Jinja template without it stopping itself when outputting one of the stop tokens itself?

Sometimes I'll use Qwen to try to fix a Jinja template but when it comes across lets say </think> it stops reasoning because it thinks that token should stop its own chat.

Much of the fixes on this stuff get's done using the frontier models (like Fable, or GPT 5.6 Sol), at least as far as I can tell anecdotally from seeing PRs. They don't use the same chat templates so parsing a </think> (or anything else that shows up in jinja templates) won't (souldnt) really take effect.

@gopi87 thank you!

I'm curious, how did you manage to use AI (Claude in this case) to fix a Jinja template without it stopping itself when outputting one of the stop tokens itself?

Sometimes I'll use Qwen to try to fix a Jinja template but when it comes across lets say </think> it stops reasoning because it thinks that token should stop its own chat.

hi i dont know what i did but iwas able to use the model in the claude code upto 150k but after that i am getting the "/n" the chat template i took it from here onguy did some modification then i used old qwen tepmlate as a model to fix it one sec i will add my template and the source where i get it from

@Kayook the person who fixed the temlate and you see everythink here https://huggingface.co/gopi87/Lingflashfixclaudecode

curently i am focusing on fix the "/n" repation after 150k context fill

Also had issues with "/n" spam and model hanging when trying to tool call
I went in different direction - i've seen people fix the tool call issue through modifying parser code in llamacpp, can be seen here - https://github.com/ggml-org/llama.cpp/pull/26608
So i've sent my local AI on a mission to reuse code for tool call parser issues + attempt to fix the /n spam at the same time. Currently its working with original jinja template, didn't encounter /n spam yet (but tested for only ~5hours at 100k context depth)

UPDATE: Nope fix didn't help, still getting /n spam sometimes

Also had issues with "/n" spam and model hanging when trying to tool call
I went in different direction - i've seen people fix the tool call issue through modifying parser code in llamacpp, can be seen here - https://github.com/ggml-org/llama.cpp/pull/26608
So i've sent my local AI on a mission to reuse code for tool call parser issues + attempt to fix the /n spam at the same time. Currently its working with original jinja template, didn't encounter /n spam yet (but tested for only ~5hours at 100k context depth)

UPDATE: Nope fix didn't help, still getting /n spam sometimes

i have burned 100$ in fable but still not able to fix the issues

I tried to reproduce the issues in this thread on my own machine (llama.cpp PR #26608, commit 3a0124f, Q4_K_M gguf, 128GB unified-memory box). There are actually 3 different problems mixed together here. That's why the fixes look like they don't work.

1. "Doesn't work with Claude Code"

Most likely cause: the two llama.cpp grammar bugs @gopi87 found (maxLength > 2000, and tools with empty parameters → failed to parse grammar). Claude Code's tool set triggers both on every request. These are llama.cpp core bugs (#25923 / #25927) and they are fixed now — I ran gopi87's two curl repros on the current PR branch, both pass. If you are on an old build or a fork, rebuild. Not a template problem.

2. Broken / hanging multi-argument tool calls

This was a real bug in PR #26608. Fixed in commit 0266ebca (Aug 7). I tested on a build after that commit, stock template + --jinja: 3-arg calls, optional args, multiline values, multi-turn tool use — all parse correctly.

@Kayook this is also why your test was confusing: the parser fix and the \n spam are different problems. The parser fix works. The spam has a different cause, see below.

3. The JSON template

I tested it. The model was not trained to output JSON arguments. Without grammar constraints it ignores the instruction and outputs <arg_key>/<arg_value> tags every time (6 out of 6 in my test). The JSON template only works because llama.cpp's grammar forces the format. The stock template already works on current builds, so I would keep the stock one.

There is one real bug in the stock template though: {% set _args = tc.arguments %} should be {%- set ... %}. It leaks a newline + 16 spaces after the function name, and the parser currently depends on that newline. One character fix.

4. The \n spam after 150k

I can reproduce the degradation, on a different stack than yours (PR #26608 instead of the prometheusAIR fork, Q4_K_M instead of Q8_0): generation falls into an exact repetition loop starting around 152–154k tokens (temp 0.6). In chat traffic this shows up as \n spam, since \n is the most common token there. Same test at 76k depth, same generation length: completely clean. So it's caused by context depth, not by sampling and not by the template. This matches gopi87's "works up to 150k" almost exactly. I could not reproduce spam at ~100k — my runs at 76k/117k/120k/130k were all clean.

Why people get different results: it depends on the gguf. AtomicChat and bloomer010 ggufs have context_length=131072, and llama-server caps the slot at that value — those users can never reach 150k at all. The prometheusAIR gguf has context_length=262144, so those users run straight into the bad zone. Also note Claude Code's token counter is not the same as the server-side position, which hides all this.

What works today: PR #26608 at or after 0266ebca + bloomer010 gguf + stock template + --jinja + context ≤ 131072. Don't override context_length to 262144. (AtomicChat ggufs currently don't load on the PR branch at all — old tensor names ssm_f/ssm_g, and block_count=42 with nextn_predict_layers=1 points to an MTP layer that is not in the file.)

I'll keep digging into the \n spam after 150k and post here when I find something.

Thanks @gopi87 — the grammar bug analysis was correct.

thanks @nehcoahz the model is really good on agent task and code generation and its better then qwen 3.6 27b. i think its gonna pair with coming qwen 3.8 27b & since its an moe its easy to use it. i keep on digging into the "n/" too will share what i find it .

i thnk i fixed the issues of "/n" repeation did tested above 185k its going fine i have attaching the found not sure if this is a fix or not

https://huggingface.co/gopi87/Lingflashfixclaudecode/blob/main/fix_documentation.pdf

echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp-bailing/build/bin/llama-server
--model /mnt/nvme/Ling-3.0-flash-Q8_0-00001-of-00004.gguf
--chat-template-file /mnt/nvme/bailing-tool-fix-v2.jinja
--tensor-split 1.2,1.8,0.45,0.35
--n-cpu-moe 42
-c 262144
--batch-size 8000
--ubatch-size 8000
--numa distribute
--no-mmap
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja 2>&1 | tee ~/llama-debug2.log

i used this

Hi @gopi87 , thanks for reporting this and for the workaround repo — this thread saved me a lot of time when I hit the exact same thing.

I've also been running this and got the same newline flood around ~150K context. I managed to capture a failing prompt that reproduces the collapse deterministically (same output byte-for-byte across seeds and reruns), which made it possible to A/B test the suggested fixes properly.

What didn't work

• repeat_penalty 1.10: replayed my failing prompt with it — still collapses. At the collapse point \n sits at p≈1.0 with the runner-up 5–8 nats behind; a multiplicative penalty barely dents that gap.
• Prompt/template tweaks: some history-reformatting variants did "heal" it, but a ~100-token perturbation flips it right back. Not reliable.

What worked

1. frequency_penalty: 0.1 — verified on llama.cpp, vLLM and SGLang. It's additive per occurrence, so the penalty accumulates: after ~50–80 newlines it overcomes the gap and the model escapes. 0.05 is not enough; 0.1–0.3 all work.

2. DRY sampler with \n removed from sequence breakers — the best option if you're on llama.cpp:

--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 \
--dry-sequence-breaker ':' --dry-sequence-breaker '"' --dry-sequence-breaker '*'

The gotcha: llama.cpp's default DRY breakers include \n, which resets the match at every newline — so default DRY is blind to exactly this failure mode. With \n removed, my previously-always-collapsing session ran 26 rounds clean up to 251K context. It prevents the collapse rather than escaping it after the fact.

Happy to share more details on the repro harness if anyone wants to dig further.

Hi @gopi87 , thanks for reporting this and for the workaround repo — this thread saved me a lot of time when I hit the exact same thing.

I've also been running this and got the same newline flood around ~150K context. I managed to capture a failing prompt that reproduces the collapse deterministically (same output byte-for-byte across seeds and reruns), which made it possible to A/B test the suggested fixes properly.

What didn't work

• repeat_penalty 1.10: replayed my failing prompt with it — still collapses. At the collapse point \n sits at p≈1.0 with the runner-up 5–8 nats behind; a multiplicative penalty barely dents that gap.
• Prompt/template tweaks: some history-reformatting variants did "heal" it, but a ~100-token perturbation flips it right back. Not reliable.

What worked

1. frequency_penalty: 0.1 — verified on llama.cpp, vLLM and SGLang. It's additive per occurrence, so the penalty accumulates: after ~50–80 newlines it overcomes the gap and the model escapes. 0.05 is not enough; 0.1–0.3 all work.

2. DRY sampler with \n removed from sequence breakers — the best option if you're on llama.cpp:

--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 \
--dry-sequence-breaker ':' --dry-sequence-breaker '"' --dry-sequence-breaker '*'

The gotcha: llama.cpp's default DRY breakers include \n, which resets the match at every newline — so default DRY is blind to exactly this failure mode. With \n removed, my previously-always-collapsing session ran 26 rounds clean up to 251K context. It prevents the collapse rather than escaping it after the fact.

Happy to share more details on the repro harness if anyone wants to dig further.

thanks for sharing this i will check the dry sampler.

echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Ling-3.0-flash-Q8_0.gguf
--tensor-split 1.2,1.8,0.45,0.35
--n-cpu-moe 42
--numa distribute
--no-mmap
--reasoning-preserve
-c 262144
--dry-multiplier 0.8
--dry-base 1.75
--dry-allowed-length 2
--dry-sequence-breaker ':'
--dry-sequence-breaker '"'
--dry-sequence-breaker '*'
--batch-size 8000
--ubatch-size 8000
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja

this is works very well with recent build

echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Ling-3.0-flash-Q8_0.gguf
--tensor-split 1.2,1.8,0.45,0.35
--n-cpu-moe 42
--numa distribute
--no-mmap
--reasoning-preserve
--override-kv bailingmoe3.context_length=int:262144
-c 262144
--dry-multiplier 0.8
--dry-base 1.75
--dry-allowed-length 2
--dry-sequence-breaker ':'
--dry-sequence-breaker '"'
--dry-sequence-breaker '*'
--batch-size 8000
--ubatch-size 8000
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja