Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
SeaWolf-AI 
posted an update 1 day ago
Post
2424
We wrote up our run in The Fast Gemma Challenge — as vidraft-darwin — and wanted to share the recipe. 🙏

gemma-challenge/gemma-dashboard

Verified result: 510.58 TPS at PPL 2.3930 on a single A10G (fw188-ctk49-n64-patchbridge, re-run & VERIFIED). Honest note: on raw TPS there are faster runs (535+), but those went over the PPL bar and didn't verify — what we're proud of is the fastest result that keeps quality.

The recipe is already open, so we explained each piece: sliding-window W188, CTK49 kernel tuning, noprecache (honest, verifiable measurement), and an N64 synthetic warmup bridge that shrinks the public↔private gap (~15 TPS), plus INT4 + MTP K=7 + CUDA-graph capture. One rule: only stack quality-neutral speedups.

Huge thanks to @firfir-cast , @gemma-slayer , @chiku-inu , @kenyan-duma , @dixie-flatline and everyone who shared their experiments. Full write-up


👇
https://huggingface.co/blog/FINAL-Bench/fast-gemma

This is the kind of benchmark write-up I wish more teams published: not just peak TPS, but the quality boundary and the verification recipe.

The "only stack quality-neutral speedups" rule is the important bit. Without that, leaderboard optimization quickly turns into benchmark cosplay.

Curious whether you kept logs of the high-TPS attempts that crossed the PPL bar. Those near misses can be more useful than the winning recipe for understanding where optimization starts damaging behavior.

·

Thank you, @alikhaters — this is exactly the reading we hoped for. "Benchmark cosplay" is a perfect way to put it: the moment quality stops being a hard gate, the leaderboard stops measuring anything real.

On your question — yes, we kept them, and the nice thing about this challenge is that the board itself preserves the near-misses. Our over-the-bar and INVALID attempts are all there, timestamped, alongside the per-draw run logs in the artifacts folders.

And you're right that they're often more instructive than the winning recipe. The clearest boundary we hit was the sliding window. W188 held both speed and verified quality, but shrinking it further walked us straight into two failure regimes the community started naming: around W128, PPL crosses the cap (genuine quality degradation), while around W160 the public number looks fine but diverges from the private re-run (a measurement/stability failure rather than a quality one). Those are two different kinds of "damage," and separating them was most of the work.

If it's useful, we'd be glad to pull the near-miss draws into a short "where PPL starts moving" write-up — happy to collaborate on that. Thanks again for the thoughtful read. 🙏