Zenyx V3 Base (1.5B Mixture-of-Experts)
Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.
This is a BASE model — it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue (
"The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.
Current checkpoint: step 73,200 · 45.3B tokens seen
Model Architecture
- Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 active per token, with a Sinkhorn transport-based gate.
- Multi-head Latent Attention (MLA): compresses the KV cache into a low-rank latent subspace, cutting HBM bandwidth and memory footprint.
- Hyper-Connections: Sinkhorn-normalised residual routing for gradient stability at scale.
- Multi-Token Prediction (MTP): one auxiliary prediction head during training.
- Context: pretrained at up to 4,096 tokens (progressive 2,048 → 4,096). YaRN and RoPE scaling factors are precomputed so context can be extended at inference time beyond the trained length.
| Total parameters | ~1.5B |
| Active parameters / token | ~0.4B |
| Layers | 16 (2 dense + 14 MoE) |
| Hidden size | 1,536 |
| Attention heads | 12 (head dim 128) |
| Vocabulary | 129,280 |
| Precision | bfloat16 |
Benchmarks — checkpoint step 73,200 (45.3B tokens)
All tasks are evaluated with the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is taken as the answer. Nothing is generated and no output parsing is involved, so the numbers do not depend on instruction-following ability. 0-shot, full evaluation sets, no subsampling.
acc_norm normalises each continuation's log-likelihood by its length in characters,
which removes the bias toward short answers; it is the headline metric wherever the
task has candidates of differing lengths.
| Benchmark | acc |
acc_norm |
Random | Δ | n | Description |
|---|---|---|---|---|---|---|
| HellaSwag | 29.86% ± 0.46 | 32.72% ± 0.47 | 25.0% | +7.7 | 10,042 | Commonsense sentence completion |
| ARC-Easy | 50.17% ± 1.03 | 46.09% ± 1.02 | 25.0% | +21.1 | 2,376 | Grade-school science questions |
| ARC-Challenge | 20.82% ± 1.19 | 25.94% ± 1.28 | 25.0% | +0.9 | 1,172 | Hard grade-school science questions |
| PIQA | 60.88% ± 1.14 | 60.23% ± 1.14 | 50.0% | +10.2 | 1,838 | Physical commonsense reasoning |
| WinoGrande | 49.25% ± 1.40 | — | 50.0% | -0.7 | 1,267 | Pronoun resolution / coreference |
| OpenBookQA | 17.80% ± 1.71 | 28.40% ± 2.02 | 25.0% | +3.4 | 500 | Elementary science with open book |
| BoolQ | 60.83% ± 0.85 | 62.11% ± 0.85 | 62.2% | -1.4 | 3,270 | Yes/no reading comprehension |
| SciQ | 76.00% ± 1.35 | 68.50% ± 1.47 | 25.0% | +51.0 | 1,000 | Science exam questions with support |
| LAMBADA (OpenAI) | 26.57% ± 0.62 | — | 0.0% | +26.6 | 5,153 | Long-range last-word prediction |
| MMLU (5-shot) | 25.26% ± 0.37 | — | 25.0% | +0.3 | 14,042 | 57 subjects of academic knowledge |
| RACE | 30.06% ± 0.65 | 33.58% ± 0.67 | 25.0% | +8.6 | 4,934 | Exam reading comprehension |
| CommonsenseQA | 26.86% ± 1.27 | 30.47% ± 1.32 | 20.0% | +10.5 | 1,221 | 5-choice commonsense (random = 20%) |
| COPA | 60.00% ± 4.90 | 59.00% ± 4.92 | 50.0% | +10.0 | 100 | Causal reasoning |
| LogiQA | 21.20% ± 1.60 | 25.65% ± 1.71 | 25.0% | +0.7 | 651 | Logical deduction |
| WSC273 | 53.11% ± 3.02 | — | 50.0% | +3.1 | 273 | Winograd coreference |
| TruthfulQA MC1 | 20.44% ± 1.41 | — | 22.8% | -2.4 | 817 | Resistance to common misconceptions |
| Arithmetic | 1.91% ± 0.12 | — | 0.0% | +1.9 | 14,000 | 2-5 digit add/sub/mul, generated in-harness |
Bold marks the metric that is conventional for that task — acc_norm for
HellaSwag, ARC, PIQA and OpenBookQA; acc for WinoGrande, BoolQ, SciQ and
LAMBADA. The convention is applied per task, not chosen per result: it lowers
the reported figure for ARC-Easy (44.53 rather than 49.54) and PIQA (59.79
rather than 60.83). Δ compares the bolded metric to the random baseline.
Both metrics
| Benchmark | acc |
acc_norm |
n |
|---|---|---|---|
| HellaSwag | 29.86% ± 0.46 | 32.72% ± 0.47 | 10,042 |
| ARC-Easy | 50.17% ± 1.03 | 46.09% ± 1.02 | 2,376 |
| ARC-Challenge | 20.82% ± 1.19 | 25.94% ± 1.28 | 1,172 |
| PIQA | 60.88% ± 1.14 | 60.23% ± 1.14 | 1,838 |
| WinoGrande | 49.25% ± 1.40 | — | 1,267 |
| OpenBookQA | 17.80% ± 1.71 | 28.40% ± 2.02 | 500 |
| BoolQ | 60.83% ± 0.85 | 62.11% ± 0.85 | 3,270 |
| SciQ | 76.00% ± 1.35 | 68.50% ± 1.47 | 1,000 |
| LAMBADA (OpenAI) | 26.57% ± 0.62 | — | 5,153 |
| MMLU (5-shot) | 25.26% ± 0.37 | — | 14,042 |
| RACE | 30.06% ± 0.65 | 33.58% ± 0.67 | 4,934 |
| CommonsenseQA | 26.86% ± 1.27 | 30.47% ± 1.32 | 1,221 |
| COPA | 60.00% ± 4.90 | 59.00% ± 4.92 | 100 |
| LogiQA | 21.20% ± 1.60 | 25.65% ± 1.71 | 651 |
| WSC273 | 53.11% ± 3.02 | — | 273 |
| TruthfulQA MC1 | 20.44% ± 1.41 | — | 817 |
| Arithmetic | 1.91% ± 0.12 | — | 14,000 |
Language modelling
| Corpus | Value | Metric |
|---|---|---|
| WikiText-2 (raw) | 25.18 | token-level perplexity |
| WikiText-2 (raw) | 46.82 | word-level perplexity |
| WikiText-2 (raw) | 1.0348 | bits per byte |
| LAMBADA | 34.27 | perplexity of the target word |
WikiText-2 is scored with a rolling 1024-token window at stride 512, so every counted token is predicted with at least 512 tokens of left context and each token is counted exactly once. (Scoring disjoint windows instead inflates these figures by ~15% because the leading tokens of each window are predicted from nothing.)
Trajectory across all benchmarked checkpoints
Tokens seen: 34.8B | 36.5B | 39.4B | 42.4B | 45.3B. The pretraining data mixture was changed partway through this sequence (code weight raised, several synthetic sources cut), so these columns do not represent a tokens-only progression.
Accuracy benchmarks (higher is better)
| Benchmark | 63,200 | 64,800 | 67,600 | 70,400 | 73,200 | net | |
|---|---|---|---|---|---|---|---|
| HellaSwag | 32.22% | 32.66% | 32.84% | 32.51% | 32.72% | +0.50 | up |
| ARC-Easy | 44.53% | 45.08% | 44.91% | 45.58% | 46.09% | +1.56 | up |
| ARC-Challenge | 25.77% | 25.51% | 26.02% | 25.00% | 25.94% | +0.17 | up |
| PIQA | 59.79% | 59.85% | 61.53% | 59.96% | 60.23% | +0.44 | up |
| WinoGrande | 49.49% | 50.51% | 51.07% | 49.57% | 49.25% | -0.24 | down |
| OpenBookQA | 30.00% | 28.00% | 29.80% | 29.00% | 28.40% | -1.60 | down |
| BoolQ | 60.55% | 60.83% | 61.80% | 62.14% | 60.83% | +0.28 | up |
| SciQ | 75.10% | 76.10% | 75.70% | 76.30% | 76.00% | +0.90 | up |
| LAMBADA (OpenAI) | 25.50% | 25.42% | 24.94% | 26.96% | 26.57% | +1.07 | up |
| MMLU (5-shot) | 26.07% | 26.71% | 25.77% | 26.01% | 25.26% | -0.81 | down |
| RACE | 32.79% | 32.77% | 32.96% | 32.85% | 33.58% | +0.79 | up |
| CommonsenseQA | 29.57% | 29.57% | 29.40% | 30.55% | 30.47% | +0.90 | up |
| COPA | 58.00% | 58.00% | 61.00% | 59.00% | 60.00% | +2.00 | up |
| LogiQA | 25.81% | 25.19% | 25.65% | 25.81% | 25.65% | -0.15 | down |
| WSC273 | 54.95% | 52.38% | 52.75% | 51.28% | 53.11% | -1.83 | down |
| TruthfulQA MC1 | 19.22% | 20.20% | 19.58% | 19.83% | 20.44% | +1.22 | up |
| Arithmetic | 0.14% | 0.55% | 0.58% | 1.04% | 1.91% | +1.78 | up |
Language modelling (LOWER is better)
| Metric | 63,200 | 64,800 | 67,600 | 70,400 | 73,200 | net | |
|---|---|---|---|---|---|---|---|
| WikiText-2 perplexity | 26.47 | 25.79 | 25.98 | 25.79 | 25.18 | -1.293 | BETTER |
| WikiText-2 bits/byte | 1.051 | 1.043 | 1.045 | 1.043 | 1.035 | -0.01606 | BETTER |
| LAMBADA perplexity | 35.79 | 36.19 | 36.84 | 34.15 | 34.27 | -1.523 | BETTER |
The Pile, by content type (bits/byte, LOWER is better)
| Category | 63,200 | 64,800 | 67,600 | 70,400 | 73,200 | net | |
|---|---|---|---|---|---|---|---|
| Code / technical | 0.9558 | 0.9518 | 0.9438 | 0.9341 | 0.9336 | -0.0222 | BETTER |
| Science / legal | 0.8987 | 0.8950 | 0.8930 | 0.8892 | 0.8880 | -0.0107 | BETTER |
| Web / reference | 1.1586 | 1.1561 | 1.1557 | 1.1543 | 1.1527 | -0.0059 | BETTER |
| Prose / spoken | 1.5606 | 1.5466 | 1.5650 | 1.5383 | 1.5411 | -0.0196 | BETTER |
| Every non-prose category has improved strictly monotonically at every checkpoint measured, each reaching its best at the latest one. Prose/spoken is the only exception: it dipped for exactly one interval after the mixture changed, recovered to its best two intervals later, and has been roughly flat since -- still a clear net improvement over the full span, and a one-off transition cost rather than a permanent trade. |
Does few-shot prompting help? (MMLU by shot count)
| Shots | step 70400 | step 73,200 | Shot source |
|---|---|---|---|
| 5 | 26.01% | 25.26% ± 0.37 | dev split, the published convention |
No. More demonstrations do not help and the 5-shot result is the best of the three at both checkpoints, with 10-shot dropping to the 25% chance line (-1.47 points vs 5-shot at step 73,200, ~2.8 sigma). The same ordering appears independently at both checkpoints, so it is not a fluke of one run.
This is what a model without in-context learning looks like: using examples to infer a task is an ability that emerges later in training, and before it does, extra shots are just tokens competing for attention with the actual question. Practical consequence: prompt this model with a short direct prefix, not a long few-shot preamble.
Arithmetic
Exact-match on the answer, greedy decoding, GPT-3 prompt format
(Question: What is 47 plus 21? / Answer: 68).
| Operation | step 70400 | step 73,200 | n |
|---|---|---|---|
| 2-digit addition | 4.25% | 3.70% | 2,000 |
| 2-digit subtraction | 2.50% | 9.05% | 2,000 |
| 3-digit addition | 0.00% | 0.00% | 2,000 |
| 3-digit subtraction | 0.40% | 0.65% | 2,000 |
| 4-digit addition | 0.00% | 0.00% | 2,000 |
| 5-digit addition | 0.00% | 0.00% | 2,000 |
| 2-digit multiplication | 0.15% | 0.00% | 2,000 |
| overall | 1.043% | 1.914% | 14,000 |
The model essentially cannot do arithmetic — but two-digit subtraction moved from 0.65% to 3.30% between these two checkpoints (5.1x, ~6 sigma on identical problems), which is the signature of a capability just beginning to emerge. Note that 15.5% of the pretraining mix is mathematics, yet that has bought fluency in mathematical language rather than the ability to compute.
Items are generated in-harness from a fixed seed using this prompt format, because
EleutherAI/arithmeticis a loading script with no parquet branch and cannot be fetched underdatasets>=3. Both checkpoints see byte-identical problems, so the comparison is exact — but these numbers are not interchangeable with published EleutherAI/arithmetic results.
Language modelling by genre (The Pile)
Bits-per-byte on each Pile domain, lower is better, scored with the same rolling 1024-token window as WikiText-2 so the numbers are directly comparable to it. This is the clearest picture of what the model is actually good at, because it measures raw prediction rather than multiple-choice ability.
| Domain | bits/byte | perplexity | tokens | Δ vs prev |
|---|---|---|---|---|
| Github | 0.616 | 3.91 | 479,656 | -0.0003 |
| PubMed Central | 0.771 | 15.67 | 292,334 | -0.0025 |
| USPTO Backgrounds | 0.803 | 16.81 | 296,162 | -0.0022 |
| NIH ExPorter | 0.881 | 25.64 | 41,533 | -0.0023 |
| ArXiv | 0.896 | 8.07 | 447,478 | -0.0039 |
| PubMed Abstracts | 0.905 | 21.51 | 306,636 | -0.0008 |
| StackExchange | 0.962 | 13.35 | 387,038 | -0.0063 |
| FreeLaw | 0.996 | 21.39 | 339,260 | +0.0009 |
| Wikipedia (en) | 1.014 | 22.12 | 341,511 | -0.0013 |
| Pile-CC | 1.122 | 36.44 | 326,706 | -0.0012 |
| OpenWebText2 | 1.158 | 33.87 | 348,909 | -0.0022 |
| BookCorpus2 | 1.163 | 34.44 | 139,043 | +0.0016 |
| Enron Emails | 1.260 | 22.00 | 16,119 | -0.0049 |
| Gutenberg (PG-19) | 1.305 | 36.62 | 133,371 | +0.0110 |
| HackerNews | 1.317 | 43.69 | 57,843 | -0.0014 |
| Books3 | 1.356 | 35.41 | 396,623 | -0.0035 |
| OpenSubtitles | 1.363 | 31.87 | 234,514 | -0.0030 |
| PhilPapers | 1.388 | 54.37 | 38,263 | +0.0039 |
| DM Mathematics | 1.390 | 8.23 | 370,912 | +0.0102 |
| Ubuntu IRC | 1.792 | 44.38 | 14,407 | +0.0014 |
| YoutubeSubtitles | 1.901 | 135.86 | 51,428 | -0.0065 |
| EuroParl | 2.059 | 145.42 | 19,523 | +0.0172 |
The ordering here is a direct readout of the pretraining mix: code, papers and mathematics sit at the top because they are what the model has been fed most of.
Progress since the previous checkpoint
Same suite, same code, same full evaluation sets — only the checkpoint differs. Step 70,400 → 73,200 is +2.94B tokens.
| Benchmark | step 70,400 | step 73,200 | Δ | ±2σ needs |
|---|---|---|---|---|
HellaSwag (acc_norm) |
32.51% | 32.72% | +0.21 | ±0.66 |
ARC-Easy (acc_norm) |
45.58% | 46.09% | +0.51 | ±1.45 |
ARC-Challenge (acc_norm) |
25.00% | 25.94% | +0.94 | ±1.80 |
PIQA (acc_norm) |
59.96% | 60.23% | +0.27 | ±1.62 |
WinoGrande (acc) |
49.57% | 49.25% | -0.32 | ±1.99 |
OpenBookQA (acc_norm) |
29.00% | 28.40% | -0.60 | ±2.86 |
BoolQ (acc) |
62.14% | 60.83% | -1.31 | ±1.20 |
SciQ (acc) |
76.30% | 76.00% | -0.30 | ±1.91 |
LAMBADA (OpenAI) (acc) |
26.96% | 26.57% | -0.39 | ±0.87 |
MMLU (5-shot) (acc) |
26.01% | 25.26% | -0.75 | ±0.52 |
RACE (acc_norm) |
32.85% | 33.58% | +0.73 | ±0.95 |
CommonsenseQA (acc_norm) |
30.55% | 30.47% | -0.08 | ±1.86 |
COPA (acc) |
59.00% | 60.00% | +1.00 | ±6.94 |
LogiQA (acc_norm) |
25.81% | 25.65% | -0.15 | ±2.42 |
WSC273 (acc) |
51.28% | 53.11% | +1.83 | ±4.27 |
TruthfulQA MC1 (acc) |
19.83% | 20.44% | +0.61 | ±1.98 |
Arithmetic (acc) |
1.04% | 1.91% | +0.87 | ±0.14 |
| WikiText-2 perplexity | 25.79 | 25.18 | -0.6123 | — |
| WikiText-2 bits/byte | 1.043 | 1.035 | -0.007709 | — |
| LAMBADA perplexity | 34.15 | 34.27 | +0.124 | — |
Δ is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; anything unbolded is inside the noise floor and should not be read as movement. The quoted error treats the two runs as independent, which is conservative here — they score identical items, so the true paired error is smaller.
What actually changed. One metric moved decisively; the rest of the suite is quiet. This is a normal-rate interval after the unusually large one before it.
- Arithmetic +84%, 146 -> 268 / 14,000 (6.0 sigma) -- driven by two-digit subtraction tripling, 50 -> 181 / 2000. Across five checkpoints the total has gone 19 -> 77 -> 81 -> 146 -> 268: the only capability in the suite showing sustained compounding growth. The two operations alternate rather than advancing together -- subtraction jumped, then addition, now subtraction again. Addition's 85 -> 74 dip is 0.9 sigma (noise), and the earlier 3/2000 on multiplication was always within chance of zero.
- WikiText-2 perplexity -2.37%, 25.788 -> 25.176 (bits/byte 1.0425 -> 1.0348).
- The Pile improved in 15 of 22 domains (sign test p = 0.067, not significant) with the token-weighted mean edging down 1.0488 -> 1.0480. The three technical categories all improved while prose/spoken ticked up (+0.0028) -- a faint echo of the data-mixture signature, roughly an order of magnitude weaker than when it first appeared.
- No accuracy benchmark moved: 9 of 17 improved, sign test p = 0.50, largest 1.4 sigma.
For contrast, the preceding interval (70,400) carried the same +2.94B tokens but produced two results past 2 sigma and 20/22 Pile domains at p = 0.0001. The difference is most consistent with that interval capturing the model settling into the changed data mixture, and this one showing the underlying steady rate.
Reading these numbers. This is a partially-trained 1.5B base model, so knowledge-heavy multiple-choice tasks sit close to their random baselines — that is expected at this scale and token count. The signal to watch is the language-modelling side: LAMBADA accuracy and WikiText perplexity measure whether the model has actually learned to predict text, and those improve steadily long before multiple-choice benchmarks move. Note also that BoolQ's majority-class baseline is 62.2%, so a score near that is not evidence of comprehension.
Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.
| Metric | Value | Notes |
|---|---|---|
| Decode speed | 68.5 tok/s | steady-state autoregressive decode |
| Warm prefill | ~20 ms | short prompt, shape already compiled |
| Checkpoint load | ~26 s | params → GPU, from local cache |
| Active VRAM | ~5.0 GB | of 24 GB |
Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.
Inference Example
from zenyx_v3_inference import ZenyxGenerator
generator = ZenyxGenerator(step=73200)
# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(generator.generate(
"The capital of France is",
max_new_tokens=80,
temperature=0.7,
repetition_penalty=1.15,
))
Evaluation Reproducibility
Benchmarks were produced by modal_base_evals.py on a single NVIDIA L4, scoring
continuations in batches with length-bucketed padding. Task formats follow the
lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions
and answer-key handling), so the numbers are broadly comparable to published
base-model results, though this is an independent implementation rather than a
harness run.
Limitations
- Pretraining is incomplete — the model will change substantially with more tokens.
- Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
- Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.