sayedM/cohere-transcribe-arabic-cpu-friendly

cohere-transcribe-arabic — CPU-friendly (int8 + Arabic CTC draft head)

A CPU-only build of CohereLabs/cohere-transcribe-arabic-07-2026, a 2.07-billion-parameter Conformer encoder–decoder ASR model. No GPU, no CUDA.

Two things are packaged here that the original does not have:

1. int8 dynamic quantization of all 515 nn.Linear modules — 87.6% of the parameters — stored as safetensors rather than a pickle.
2. An Arabic CTC draft head (57.5M parameters) for speculative decoding, which makes short clips 1.42× faster at a cost of 0.04 percentage points of word error rate.

Together: 158.5 s → 4.82 s on a 33-second Arabic clip, on a six-core desktop CPU.

---

The draft head, and why it helps

The decoder here is autoregressive: one forward pass per token, ~130 per chunk. On a CPU at batch 1 that is not an arithmetic problem — it is a per-call problem. Measured on this machine:

tokens per decoder forward	1	4	8	16	32
ms per forward	22.06	24.55	27.38	27.17	28.72
ms per token	22.06	6.14	3.42	1.70	0.90

Thirty-two tokens cost 1.30× what one token costs. The per-token price falls 24-fold. Almost all of a decoder call is fixed overhead, and the arithmetic in between is nearly free.

That is exactly the condition speculative decoding needs. If something cheap can guess the next K tokens, the decoder can check all of them in a single forward for roughly the price of checking one.

The CTC head is that cheap thing. It sits on the encoder output and is frame-synchronous and non-autoregressive — CTC assumes tokens are conditionally independent given the audio, so the whole hypothesis comes out of one forward pass. That independence assumption makes it a poor transcriber and an excellent drafter. Measured cost: 22.8 ms on a 10-second clip, against a 1069 ms decoder — a 2.1% tax.

The effect, measured: 2.66 tokens retired per decoder call instead of 1, which is 1.29×–1.48× wall clock depending on the clip, as shown in the figure above.

A wrong guess cannot produce a wrong word

The verifier keeps only the prefix its own argmax agrees with, then appends one token taken straight from the decoder. A drafted token reaches the output only if the decoder would have produced it anyway. A completely wrong draft degrades to ordinary greedy decoding. What a bad draft costs is time, not accuracy — which is why the accuracy column below barely moves.

---

Training data for the CTC head

dataset	oddadmix/lahgtna-v3-small — multi-dialect Arabic speech
used for training	4,256 clips · 18.33 h from the train split (median clip 15.3 s, filtered to 1–30 s)
held out	1,024 clips · 4.58 h from the test split, never trained on
labels	the base model's own transcripts — self-distillation
epochs	12, AdamW, F.ctc_loss with blank = pad_token_id = 2

Only the audio from that dataset was used. The targets are not its transcripts — they are what CohereLabs/cohere-transcribe-arabic-07-2026 itself emits on that audio. The drafter's job is to predict this model's output, so training it on anything else would teach it to propose tokens the verifier would reject. It also means no human annotation was involved anywhere in this head.

Labels were produced by an fp32 GPU replica of the deployed int8 model rather than by bf16, which cut label noise against the deployed model from 2.81% to 1.74% WER.

Vocabulary saturates almost immediately. Only 274 of the 16,384 vocabulary entries contain an Arabic character at all — the rest belong to other scripts the multilingual base model supports — and the tokenizer is sub-word for Arabic at ~2.4 tokens per word. After 0.83 hours, the types seen 10+ times already covered 99.24% of all token occurrences. Past that point more data buys acoustic quality, not new symbols.

The head is data-limited, not capacity-limited: acceptance went 0.143 at 0.83 h to 0.331 at 18.3 h, and test loss was still improving on the metric that matters when labelling stopped for time. More hours would help.

---

Where the time goes

Two independent measurements agree on this: forward hooks attributing time to architectural components, and torch.profiler attributing it to ATen kernels. The kernel view:

ATen kernel	self CPU	share	calls	what it is
quantized::linear_dynamic	900.9 ms	58.5%	4,079	the int8 GEMMs — the real work
aten::mkldnn_convolution	264.4 ms	17.2%	149	conformer convs, still fp32
SDPA flash attention	55.1 ms	3.6%	928	attention proper
aten::aminmax	36.1 ms	2.3%	4,079	the per-call activation scale — pure dynamic-quant overhead
aten::cat	34.9 ms	2.3%	952	KV cache growing one token at a time
view / empty / as_strided	~74 ms	4.8%	48,600	dispatch overhead, no arithmetic

aminmax fires exactly once per linear_dynamic — that is dynamic quantization recomputing an activation scale on every single call. And 48,600 calls to pure-metadata operations do no arithmetic at all. The decoder issues roughly 3,520 of the 4,079 GEMM calls, each on a 1×1024 vector.

---

Performance

Measured on an Intel i5-12400F (6 cores / 12 threads, AVX2, no AVX-512, no AMX), 32 GB RAM, Windows 11, torch 2.14.0+cpu, transformers 5.16.1. Wall clock includes feature extraction, encoding, decoding and detokenization. RTFx = seconds of audio per second of compute; 1.0× is real time.

The three precisions, 33.3 s Arabic clip

weights	resident	wall	RTFx
bf16 — the original's own dtype	4.13 GB	158.5 s	0.21×
fp32	8.26 GB	14.2 s	2.35×
int8 dynamic — this repo	2.84 GB	7.8 s	4.28×

bf16 is the checkpoint's native dtype and the worst possible choice on this class of CPU: AVX2 has neither AVX512-BF16 nor AMX, so PyTorch emulates it in software and the model runs five times slower than simply listening to the audio. That single line is the largest factor in the headline number.

End to end, 954 seconds of mixed Arabic and English

int8, batch 16, 12 threads, preallocated KV cache, draft head enabled:

file	audio	wall	RTFx
sample1 (ar)	3.6 s	0.83 s	4.27×
sample2 (ar)	8.5 s	1.67 s	5.10×
ElevenLabs TTS (ar)	33.3 s	4.82 s	6.90×
WhatsApp voice note (en)	76.3 s	11.57 s	6.59×
14-minute talk (ar)	832.8 s	98.36 s	8.47×
total	954.4 s	117.3 s	8.14×

What the draft head is worth, referenced to fp32

fp32 is the only reference that separates different from worse:

configuration	wall	vs int8	WER from fp32
fp32 greedy	373.75 s	—	0 (reference)
int8 greedy	207.47 s	1.00×	2.06%
int8 + CTC draft head	145.76 s	1.42×	2.10%

Accuracy cost of int8, by normalization level

normalization	WER	CER	differing words
raw	5.05%	1.56%	122 of 2,414
no punctuation	3.63%	1.19%	88
+ hamza / ta-marbuta unified	2.06%	0.80%	50

Quote 2.06%, not 5.05%. Of 122 raw differences, 73 are orthographic convention — hamza seating (أ→ا), ta-marbuta (ة→ه), punctuation — not a different word being heard. Standard Arabic ASR evaluation normalizes these. Per file at the normalized level: two clips are identical to fp32, and the 14-minute recording — the only statistically meaningful one at 2,086 words — is 2.25%.

For reference: the same model on a GPU

	wall (954 s of audio)	RTFx	WER from fp32
this repo, CPU, batch 16	117.30 s	8.14×	2.10%
RTX 3090 Ti, bf16, batch 1	48.91 s	19.51×	0.25%
RTX 3090 Ti, bf16, batch 16	8.91 s	107.07×	0.16%

A GPU is both faster and more accurate — bf16 is a far lighter perturbation than int8. This repo is for when there is no GPU. Worth knowing: the gap is 4.2× at batch 1 and 13.2× batched, because only the GPU gains from being fed more work. The draft head also transfers to the GPU, where it gives 1.40× at batch 1 — almost exactly what it gives on the CPU, for the same underlying reason.

---

Usage

pip install "transformers>=5.4" torch safetensors soundfile librosa

from huggingface_hub import snapshot_download
import sys

repo = snapshot_download("sayedM/cohere-transcribe-arabic-cpu-friendly")
sys.path.insert(0, repo)                      # the loader ships inside the repo

from cpu_model_loader import load_cpu_model, load_draft_head, transcribe

model, processor = load_cpu_model(repo)
head, _ = load_draft_head(repo)               # optional; Arabic only

print(transcribe(model, processor, "audio.mp3", language="ar", head=head))

Without the draft head, drop head=head. language is "ar" or "en" and is not optional in practice: an English recording decoded as Arabic turned the name Nasser into NASA.

Loading takes about 10 s and needs ~2.9 GB of RAM.

---

How the quantization works

Every nn.Linear — 1.81 B of the 2.07 B parameters — is stored as int8. Convolutions, layer norms and embeddings stay fp32. Weights use one per-tensor scale, float32(max|W| / 127.5), with a zero point of 0. Activations are quantized per call by quantized::linear_dynamic, to 7 bits (reduce_range), from the runtime minimum and maximum of whatever tensor the layer is handed.

That last detail has a visible consequence: there is one activation scale for the whole tensor, so handing a layer a wider tensor quantizes it more coarsely. This is why the model's output shifts slightly with batch size, and why speculative decoding cannot be bit-identical to greedy.

The draft head architecture: one ParakeetEncoderBlock copied from encoder.layers[47], then a projection, a LayerNorm and an output matrix copied from the base model's decoder.proj, decoder.norm and proj_out. Folding those three into a single matrix — the obvious "free" initialisation — measured worse than random init, because the real path contains a LayerNorm and a LayerNorm cannot be folded into a matrix product.

---

Limitations

• The draft head is Arabic-only. On English it drafts noise, every draft is rejected, and the wider verify forward makes it a net loss — measured 0.95×. transcribe() applies it only to single-chunk audio; pass head=None for other languages.
• Speculative decoding is batch-1 only. transformers' assisted generation does not support more. Long audio is already batched, which amortizes the decoder anyway.
• Do not benchmark the draft head against int8 greedy. It differs by ~4% WER there, which is the quantizer's sequence-length sensitivity, not drafting error. Against fp32 the cost is 0.04 pp.
• CPU only. quantized::linear_dynamic is registered for the CPU dispatch key alone; moving this model to CUDA raises NotImplementedError. For a GPU, use the base model in bf16.
• int8 costs accuracy. 2.06% WER from fp32. If you have the RAM and can spare the speed, fp32 is more faithful; if you have a GPU, bf16 is both faster and more accurate.
• The draft head saw 18.3 h of one dataset's dialect mix. Acoustics far from it will accept less.
• Inherits every limitation of the base model, including its language and domain coverage.

---

Verification

The export is checked against the original quantized model before publishing, by cpu_model_loader.verify():

• every one of the 515 quantized layers is present, with identical scales and int_repr;
• sampled layers dequantize to a maximum difference of 0.0;
• both models generate byte-identical token sequences on the same audio.

Two defects that check caught and that would otherwise have shipped silently: a non-persistent buffer (encode_positions.inv_freq) that state_dict() does not report, and a GenerationConfig rebuilt from config.json that lost decoder_start_token_id, which changed decoding while leaving the weights bit-perfect.

Files

file
model.safetensors	int8 weights, their per-tensor scales, and every fp32 parameter and buffer
quant_map.json	which modules are quantized, their shapes, and the scheme
draft_head.safetensors, draft_config.json	the Arabic CTC draft head
cpu_model_loader.py	loads the above; no pickle, no trust_remote_code
draft_head.py, spec_decode.py	the head definition and the speculative-decoding drafter
tokenizer / processor / config	copied unchanged from the base model

License and attribution

Apache 2.0, inherited from CohereLabs/cohere-transcribe-arabic-07-2026, copyright Cohere Labs.

Changes made to the original (Apache 2.0 §4b): weights of all nn.Linear modules quantized to int8 as described above; a separately trained CTC draft head added; no change to the architecture, the tokenizer, or the training data of the base model.

The base repository is gated. This derivative is not, so if you need the original weights, please obtain them from Cohere Labs directly and accept their terms. Speech data for the draft head came from oddadmix/lahgtna-v3-small; please observe that dataset's own licence and terms for any redistribution of the audio itself (none of it is included here).