MiniCPM5-2B (LiteRT-LM)

This repository hosts the LiteRT-LM (LiteRT formerly known as TensorFlow Lite) version of openbmb/MiniCPM5-2B, optimized for fully on-device inference on mobile and edge hardware.

---

Available Models

Recommend to use below both CPU and GPU compatible models, originally from mlboydaisuke/MiniCPM5-2B-LiteRT Requires litert-lm ≥ 0.16 (thought channel + ThinkingConfig); measured here on 0.17.0.

File	Recipe	Size
MiniCPM5-2B_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	1.55 GB
MiniCPM5-2B_int8.litertlm	int8 dynamic on linears + embedding; fp32 activations declared (see notes)	2.60 GB

The int4 file is the phone file (smaller, fastest GPU decode on every platform measured) — best used for direct answers or short reasoning; see the thinking-mode note below. int8 is the file for reasoning that has to complete: its thinking chains are ~3–4× shorter than int4's on the same questions and terminate where int4 runs into the token budget. int8's main weight section is 2.33 GB, above the single-section mmap ceiling of default-entitlement iOS apps, so it is a desktop / Android build.

Both files carry the checkpoint's own chat_template.jinja byte for byte and declare the thought channel, so enable_thinking, the tool-calling format, and the runtime's ThinkingConfig all work unchanged (details under Conversion notes).

Further, below are some CPU-only models for exploration:

• minicpm_wi4c_wi8_afp32.litertlm: This model features mixed INT4/INT8 weight-only quantization with FP32 activations (afp32). MLP projections use channelwise INT4 with Hadamard rotation; all remaining weights (attention, embedding, and lmhead) use channelwise INT8.
• minicpm_wi8_afp32.litertlm: This model features weight-only INT8 quantization (wi8) with FP32 activations (afp32).

What is MiniCPM?

MiniCPM5-2B is part of the MiniCPM5 series from OpenBMB. It is a dense 2.5B-parameter Transformer built specifically for on-device, local, and resource-constrained deployment.

Highlights

• 🧠 Hybrid Reasoning — a single checkpoint serves as both a fast assistant and a deliberate reasoner via a built-in <think> template (enable_thinking).
• 🛠️ Tool use — native tool-calling support in the chat template.
• 📏 Long context — native 131,072-token context length.
• 📱 Built for the edge — compact footprint designed for local assistants, coding agents, and tool-use workflows.

Model Information

Item	Value
Type	Causal Language Model
Architecture	Standard LlamaForCausalLM
Parameters	2,516,756,480 (~2.5B)
Non-Embedding Parameters	1,981,982,720
Layers	42
Attention Heads (GQA)	16 (Q) / 2 (KV), head dim 128
Hidden / Intermediate Size	2048 / 6144
Vocabulary Size	130,560
Context Length	131,072

---

Use the model

Edge Gallery App (Android)

1. Get the App: Install the app from Google Play or download the latest APK from the GitHub releases page.
2. Importing the Model: Navigate to the Model manager within the app and click the "+" (plus) icon in the bottom-right corner. Two options will appear:
   • Import from HF (Recommended): Select this option, and a dialog box will appear showing an example Hugging Face model URL. Enter the HF link for the desired .litertlm model and click submit. The model will then appear in your list, and you can proceed to download it (a Hugging Face account login is required).
   • From local model file: First, download the .litertlm model directly to your Android device, OR download it to your computer and push it via ADB (e.g., adb push MiniCPM5-2B_int4.litertlm /sdcard/Download/). Then, select this option, choose the downloaded file from your storage, configure your preferred parameters, and tap "Import".

For full details on importing models and other features, see the Edge Gallery App Wiki.

To build the demo app from source, please follow the instructions from the GitHub repository.

Try It (Desktop/CLI)

Install uv and run the model directly from the LiteRT-LM command line:

uv tool install litert-lm
uvx litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm --prompt="What is the capital of France?"

# local file, GPU backend
litert-lm run ./MiniCPM5-2B_int4.litertlm --backend gpu --cache no --prompt "..."

Controlling thinking

Thinking is the model's default: with no ThinkingConfig it decides for itself and, in practice, reasons before every answer (100–700 characters on trivial questions, thousands on math). To control it:

• Give it a generous output budget (≥ 2048 tokens; 4096 for math). Truncated mid-thought it produces no final answer at all.
• The bundle declares the thought channel (<think> / </think>), so the streamed text contains only the answer, and runtimes that expose ThinkingConfig can cap or read the reasoning separately.
• enable_thinking=false (via ThinkingConfig or the conversation's extra context — both reach the template) switches the model to direct answers: two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers below. enable_thinking=true pre-fills the think opener explicitly. On the CLI: --thinking false / --thinking-budget N.
• Sampling: OpenBMB recommends temperature 1.0, top_p 0.95; the gates below are greedy.
• Prompt format is ChatML (<|im_start|>role\n…<|im_end|>\n), 4096-token KV budget, six prefill signatures (1024, 256, 64, 16, 4, 1).

Correctness

Both files score 8/8 on an 8-question sanity gate on both CPU and GPU (Apple M4 Max), with the reasoning arriving on the thought channel and only the final answer in the streamed text. The bf16 PyTorch model scores 8/8 on the same gate with thinking on or left to the model, and 6/8 with thinking forced off ("opposite of hot" → "Cool.", the rhyme line → "Green.") — read any thinking-off result against that, not against 8/8.

GSM8K (first 100 test questions, greedy, 0-shot chain-of-thought prompt, thinking off — the protocol OpenBMB's own MiniCPM5 cards use, max 2048 new tokens, identical prompt and extraction on every row):

Configuration	GSM8K
bf16 PyTorch (MPS), upstream template	92 %
int8, CPU	91 %
int4, CPU	86 %
int4, GPU (Metal)	87 %

int8 is at parity (7 of its 9 misses are the bf16 model's own). int4 costs about five points on this 42-layer model; the GPU's default fp16 activations cost nothing measurable in no-think mode.

Thinking mode is where int4 shows its damage. On ten GSM8K questions with thinking on and a 3584-token budget, the bf16 model closes its reasoning on 9/10 with ~3,000-character chains and int8 reproduces that question for question (9/10, median ~3,200 characters on CPU), while int4 closes 0/10 on CPU (median ~13,700 characters — it keeps re-checking and runs into the budget; the answer is usually right inside the thought text but never gets emitted). On the Metal GPU with the runtime's default fp16 activations the same int4 file happened to close 10/10 with ~8,000-character chains, but that is fp16 rounding steering the trajectory, not a property to rely on. If your use needs the reasoning to finish, use int8 or turn thinking off on int4.

On a Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno) both files generate correctly on GPU and CPU with full OpenCL delegation — 1873/1873 nodes on every prefill signature and decode, zero rejected ops — and the runtime separates the reasoning on-device ([thought] … [/thought], then the answer).

On an iPhone 17 Pro the int4 file passes the same 8-question gate on both backends (on-device byte count verified against the source file): Metal GPU 7/8 (init 5.7 s) and CPU 7/8 (init 2.2 s); the one miss on each leg is the rhyme line inside the 8-question composite prompt, answered "green" — the same answer the bf16 model gives with thinking off, so a prompt-format artifact rather than conversion damage.

Multi-turn: three-turn conversations (introduce a name and a city, an arithmetic question, then "which city do I live in?") hold on both files under all three thinking modes with the name and city recalled — the template's history rendering stays consistent with what the runtime already streamed.

Performance

litert-lm benchmark (litert-lm 0.17.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, serialized, ≥300 s rest before each GPU reading; each backend confirmed to generate real text before its number was recorded:

File	Backend	Prefill (256)	Decode	TTFT	Init
int4	GPU (Metal)	1699 tok/s	92.8 tok/s	0.16 s	3.7 s
int4	CPU	149 tok/s	31.1 tok/s	1.76 s	4.5 s
int8 (fp32 activations)	GPU (Metal)	1405 tok/s	74.7 tok/s	0.20 s	3.0 s
int8 (fp32 activations)	CPU	161 tok/s	30.0 tok/s	1.62 s	15.0 s

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):

File	Backend	Prefill (205)	Decode	TTFT	Init	Peak RSS
int4	GPU (OpenCL)	401–411 tok/s	16.1–18.6 tok/s	0.56 s	11.2–13.1 s	1.14 GB
int4	CPU	39–72 tok/s	15.6–15.8 tok/s	2.9–5.3 s	3.1–5.7 s	2.12 GB
int8 (fp32 activations)	GPU (OpenCL)	150–160 tok/s	10.9–12.8 tok/s	1.4 s	4.0–6.3 s	1.10 GB
int8 (fp32 activations)	CPU	103–157 tok/s	11.7 tok/s	1.4–2.1 s	0.3 s	2.90 GB

GPU wins prefill (5–10× on the phone, 11× on the Mac) and time-to-first-token everywhere; on Adreno the int4 GPU decode edge over the same-device CPU is modest (~1.1×), while the int8 file's fp32 activations bring its GPU decode level with its CPU.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1 from a pristine released stack (reproduction: hf-to-litertlm, bash scripts/reproduce_llm.sh minicpm5-2b). Four things make these files run on both backends:

• The chat template is the checkpoint's chat_template.jinja, byte for byte, embedded on the runtime's jinja path — the same packaging as litert-community/MiniCPM5-1B. That is what keeps enable_thinking and the tool-calling format available to the app; the thought channel is declared alongside it.
• Start token <s> is correct for this family: the template's own {{ bos_token }} renders empty at runtime and the engine prepends the metadata start token, so the model sees exactly one <s> as it does upstream. Stops are the model's </s> and <|im_end|>.
• int4 needed a zero-scale fix. Decoder layer 0's MLP contains 13 all-zero rows; blockwise quantization emits a zero scale for each of their blocks, which the CPU (XNNPACK) path refuses to load (unsupported scale value (0.000000) … for INT4 tensor) while the GPU path silently accepts. The scales were replaced in place by a tiny positive epsilon (the quantized values in those blocks are zero, so the dequantized weights are unchanged) — 3,328 bytes of a 1.55 GB file.
• int8 declares fp32 activations in-bundle (prefer_activation_type = "fp32" in the bundle's model.toml; weights untouched). With the runtime's default fp16 GPU activations, the int8 model's reasoning on one gate question ran 2000+ tokens without closing </think> (it fails the 8-question gate); with fp32 declared it closes in ~450 tokens and passes 8/8, at a cost of ~14 % GPU decode speed. On a ten-question thinking-on GSM8K subset the two dtypes were closer (fp16 finished 10/10, fp32 7/10), so this is a measured trade for the gate, not a cure. The int4 file keeps the fp16 default: it passes the gate there, and declaring fp32 only reproduces the CPU reference's non-terminating chains (see Correctness).

Also: the 130560×2048 input-embedding table sits in its own section (embedder externalised); the tokenizer is embedded as the upstream tokenizer.json (byte-level BPE survives intact); int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding, int8 is dynamic per-channel on linears and embedding. A block-128 int4 variant was built and rejected: it loses 8 GSM8K points (79 vs 87, thinking off), fails the 8-question gate on CPU, and does not shorten the thinking chains.

Machine-readable manifest

litertlm_manifest.json in this repo describes both recommended files (sha256, size, sections, context length, verified backends, the measured rows above with their conditions, and known issues) for tooling that picks a file per device.

Links

• 🤗 Original model (BF16): openbmb/MiniCPM5-2B
• 🤗 Smaller sibling: litert-community/MiniCPM5-1B
• 📦 GitHub: OpenBMB/MiniCPM
• 🛠️ LiteRT docs: ai.google.dev/edge/litert

---

License

Released under the Apache-2.0 License, consistent with the upstream openbmb/MiniCPM5-2B.

Changes from the original work (for the two recommended files): weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; 13 all-zero weight rows' quantization scales set to an epsilon (no numeric change); tokenizer and chat template repackaged into the .litertlm bundle with runtime metadata (thought channel, stop tokens, activation-dtype preference). No fine-tuning.

Citation

@article{minicpm4,
  title={MiniCPM4: Ultra-efficient LLMs on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}