janakhpon/mon_tokenizer

Mon Tokenizer

A Unigram tokenizer for Mon (mnw), Burmese and English, with full byte fallback.

Mon mixes with Burmese constantly and English routinely, so all three are trained on and measured separately. Anything else on the page (Thai, emoji, IPA, CJK) round-trips through byte fallback rather than being lost.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("janakhpon/mon_tokenizer")
ids = tokenizer.encode("ဂွံအခေါင်အရာမွဲသ္ဂောံဒုင်စသိုင်ကၠာကၠာရ။", add_special_tokens=False)
assert tokenizer.decode(ids) == "ဂွံအခေါင်အရာမွဲသ္ဂောံဒုင်စသိုင်ကၠာကၠာရ။"

Or as a standalone package, with no transformers dependency:

pip install "mon-tokenizer>=1.0.0"

from mon_tokenizer import MonTokenizer

tokenizer = MonTokenizer()
result = tokenizer.encode("ဂွံအခေါင်အရာမွဲ")

Both produce identical token ids. The package's model_card.json carries the numbers below in machine-readable form, and a test fails if they drift from what the artifact does.

Measured

Vocabulary 64,256 · Unigram · trained on the train split of an 893,936-line / 85.8M-character corpus · scored on the whole validation split.

stratum	chars/token	tokens/line	syllable violations	round-trip	byte-fallback tokens
Mon	4.686	16.4	1.07% (n=492,469)	100%	20 / 486,631
Burmese	4.117	13.9	0.93% (n=25,546)	100%	0 / 24,942
English	4.112	34.6	— (n=0)	100%	96 / 453,350
mixed script	3.804	24.6	0.81% (n=28,133)	100%	116 / 59,440

98.74% of the distinct characters in the Mon validation split are single tokens — 392 of 397, over all 29,600 lines. The remaining five (a combining diaeresis below, a Greek capital pi, two Sinhala letters and one emoji; seven occurrences in 2.28M characters) round-trip through byte fallback at one to four tokens each.

Four notes on reading these:

• Compression describes a corpus, not a tokenizer. Quote it with the corpus or not at all. On a different distribution you will get a different number.
• Byte fallback is given as a count, not a rate. Mon's 20 tokens in 486,631 round to a two-decimal zero, which reads as never; Burmese's zero is a real zero. Those are different facts and a percentage renders them identically.
• Syllable violations count token boundaries falling inside a Myanmar syllable. They are measured with a purpose-built segmenter, not Unicode grapheme clusters — UAX #29 puts a cluster break before ာ, so \X scores a cut through the middle of ကျော် as clean. The denominator is given because English legitimately has none.
• Round-trip compares after normalization, which is a deliberate transform and not a loss.

Everything round-trips

for text in ["🙏 emoji", "ภาษาไทย", "漢字", "Ωπ√∫", "ကျော် page 42 — “quoted” ၏"]:
    assert tokenizer.decode(tokenizer.encode(text, add_special_tokens=False)) == text

The vocabulary carries all 256 <0xNN> byte pieces, so nothing is unrepresentable. This matters if you are feeding OCR output into a corpus: without byte fallback, characters outside the vocabulary are not flagged — they are deleted, leaving fluent-looking output with content silently missing.

Normalization travels with the model

The normalizer is stored inside tokenizer.json, so it cannot drift from the weights: invisible characters stripped, Unicode space separators folded to U+0020, then NFC. Runs of spaces are preserved.

That is a fix, not a feature. The previous release applied no normalization at encode time while having been trained on normalized text, so a single ZERO WIDTH SPACE cost five tokens instead of one.

Special tokens

token	id
<unk>	0
<s>	1
</s>	2
<pad>	3

All four have real ids. In the previous release <pad> was declared but never assigned one, so padding a batch padded with <unk> — silently.

Upgrading from the previous release

Every token id has changed. This is a retrain on a new algorithmic stack, so any embedding matrix built against the old vocabulary must be rebuilt. Pin mon-tokenizer<1.0 if you are not ready.

Two things about the previous release are worth stating plainly, because they were live:

• tokenizer.json here was the 4,000-piece predecessor, not the 32,000-piece model the card described. AutoTokenizer prefers tokenizer.json over tokenizer.model, so anyone loading this repo through the documented path got a tokenizer measuring 0.93 chars/token against an advertised 5.22. This release replaces it; there is no tokenizer.model any more, so the two cannot disagree again.
• The advertised 5.22 chars/token did not reproduce on an independent corpus. The numbers above were measured on held-out text and are stated with the corpus they came from.

Why Unigram

Measured at equal vocabulary on this corpus: BPE compresses better everywhere (+6.2% Mon, +25.8% English) but splits Myanmar syllables 2.5× more often. Byte-level BPE is far behind: Myanmar is three UTF-8 bytes per character, giving 1.524 chars/token against 4.507, both at 48,000 pieces over the 29,600-line Mon validation split.

Unigram wins for an OCR consumer, where the syllable is the unit a reader sees. Full reasoning and the rejected alternatives: docs/architecture.md.

Links

• Package and source: Code-Yay-Mal/mon_tokenizer — trainer, metrics and the full architecture rationale
• Corpus: MonCorpusCollection
• Ecosystem: Awesome Mon NLP

License

MIT for the card and the configs. Not for tokenizer.json.

The vocabulary was fitted to a corpus this project neither collected nor owns, so it is a derived work and its terms follow that corpus. The Mon portion comes from MonCorpusCollection, whose largest source is CC BY-SA 4.0 Mon Wikipedia. Its second largest, the Independent Mon News Agency archive, has no established licence at all.

LICENSE states which sources are established and which are not. Nothing here grants rights over tokenizer.json, because none were obtained. The front-matter said mit until 2026-08-13, which claimed more than this repository holds.