failed09/bashkir-lid

Bashkir LID

A compact language identification model for Bashkir and Russian. It is designed for corpus filtering and local applications that need lightweight CPU inference. The model returns a language label and probabilities for each input text.

Model Architecture

Property	Description
Task	Binary language identification: ba or ru
Features	Character TF-IDF n-grams (char_wb, lengths 2–4)
Classifier	Linear classifier exported to ONNX
Runtime	ONNX Runtime on CPU
Preprocessing	Unicode lowercase and character features, provided by lid.py
Outputs	Language labels and class probabilities

The ONNX graph takes sparse character features, not raw strings. The Python adapter handles preprocessing, batching and artifact checksum validation. No custom ONNX operators or Transformers installation are required. Model version, feature count and export checks are recorded in META.json; the runtime contract is in config.json.

Loading

pip install huggingface_hub onnxruntime numpy

import sys
from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    "failed09/bashkir-lid",
    allow_patterns=[
        "lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
    ],
)
sys.path.insert(0, model_dir)
from lid import LanguageIdentifier

lid = LanguageIdentifier(model_dir=model_dir)
texts = ["Мин башҡорт телен яратам.", "Сегодня хорошая погода."]
print(lid.predict(texts).tolist())
print(lid.predict_proba(texts))
print(lid.classes.tolist())  # probability-column order

The example imports the repository's Python adapter. For reproducible deployments, pin revision in snapshot_download to a reviewed commit. After downloading, inference runs locally without network access. For a local checkout with model files alongside lid.py, LanguageIdentifier() also works. The source project's model/ layout remains supported.

Training and Evaluation

The model uses a character-level linear classifier trained for the BA/RU decision. Training texts are not distributed in this repository.

The ONNX export was checked against the original classifier for prediction and probability agreement, including Unicode edge cases. These checks validate the export, rather than measure language-identification accuracy. Evaluation in the source project uses weak labels; independently reviewed language-accuracy results are not claimed here.

Limitations

This is a binary classifier, not a general multilingual language detector. It always selects Bashkir or Russian and does not provide an unknown class. Tatar and other languages, short strings, names, OCR artifacts and mixed-language text can be ambiguous. Valid Bashkir text does not always contain Bashkir-specific letters. Empty strings and technical noise should be handled separately when filtering a corpus.

License

The code and model export are released under Apache-2.0. Training texts remain in the source project and are not included here.

Citation

@software{failed09_bashkir_lid,
  title = {Bashkir LID},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-lid}
}

An open-source tool for the Bashkir language.