faodl/agri-production-classifier

AMIS Commodity Classifier

This model repository contains artifacts from an AMIS commodity relevance classifier training run. It includes the Transformer model, any configured TF-IDF or sentence-embedding baselines, prediction files, and the training report.

• Dataset: faodl/amis-agri-production-v03
• Dataset subset: ``
• Text column: chunk_text
• Label column: label
• Transformer: FacebookAI/xlm-roberta-base
• Generated at: 2026-05-19T18:39:45.207640+00:00

Dataset Summary

Split	Rows	Label 0	Label 1	Unique groups	Mean text length
train	5937	4830	1107		670.8
validation	1280	1011	269		678.0
test	1388	1106	282		659.2

Threshold Comparison on Test Split

Model	Threshold	Accuracy	Precision	Recall	F1	ROC AUC	Average precision
logistic_tfidf	0.500	0.899	0.741	0.773	0.757	0.934	0.813
logistic_tfidf	0.479	0.897	0.729	0.784	0.756	0.934	0.813
xgboost_tfidf	0.500	0.898	0.872	0.582	0.698	0.916	0.791
xgboost_tfidf	0.197	0.878	0.674	0.777	0.722	0.916	0.791
embedding-logistic_sentence_embeddings	0.500	0.880	0.646	0.904	0.753	0.949	0.812
embedding-logistic_sentence_embeddings	0.505	0.880	0.647	0.904	0.754	0.949	0.812
embedding-svm_sentence_embeddings	0.500	0.885	0.728	0.691	0.709	0.947	0.807
embedding-svm_sentence_embeddings	0.261	0.883	0.663	0.865	0.751	0.947	0.807
embedding-lightgbm_sentence_embeddings	0.500	0.890	0.741	0.709	0.725	0.950	0.810
embedding-lightgbm_sentence_embeddings	0.161	0.896	0.704	0.837	0.765	0.950	0.810
transformer	0.500	0.901	0.733	0.809	0.769	0.957	0.827
transformer	0.376	0.896	0.696	0.869	0.773	0.957	0.827

Confusion Matrices on Test Split

Rows are true labels and columns are predicted labels.

logistic_tfidf at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1030	76
RELEVANT	64	218

logistic_tfidf at threshold 0.479

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1024	82
RELEVANT	61	221

xgboost_tfidf at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1082	24
RELEVANT	118	164

xgboost_tfidf at threshold 0.197

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1000	106
RELEVANT	63	219

embedding-logistic_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	966	140
RELEVANT	27	255

embedding-logistic_sentence_embeddings at threshold 0.505

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	967	139
RELEVANT	27	255

embedding-svm_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1033	73
RELEVANT	87	195

embedding-svm_sentence_embeddings at threshold 0.261

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	982	124
RELEVANT	38	244

embedding-lightgbm_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1036	70
RELEVANT	82	200

embedding-lightgbm_sentence_embeddings at threshold 0.161

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1007	99
RELEVANT	46	236

transformer at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	1023	83
RELEVANT	54	228

transformer at threshold 0.376

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	999	107
RELEVANT	37	245

Validation-Tuned Thresholds

• logistic_tfidf: threshold 0.479 (validation F1 0.778); test F1 change vs 0.5: -0.001.
• xgboost_tfidf: threshold 0.197 (validation F1 0.717); test F1 change vs 0.5: +0.024.
• embedding-logistic_sentence_embeddings: threshold 0.505 (validation F1 0.797); test F1 change vs 0.5: +0.001.
• embedding-svm_sentence_embeddings: threshold 0.261 (validation F1 0.785); test F1 change vs 0.5: +0.042.
• embedding-lightgbm_sentence_embeddings: threshold 0.161 (validation F1 0.815); test F1 change vs 0.5: +0.040.
• transformer: threshold 0.376 (validation F1 0.822); test F1 change vs 0.5: +0.004.

Artifacts

• logistic_tfidf: /content/agri-production-classifier/baselines/logistic
• xgboost_tfidf: /content/agri-production-classifier/baselines/xgboost
• embedding-logistic_sentence_embeddings: /content/agri-production-classifier/baselines/embedding-logistic
• embedding-svm_sentence_embeddings: /content/agri-production-classifier/baselines/embedding-svm
• embedding-lightgbm_sentence_embeddings: /content/agri-production-classifier/baselines/embedding-lightgbm
• transformer: /content/agri-production-classifier/transformer

Inference

Install the runtime dependencies:

pip install transformers torch huggingface_hub pandas joblib scikit-learn xgboost sentence-transformers lightgbm

Transformer

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "faodl/agri-production-classifier"

texts = [
    "Rice export prices increased after new procurement rules were announced.",
    "The finance ministry released its monthly fuel tax bulletin.",
]

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, subfolder="transformer")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, subfolder="transformer")
threshold = float(getattr(model.config, "threshold", 0.5))

encoded = tokenizer(
    texts,
    truncation=True,
    padding=True,
    max_length=256,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**encoded).logits
    probabilities = torch.softmax(logits, dim=-1)[:, 1].tolist()

for text, probability in zip(texts, probabilities):
    label = model.config.id2label[int(probability >= threshold)]
    print({"text": text, "probability_positive": probability, "label": label})

TF-IDF Baselines

Available baseline names in this run: "logistic", "xgboost".

import json
import joblib
from huggingface_hub import hf_hub_download

MODEL_ID = "faodl/agri-production-classifier"
BASELINE = "logistic"

texts = [
    "Maize production forecasts were revised after delayed rains.",
    "The central bank published new exchange rate statistics.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}_tfidf.joblib",
)
report_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename="report.json",
)

pipeline = joblib.load(model_path)
with open(report_path, encoding="utf-8") as handle:
    report = json.load(handle)

threshold = next(
    result["validation_best_threshold"]["threshold"]
    for result in report["results"]
    if result["model_type"] == f"{BASELINE}_tfidf"
)

probabilities = pipeline.predict_proba(texts)[:, 1]
for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})

Sentence-Embedding Baselines

Available embedding baseline names in this run: "embedding-logistic", "embedding-svm", "embedding-lightgbm".

import joblib
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer

MODEL_ID = "faodl/agri-production-classifier"
BASELINE = "embedding-logistic"

texts = [
    "Wheat export inspections rose as demand from importers increased.",
    "The sports ministry announced a new stadium renovation plan.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}.joblib",
)
artifact = joblib.load(model_path)
embedding_model = SentenceTransformer(artifact["embedding_model_name"])
embeddings = embedding_model.encode(
    texts,
    batch_size=artifact.get("embedding_batch_size", 64),
    convert_to_numpy=True,
    normalize_embeddings=artifact.get("normalize_embeddings", True),
)
probabilities = artifact["classifier"].predict_proba(embeddings)[:, 1]
threshold = artifact["validation_best_threshold"]["threshold"]

for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})

Files

• REPORT.md: Markdown report for this training run.
• report.json: Machine-readable report containing metrics and thresholds.
• transformer/: Fine-tuned Transformer artifacts, when Transformer training is enabled.
• baselines/: TF-IDF and sentence-embedding baseline artifacts, when baseline training is enabled.
• */validation_predictions.csv and */test_predictions.csv: Split-level predictions.