H.G. BERT Small: AI like it's 1899

The year is 1899. It's still a horse and buggy world. Einstein hasn't published his famous annus mirabilis papers setting the foundation for Physics as we understand it today. The world is advancing at a rapid pace roaring into the 1900s. What if AI models were trained in 1899 and given to the best minds of the day? Could there have been alternate paths for discovery on par or perhaps even ahead of where we are in 2026?

This article introduces the new H.G. BERT Small series of models. This is a 22.7M parameter BERT encoder-only model trained on Historical English Books from 1700 - 1899.

The goal of this model is to interact with 19th century literature using only knowledge known as of 1899. Every piece of training data is from before 1900.

The following new models are released as part of this effort. All models have an Apache 2.0 license.

---

Developing the Knowledge Base

In 1899, all knowledge known to humanity is locked up into books. Luckily, these books are now all in the public domain and a dataset can be built that combines literature, philosophy, religion, biology, physics, math and other fields of science. The Library of Congress Classification codes give a good overview on the types of books available.

A Historical English Books dataset was curated with a selection of books from a number of available sources with public domain books.

While a lot of changed, many of the foundational principles of mathematics and other fields are unchanged.

---

Building a Strong Baseline

The next step was to build a 22.7M parameter BERT encoder-only model was trained on this book dataset. Books published from 1700 - 1899 were selected. A tokenizer was built from scratch. Everything is as of 1899.

The model was trained using masked language modeling with the following code.

from datasets import load_from_disk
from transformers import AutoTokenizer
from transformers import BertConfig, BertForMaskedLM

from txtai.pipeline import HFTrainer

# Historical English Books (Knowledge through 1899)
domain = "victorian"
dataset = load_from_disk("datasets/historical-english-books")
dataset = dataset.filter(lambda x: 1700 <= x["year"] < 1900 and x["quality"] >= 0.65)
dataset = dataset.shuffle(seed=42)
epochs = 4 # 5_183_595_455 total tokens

# Standard tokenizer
tokenizer = AutoTokenizer.from_pretrained(f"tokenizers/{domain}")

# Configuration - bert small
config = BertConfig(
       hidden_size=384,
       num_hidden_layers=6,
       num_attention_heads=6,
       intermediate_size=1536
)

# Model to train
model = BertForMaskedLM(config)

print(config)
print("Total parameters:", sum(p.numel() for p in model.bert.parameters()))

train = HFTrainer()

#
# Train using MLM
#
# Settings copied from original BERT training - override when HF Trainer defaults don't match
#   - BERT Paper (pg. 13): https://arxiv.org/pdf/1810.04805
#   - BERT Tiny Paper: https://arxiv.org/pdf/1908.08962
#   - BERT Parameters: https://github.com/google-research/bert/blob/master/optimization.py#L59
#   - HF Trainer defaults: https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments
train((model, tokenizer), dataset, task="language-modeling", output_dir="output",
       fp16=True, learning_rate=1e-4, per_device_train_batch_size=64, num_train_epochs=epochs,
       warmup_steps=2500, weight_decay=0.01, adam_epsilon=1e-6,
       tokenizers=True, tokenizebatch=10, dataloader_num_workers=20,
       save_strategy="steps", save_steps=5000, logging_steps=500,
)

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

---

Training a Small Embeddings model

Next a sentence-transformers model was fined-tuned to generate vector embeddings.

The training dataset was generating using a random sample of Historical English Books title-abstract pairs and query-sentences pairs.

This model was trained using the following three step process.

• Train an initial model using MultipleNegativesRankingLoss
• Mine hard negatives using the initial model
• Train a new model using the generated dataset with hard negatives

Unlike other models in this small domain model series, this model did not use distillation. This prevents against data leakage and learning modern similarity patterns. This keeps the model focused on 1899 and prior.

The training code is shown below.

import json
import logging

from datasets import Dataset, load_from_disk
from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    models,
    losses
)
from sentence_transformers.util import mine_hard_negatives

logging.basicConfig(level=logging.INFO)

def load(path):
    rows = []
    with open(path, "r", encoding="utf-8") as inputs:
        for line in inputs:
            row = json.loads(line)
            rows.append({
                "query": row[0],
                "document": row[1],
            })

    return rows

def run(domain):
    # Embeddings model
    embeddings = models.Transformer("hgbert-small")

    # Pooling model
    pooling = models.Pooling(embeddings.get_embedding_dimension())

    # Create sentence-transformers model
    model = SentenceTransformer(modules=[embeddings, pooling])

    # Load training data
    train = (
        load(f"training/{domain}-similarity-train.jsonl") +
        load(f"training/{domain}-similarity-train-questions.jsonl") +
        load(f"training/{domain}-similarity-train-sentences.jsonl") +
        load(f"training/{domain}-similarity-train-sentences-questions.jsonl")
    )
    train = Dataset.from_list(train)

    train = mine_hard_negatives(
        dataset=train,
        model=SentenceTransformer(f"{domain}bert-embeddings"),
        range_min=5,
        range_max=100,
        max_score=0.8,
        relative_margin=0.02,
        num_negatives=5,
        sampling_strategy="random",
        batch_size=128,
        use_faiss=True,
        faiss_batch_size=2048,
        output_format="n-tuple",
        #query_prompt_name="query",
        #corpus_prompt_name="document",
    )
    train.save_to_disk("hard-negatives")

    path = f"{domain}bert-embeddings"
    args = SentenceTransformerTrainingArguments(
        output_dir=path,
        num_train_epochs=3,
        per_device_train_batch_size=64,
        per_device_eval_batch_size=64,
        gradient_accumulation_steps=2,
        fp16=True,
        learning_rate=5e-5,
        save_steps=500,
        logging_steps=500,
        dataloader_num_workers=20
    )

    # Create the trainer & start training
    trainer = SentenceTransformerTrainer(
        model=model,
        args=args,
        train_dataset=train,
        loss=losses.MultipleNegativesRankingLoss(model)
    )
    trainer.train()
    model.save(path)

# Train model
run("victorian")

---

Evaluation Results

A BEIR-compatible dataset was generated to facilitate the evaluation process.

Evaluation results are shown below. NDCG is used as the evaluation metric.

This model is a solid performer at a small size. It beats all models by a comfortable margin including ones with 8 billion parameters!

It can be used in CPU-only setups without trading off much on the accuracy front. It shows how small models can excel at specialized domains, requiring less compute and disk space.

---

Wrapping up

This article introduced the new H.G. BERT Small series of models. It's a model frozen in time from 1899. More to come from that alternate timeline in the future!

If you're interested in building custom models like this for your data or domain area, feel free to reach out!

NeuML is the company behind txtai and we provide AI consulting services around our stack. Schedule a meeting or send a message to learn more.

We're also building an easy and secure way to run hosted txtai applications with txtai.cloud.