TULLUS/codeparrot-small-multi 🦜

A clean, verified Safetensors conversion of codeparrot/codeparrot-small-multi.

This repository preserves the original model weights while normalizing the legacy GPT-2 embedding representation from two physically duplicated matrices into properly tied input/output embeddings.

What this is

This is a checkpoint conversion, not a new training run.

The original CodeParrot checkpoint was loaded from its legacy pytorch_model.bin, inspected, normalized, saved as Safetensors, reloaded, and then subjected to exact tensor verification.

The goal was to produce a clean modern Transformers checkpoint without changing the learned weight values.

Source

Original model: codeparrot/codeparrot-small-multi

Original architecture: GPT-2 / GPT2LMHeadModel

Original tokenizer: GPT-2 tokenizer

Vocabulary size: 32,768

BOS token ID: 0

EOS token ID: 0

PAD token: none

Original checkpoint SHA256:

0207f6b427e3cbf1bcb9726abb6bbba6620e9912e1abf23087ccdef61818ffb2

Conversion details

The legacy checkpoint contained two separate physical copies of the embedding matrix:

transformer.wte.weight
lm_head.weight

Both tensors were verified to have identical values:

shape:            (32768, 768)
exact equality:   True
max difference:   0.0
same storage:     False

The model was then normalized with:

tie_word_embeddings = True

After normalization:

same storage:      True
exact equality:    True

The resulting checkpoint therefore uses a single shared embedding tensor for the input embeddings and language-model head.

Parameter count

The legacy checkpoint physically stored:

136,174,080 parameters

This included the duplicated embedding storage.

After tying the identical embeddings, the clean model contains:

111,008,256 unique parameters

The resulting Safetensors file is approximately:

444,048,000 bytes
423.48 MiB

Verification

The conversion was validated after saving and reloading the Safetensors checkpoint.

Verified checks include:

• Original source SHA256
• Source tokenizer length
• BOS/EOS token IDs
• Source embedding values
• Source physical parameter count
• Embedding storage normalization
• Safetensors serialization
• Reload into GPT2LMHeadModel
• tie_word_embeddings=True
• Final tied embedding storage
• Metadata consistency
• Exact tensor equality

Final verification result:

Weights: EXACTLY PRESERVED
Embedding configuration: TIED
Unique parameter count: 111,008,256
BOS/EOS IDs: 0 / 0
Exact tensor verification: PASS

The final comparison verified all 149 logical model tensors after reload. Safetensors physically stores 148 tensors because the tied lm_head.weight is represented by the shared transformer.wte.weight storage.

About the conversion warning

During loading of the original legacy checkpoint, Transformers reported the following unexpected entries:

transformer.h.{0...11}.attn.bias
transformer.h.{0...11}.attn.masked_bias

These are legacy GPT-2 attention-mask buffers rather than learned model weights. They do not represent missing trained parameters, and the complete learned tensor set was subsequently verified exactly after conversion and reload.

Intended use

This checkpoint is intended as a clean starting point for experimentation with the CodeParrot GPT-2 architecture, including further fine-tuning and research experiments.

It is also the clean base checkpoint for the planned:

TULLUS-CodeParrot-???

training experiment.

Example usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TULLUS/codeparrot-small-multi"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "def fibonacci(n):"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Provenance

This repository is a conversion of the original:

codeparrot/codeparrot-small-multi

No claim is made here that the converted checkpoint is a new trained model. The purpose of this repository is to provide a clean Safetensors representation with normalized tied embeddings while preserving the original learned values.

---

TULLUS BAKES 🍪

Fresh From The AI Ovens, It's The Goods..