meadbee/Yuna-130M-Instruct

YunaGPT-124M V1 Instruct

A compact, English-first instruction model fine-tuned from YunaGPT-124M V1 Base.

Important: This is a small experimental model, not a production assistant. It can follow simple instructions but frequently produces incorrect, confused, repetitive, or invented information.

Overview

YunaGPT-124M V1 Instruct is the general instruction-following variant of the Yuna model family. It starts from the pretrained Base checkpoint and applies response-only supervised fine-tuning (SFT): the instruction is visible as context, while training loss is applied to the response and its end-of-text token.

This variant is intended for short, single-turn requests. It is better suited to questions and instructions than the Base model, but its compact size strongly limits its knowledge, reasoning, consistency, and reliability.

Project background

Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's Build a Large Language Model (From Scratch). It later moved to Hugging Face's native LLaMA implementation and grew into an experiment in how far a model could be trained on a home RTX 3090. The broader project also explores synthetic Final Fantasy X data, creative-writing SFT, role-play conversation SFT, and preference optimization.

The model's knowledge of Final Fantasy or any other subject should not be treated as factual. It may combine learned names and concepts with convincing hallucinations.

Model summary

Item	Value
Parameters	124,445,376
Model class	LlamaForCausalLM
Training stage	General instruction SFT
Lineage	Base → Instruct
Context length	2,048 tokens
Vocabulary	24,000 tokens
Tokenizer	Byte-level BPE
Hidden layers	25
Hidden size	576
Attention / KV heads	9 / 3
Weight format	safetensors, FP32
Primary language	English

Prompt format

This checkpoint does not use a standard chat template. It was trained with the following instruction wrapper:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{optional_input}

### Response:

Omit the entire ### Input section when no additional input is needed. Preserve the headings and blank lines for the closest match to training.

Run it yourself

Install the runtime dependencies:

pip install torch transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()


def format_prompt(instruction: str, input_text: str = "") -> str:
    prompt = (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        f"### Instruction:\n{instruction.strip()}"
    )
    if input_text.strip():
        prompt += f"\n\n### Input:\n{input_text.strip()}"
    return prompt + "\n\n### Response:\n"


prompt = format_prompt("Explain why the sky appears blue in two sentences.")
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

Replace the placeholder repository name with the final Hugging Face model ID or a local folder. The generation settings are starting points, not validated optimal values.

Architecture

Component	Configuration
Architecture	Decoder-only Transformer
Attention	Grouped-Query Attention (GQA)
Hidden size	576
Intermediate size	2,048
Layers	25
Attention heads	9
Key/value heads	3
Head dimension	64
Activation	SiLU / SwiGLU feed-forward blocks
Normalization	RMSNorm, epsilon 1e-6
Position encoding	RoPE, theta 10,000
Maximum positions	2,048
Input/output embeddings	Tied

Training

The instruction stage was configured for four epochs with a batch size of 1 and a peak learning rate of 5e-5. Examples were filtered for length and quality, deduplicated, and trained with prompt masking so only the assistant response and EOS target contributed to the loss.

The general instruction mixture was built from:

• HuggingFaceH4/no_robots;
• databricks/databricks-dolly-15k;
• the self_instruct portion of HuggingFaceH4/helpful_instructions.

Programming-heavy and explicit mathematics prompts were intentionally filtered because this model was not designed as a coding or math specialist. Dataset names are listed for provenance; their individual licenses, terms, and attribution requirements still apply.

Intended uses

• Simple single-turn instruction-following experiments.
• Educational study of supervised fine-tuning on a compact model.
• Local prototyping with human review.
• A starting point for additional task-specific fine-tuning.

Limitations and safety

Expected limitations include:

• hallucinated facts, names, quotations, and numbers;
• weak reasoning, arithmetic, coding, and multi-step planning;
• inconsistent instruction following and requested-length control;
• repetition, topic drift, malformed answers, and abrupt endings;
• no persistent memory or reliable multi-turn chat behavior;
• unreliable multilingual performance;
• possible biased, offensive, sexual, or otherwise unsafe generations inherited from source data;
• possible reproduction of information or phrases present in the training data.

Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions. Do not deploy it as an unsupervised public-facing assistant. Verify important claims using trustworthy external sources.

Evaluation status

No standardized capability, factuality, bias, toxicity, privacy, or safety benchmarks are included with this release. The model author's informal assessment was approximately 3/10 for overall assistant quality; this is a candid subjective impression, not a benchmark result.

Related variants

• Base: raw next-token completion checkpoint.
• Story: creative-writing SFT branch using the instruction wrapper.
• Conversation: role-play dialogue variant continued from Story and using a different prompt format.

License and attribution

No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' terms or attribution requirements.

---

YunaGPT-124M V1 Instruct is an experimental research model. Use its responses with human review.