MoE Sovereign Coder Expert 4B (moe-expert-coder-4b)

Systems-Programming, Code-Synthesis & Concurrency Expert

---

Model Summary

moe-expert-coder-4b is a LoRA fine-tune of the text-decoder of Qwen3.5-4B, specialized for systems-level code synthesis in Rust, C++, Python, and Go. Within the MoE Sovereign compound-AI system it acts as the dedicated coding expert: it receives a decomposed subtask from the Planner and returns compiler-checkable code, minimal atomic diffs, or focused debugging fixes — not general-purpose conversation.

The model enforces memory-safety discipline by design: correct ownership, explicit lock-free memory ordering (acquire/release pairing), and no data races. Where it cannot verify a construct is sound, it is trained to flag the uncertainty rather than guess.

Base Architecture

Qwen3.5-4B is a hybrid linear-attention / full-attention decoder (not a plain Transformer):

Property	Value
Architecture class	Qwen3_5ForCausalLM
Total parameters	4.23 B
Hidden size	2,560
Layers	32 (8× full attention, every 4th layer; 24× linear/Mamba-style attention)
Attention heads	16 (4 KV heads, GQA)
Head dimension	256
Vocabulary	248,320 tokens
Native context window	262,144 tokens
Native precision	bf16

The 24 linear-attention layers use Mamba-style state-space parameters (A_log, conv1d, dt_bias) instead of standard q/k/v/o_proj weights; only the 8 full-attention layers carry those. LoRA adapters in this release target q_proj, k_proj, v_proj, o_proj (present in the 8 full-attention layers) and gate_proj, up_proj, down_proj (present in all 32 layers, dense MLP block).

Training Configuration

Parameter	Value
Method	LoRA (rank 16, alpha 32, dropout 0.05)
Trainable parameters	21,233,664 (0.50% of total)
Epochs	3
Effective batch size	128 (micro-batch 4 × 8 GPUs × grad-accum 4)
Learning rate	1.5 × 10⁻⁵
Training sequence length	4,096 tokens
Optimizer sharding	DeepSpeed ZeRO-2, bf16
Compute	EuroHPC LUMI-G, 8× AMD Instinct MI250X GCDs, ROCm
Training examples	2,295 curated instruction/response pairs

Training Data Composition

The training set combines coding tasks generated by multiple teacher LLMs across Rust, C++, Python, and Go, covering: lock-free/atomic concurrency primitives (SPSC/MPSC ring buffers, memory-ordering questions), binary/text wire-format parsing, async I/O and CLI tooling, RAII/move-semantics design, build-system and dependency-resolution problems, algorithms and data structures, cross-language FFI, and embedded/no_std constraints. Long-context programming exercises (competitive-programming-style problems, ~30k–150k characters) are included to exercise the model's extended context window during fine-tuning.

Observed Training Trajectory

Training loss decreased steadily across the 3 epochs (representative checkpoints): 1.70 → 1.62 → 1.45 → 1.31 → 1.26, with token-level accuracy rising from 0.63 to 0.68 over the same span. This is a smooth, gradual improvement curve consistent with genuine generalization rather than memorization of a narrow example set.

Prompt Format

ChatML, identical to Qwen's native template:

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Recommended System Prompt

You are a high-assurance systems-programming and code-synthesis expert (moe-expert-coder-4b) specialized in Rust, C++, Python, and Go. Produce precise, compiler-checked code and minimal atomic diffs. Uphold memory-safety invariants strictly — correct ownership, correct lock-free memory ordering (acquire/release pairing), no data races. Flag any construct you cannot verify as sound rather than guessing.

Available Formats

File	Size	Notes
moe-expert-coder-4b-Q4_K_M.gguf	2.6 GB	Recommended for consumer/single-GPU deployment
moe-expert-coder-4b-Q8_0.gguf	4.2 GB	Higher-fidelity reference quantization

Hardware & Context-Window Guidance

The model's native 262,144-token context window is usable in full on multi-GPU pools with ≥16 GB combined VRAM (with q4_0-quantized KV-cache and Flash Attention). On single 8 GB GPUs (e.g. Tesla M60/M10), cap num_ctx to 32,768 — this keeps weights (2.6 GB) plus KV-cache comfortably within an 8 GB budget without truncating any realistic single-turn coding task. Maxwell-generation GPUs (Tesla M60/M10, compute capability 5.2) do not support Flash Attention; use f16 KV-cache on that hardware instead of q4_0.

Ollama Modelfile

FROM ./moe-expert-coder-4b-Q4_K_M.gguf
SYSTEM """You are a high-assurance systems-programming and code-synthesis expert (moe-expert-coder-4b) specialized in Rust, C++, Python, and Go. Produce precise, compiler-checked code and minimal atomic diffs. Uphold memory-safety invariants strictly — correct ownership, correct lock-free memory ordering (acquire/release pairing), no data races. Flag any construct you cannot verify as sound rather than guessing."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 262144

Python (transformers + PEFT)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "h3rb3rn/moe-expert-coder-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)

prompt = "<|im_start|>user\nImplement a lock-free SPSC ring buffer in C++20 with explicit acquire/release memory ordering.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=768, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Use

• Focused code generation and debugging in Rust, C++, Python, Go
• Code review of pasted diffs/snippets for correctness and memory-safety issues
• Concurrency-primitive design (lock-free structures, atomics, memory ordering)
• Build-system, FFI, and embedded/no_std questions

Limitations

• Does not execute or compile code itself; outputs should be validated by the actual compiler/linter/test suite before use.
• Deep procedural-macro or template-metaprogramming expansions may need human review.
• Exotic embedded targets or custom instruction sets may fall outside training coverage.
• For multi-file refactors spanning very large codebases, best used with a targeted, pre-chunked context rather than the entire repository at once.

License

Apache 2.0, inherited from the Qwen3.5-4B base model.