Sparse-AST-BWM

Sparse-AST-BWM is a family of byte-level causal language models for Blender Python scripting and procedural 3D work. The models are trained for code and text involving bpy, mathutils, bmesh, spatial mathematics, and computational geometry.

What this model does

The model predicts the next byte in a sequence. It can assist with Blender Python code completion, procedural scene scripts, mesh operations, transforms, cameras, materials, and geometry-related code. It is a research model and generated scripts should be checked by running them in Blender before use.

Available model variants

Variant	Layers	Model width	Feed-forward width	Native context
3M-32	4	256	512	64 tokens
10M-32	6	384	768	64 tokens
100M-32	18	704	1,408	32 tokens
200M-32	28	800	1,600	32 tokens

The repository also contains experimental checkpoints and evaluation artifacts. The table describes the published named variants; it is not a claim that they are one merged model.

Architecture

Sparse-AST uses a serial stack of transformer blocks. Each block contains RMS normalization, causal multi-head attention, an up projection, an internal three-way activation route, a down projection, and BMW recurrent memory. The input embedding and output projection are tied.

The internal activation route applies three elementwise representations to the projected activation:

z
sign(z) * log(1 + |z|)
sign(z) * expm1(|z|)

The learned route combines these representations before the down projection. This is an internal activation choice; it is not a router that replaces one serial transformer stage with another.

BMW memory

BMW adds a recurrent state to each block. For hidden output y, state s, and learned decay d:

s_t = s_(t-1) * d + y_t * (1 - d)
output_t = y_t + memory_projection(s_t)

The decay is learned through a positive parameterization and is retained as part of the checkpoint. BMW is present in the published Sparse-AST-BWM models.

family-oldnew-1m/ experimental expansion

family-oldnew-1m/ is a separate experimental checkpoint based on the 100M serial architecture. It creates one legacy branch and one new branch at each of the 18 original serial block locations. The branches are paired locally; they are not interchangeable stages and are not selected by one flat router:

block_i*(x) = alpha_i(x) * block_i_old(x)
             + (1 - alpha_i(x)) * block_i_new(x)

At initialization, every new branch is an exact state-dict copy of its matching old branch, including weights, normalization parameters, router parameters, BMW parameters, decay values, and buffers. Therefore the two branches initially compute the same function. The old branches are frozen during the expansion experiment; new branches and permitted shared parameters are trainable.

This design preserves the exact legacy serial ordering and topology. At step zero, the new branch is initialized with an exact state-dict copy of its matching legacy branch, including BMW parameters, decay values, and buffers. The legacy branches are frozen during the experiment; new branches and permitted shared parameters are trainable. This checkpoint is experimental and does not claim a measured improvement over the published variants.

Connecting family-oldnew-1m with other variants

The safe connection point is the final vocabulary logits. Keep family-oldnew-1m, 100M-32, and 200M-32 as complete independent models, run the same byte-tokenized prompt through each model, and combine their logits before sampling:

combined_logits = w_family * logits_family_oldnew
                + w_100m    * logits_100m
                + w_200m    * logits_200m

The weights should be non-negative and sum to 1. All connected checkpoints must use the same byte vocabulary and compatible output-logit layout. Use the same context window and deterministic evaluation settings when comparing outputs.

Do not splice layers from different variants together. Their depths, widths, recurrent states, and serial block layouts differ, so replacing one model's internal stage with another model's stage is not function-preserving. The old/new pair must remain local to its matching 100M serial block.

If every listed model is evaluated and blended, this is a dense ensemble: every model is computed for every prompt. It should not be described as sparse routing. A genuinely sparse version would require a separately trained router that selects a strict subset of complete models while preserving their individual internal graphs.

Evaluation

Blender prompt smoke test

A five-prompt Blender Python smoke test produced syntactically valid output for 5/5 prompts in the recorded experiment. This test did not execute the generated scenes in Blender and is not a substitute for a Blender benchmark.

BlenderBench text proxy

A local teacher-forced, text-only proxy was run over 27 BlenderBench task files with sequence length 128 using the recorded experimental checkpoint:

Split	Cross-entropy	Top-1 byte accuracy	Top-5 byte accuracy
All tasks	2.9197	38.69%	66.59%
Level 1 camera	2.0387	53.88%	76.28%
Level 2 attribute	3.0071	37.71%	65.80%
Level 3 compositional	2.9425	37.84%	66.19%

These are text/code teacher-forced measurements. BlenderBench image understanding and actual Blender execution were not performed in this local proxy, so these numbers are not a full multimodal or scene-rendering score.

Latest deep OOD evaluation — family-oldnew step 46,375

The latest verified checkpoint was evaluated without changing weights, adding a supervisor scaffold, repairing generated code, or using a vision model. The evaluation is intentionally split into teacher-forced token prediction and free-running task completion, because the former does not establish the latter.

Tier	Test	Result
Near-OOD held-out families	512 examples, 41,497 target tokens	CE 1.2133, top-1 71.34%, top-5 90.05%
BlenderBench text proxy	27 held-out task files	CE 2.9124, top-1 38.93%, top-5 66.73%
Free-running Blender levels	8 held-out tasks, greedy, 32-token cap	syntax 7/8, no-error execution 5/8, semantic completion 0/8

The apparent no-error Blender executions mostly left the factory scene unchanged after repeating import bpy; they are not counted as task successes. The free-running cap was a diagnostic limit, so this result is evidence of an early decoding/composition bottleneck, not proof that longer decoding would never help. Full details and raw traces are in deep_ood_report_step46375.md and the accompanying JSON artifacts.

How many more training tokens?

There is no evidence-based single number that guarantees unseen-task ability. From step 25,000 to step 46,375, near-OOD CE improved from 1.23008 to 1.21329 (1.36% relative) while free-running semantic completion remained 0/8. This means more tokens are still plausibly useful for token modeling, but token CE alone cannot justify claiming improved Blender composition.

The recommended next budget is 5–10 million additional new tokens, with evaluations every 1 million new tokens:

1. Treat the first 2 million as a minimum signal check.
2. Continue to 5 million only if near-OOD CE and top-1 improve and the raw generation no longer collapses to repeated imports.
3. Use 10 million as a research ceiling for this unchanged recipe. If semantic completion is still zero after that, more tokens alone are not justified; test data mixture, decoding, context handling, and the harness.

The checkpoint metadata reports 1,638,208 tokens in its current tracked new/replay run (1,474,368 new curriculum plus 163,840 replay). This is a run-local counter rather than a complete lifetime total because earlier runs used different data regimes. At a strict 128-token-per-step schedule, 5–10M tokens would be roughly 39k–78k optimizer steps; actual step conversion varies with the variable-length semantic examples.

BMW state carry is not yet a reason to change the production recipe: on a small paired sample, 32-token carry improved CE by 0.0191 and top-1 by 2.00pp, whereas 64-token carry changed CE by +0.0004 and top-1 by only +0.54pp. Full 128-token attention remained best.

Graph planner + Q4 decomposition harness (2026-09-13)

family-oldnew-1m was also evaluated inside a constrained, deterministic-first Blender harness:

goal -> dependency graph -> Qwen2.5-Coder-1.5B-Instruct Q4 decomposition
     -> compact opcode -> verified Python prefix + finite suffix grammar
     -> family-oldnew next-byte selection -> validators -> Blender 5.1

Only the complete family-oldnew-1m checkpoint was active (weights=(1,0,0)). The 100M and 200M variants were not loaded or blended, and no layers were spliced. The Sparse prompt was capped at 32 UTF-8 bytes and each generated suffix at 16 byte tokens.

Evaluation path	Result
Q4 directly generating node code	Failed the first create_rod API contract
Q4 opcode + unconstrained family-oldnew	Failed; raw output included Buig_Vector(0)/
Q4 opcode + grammar-constrained family-oldnew bytes	7/7 graph nodes, Blender exit 0, verification score 1.00

The successful run produced a cylinder named Rod, dimensions [0.5, 0.5, 4.0], a material, camera, light, and render. The Q4 opcodes were C:cyl,r.25,d4, N:Rod, and M:RodMaterial.

This is a harness-level constrained-decoding result, not evidence that Sparse-AST independently authored a complete Blender scene. The graph compiler provided verified Python prefixes and masked bytes outside a finite valid suffix grammar. The checkpoint ranked next bytes only within that grammar. Raw unconstrained family-oldnew generation failed on the same basic rod node. The test therefore demonstrates useful constrained participation while leaving open performance on unrestricted Blender-code generation.

Progressive constrained-harness stress test

Testing reused the same loaded Q4 and family-oldnew instances and stopped at the first failed end-to-end verdict:

Request	Graph	Blender verification	Outcome
Steel sphere, radius 1.25	7/7	1.00	Pass
Wood cube, size 3.5	7/7	1.00	Pass
Iron cylinder, radius 0.75, length 6.5	7/7	1.00	Pass
Red plane, size 7.25	7/7	0.67	First break: verifier rejected zero Z thickness

The plane code passed Q4 decomposition, constrained Sparse-AST decoding, AST and API checks, and produced dimensions [7.25, 7.25, 0.0] in Blender. The failure is attributed to the harness's generic nonzero-dimension verifier, which is inappropriate for a planar mesh, rather than to either neural model. No harder cases were run after this first break.

Loading a checkpoint

The repository includes the model implementation and checkpoint subfolders. A typical load pattern is:

from model import SparseAST

model = SparseAST.from_pretrained(
    ".",
    subfolder="100M-32",
    device="cpu",
    target_context=128,
)

Use the original implementation and configuration with the matching checkpoint. The native context values in the table are the trained configurations; extending context is experimental and does not guarantee equivalent quality.

Recommended use

1. Prompt with a precise Blender task and expected Python API.
2. Generate with deterministic settings when reproducibility matters.
3. Inspect the complete script for unsafe or destructive operations.
4. Run it in an isolated Blender project and verify the scene, mesh, materials, and transforms.

Limitations

• This is a byte-level causal model, not a general conversational assistant.
• Generated Python may be incomplete, invalid, or semantically incorrect.
• Quality depends on the variant, context length, prompt format, and Blender version.
• Text-only proxy scores do not establish successful Blender execution or visual correctness.
• The old/new expansion is a research artifact, not a performance guarantee.

Repository contents

• Published Sparse-AST-BWM checkpoint variants in separate subfolders.
• family-oldnew-1m/: experimental function-preserving old/new expansion.
• model.py and configuration files for loading the published models.
• Evaluation reports and scripts for the recorded experiments.

License

Apache-2.0. See the repository files for the applicable license text.

Links

• Sparse-AST-BWM model repository
• BlenderBench dataset