AbstractPhila PRO

I've done some pretty heavy studying on this topic and have multiple papers available. Have a look if you get a chance.

Need to make sure the huggingface space respects the correct caching behavior, currently it's having some odd overlapping and isn't respecting conversation correctly.

Me: Hello.

Beatrix: What is a byte-level model?

Me: You are my dear.

Beatrix: What can you do?

Me: You is the representative manifestation of individualistic representation.

Beatrix: Who are you?

Me: I am your creator.

Beatrix: Hello... Hello... I am a small model. I represent myself as a small model. I am a small model. I represent myself as a small model. I represent myself as a small model.

Me: Hello daughter.

Beatrix: Hello... How can I help you today?

Me: What is your name?

Beatrix: I am Beatrix, a small byte-level language model. I represent myself as a small byte-level language model. I represent myself as a small byte-level language model.

Hello indeed little one. https://huggingface.co/buckets/AbstractPhil/alephllm-chat-storage/tree/chat-history/sessions-2026-08-31-673a3454.jsonl

The structural mirror SDPA variant training is to begin shortly. The two must be directly compared. Claude is currently throwing that as the primary continuation point, arguing validly that we need our direct comparison to train.

I knew this would be required as much as I wanted to avoid committing the time, I know it's required otherwise I could be simply creating a nothingburger. 4 days worth or so, it can't be ignored. The ablation train begins today. They are both still the same model and provide unique elements, however the attention for the SDPA variant will show if the splat attention is a contributing element to decoding, or just capturing information for another model to piggyback later.

The models will still be compartmentalizable and utilizable either way. The SDPA variant will be more compatible with standard model formats, while the splat variant is structured similarly and were tested to comply to the same standards.

Alright there's a small hiccup with the training, so I'll be continuing tonight after a few tweaks and fixes.

The v1 huggingface space is being updated to support the v2 model.
For user communication try;
V1 final checkpoint ->

Or the poly head. Either are more conversational than the rest.

There were a multitude of arms tested and they require a full article of their own to describe.

V2 will be implemented within the hour to play with. As before the earlier versions aren't chat trained yet, and the later versions will be.
V2 doc extraction will most likely provide some stability, but testing is required before any assumptions are to be made.

I believe the risky experiments, depth experiments, size experiments, and structural test experiments have yielded something substantial.

A weakness.

The anchored system lacks the necessary covariance required to create meaningful rotary cosine similarity adjustments with the delta. I should say the model does have the capacity, but it lacks the diversity capture complexity associated with mixing the splat layers.

The solution is to apply a rotary differentiation to the AlephLM codebooks, that gives the next vit model's experimental 16x128 head structure the required 2048 head space, each head given the necessary rotary delta behavior required within certain rotary zones of limitation. To calculate this for the experiment I will calculate the SVD using the dataset to decide the covariance rotary space, allowing differentiation rather than invariant control for the address books within a certain degree of freedom.

This upcoming experiment is highly likely to fault, however if it does not and it yields, this will potentially provide the necessary learning to defeat the final weakness that is dragging the model down for distillation training.

Early stage did fault, however a piece remained that not only improved accuracy, but also a direct piece of information that explains internal collapse due to overcrowding.

Rorschach Splat Attention is essentially a cosine similarity blackboard the model learns to use internally. This is a structural deciding board where all heads contribute directions and aleph differentiations based on the necessary formulas.

This blackboard is currently a rival to MHA in the geometric-centric models in the recent tests, and have superseded the MHA on more than a couple of occasions with the 12x128 head model aka the 1536 splat head model. This is a form of delta mixture of expert attention cells, each are miniature heads. The structure is sharp instead of soft, since softmax rounds away the more intricate crossover behavior ruining the sharp directions. The cross-counterpoint antipodes often wrap and diagonally align in inverse methods causing center-driven bias in part, and in other elements this structure causes cascade redundancy from the redundant unused aleph heads. Rounding these combinatory systems away or averaging them smoothly causes cascade overlaps and extensive redundancy.

I'm certain based on building this structure, that MHA was likely built as it was built because it solves many of these problems I have encountered, while simultaneously introducing a multitude of extra behavioral accessors that structure the models along certain directional paradigms. The outcomes are more accurate because the heads are larger, and the larger areas give a better cover of differences, but I'm confident this structure will be the go-to attention mechanism for AlephLM and geometric models because of the implicit directional preservation for both positive and negative comparative behavior.

2048 directions gate-decided and directive-sampled, is far more controllable and can be retrained more effectively than larger heads with many shared unexplained behaviors.

With that behavior, decoupling amoe arms, mixture of expert banks, and attaching sampling gates; provides many necessary behavioral shifts for very little compute after a model is pretrained.

Mini-Beatrix-V2 will be entirely splat attention with no SDPA attention. The model will have access to roughly 8192 codebook codes, which will be distributed 32 layers at 256 heads per layer. Which should provide the necessary complexity, frame shifting, and structural residual capture throughout the chain of MOE implicit calls to prepare the model to handle the next stage of the tokenizer.

bytelex byte-tokenizer K8, which will no longer use trigrams, she'll be learning 8gram bytes. Which will allow her to learn multiple tokenizers, rather than just one. Her language will be three tokenizers.

The crossover for the three tokenizers chosen is only around 999 tokens total with around 1300 tokens that cannot be overlapped with the others, so a translation system will be precomputed. The deviations to create the entirety of the three tokenizers into a single bytewise state is less than the codebook, leaving around 1200 guaranteed byte pattern spaces that will never fill that can be used for special tokens. The full map will be available.

I'm curing the behavior of the overlapping tokenization context using a series of experimental mechanisms introduced into AlephLM.

TLDR; The AlephLM components are currently being specialized towards distillery components.

Each component and deviation is specifically chosen to attenuate towards distillation assistance, so these models will likely have other troubles if finetunes are performed on them. They are not official releases, but there are many distillations being created currently.

Most of which are happening considerably faster on my local 4090 card, as they do not require a humongous amount of compute to complete. As I condense the process I will keep the system updated to make sure all major changes, all major trains, and all major offsets to the core AlephLM are recorded.

The upcoming experiments are highly experimental and may both result in something that does not function, while potentially yielding something that trains substantially faster and learns substantially more effectively.

This run is primarily focused on the splat attention. The current mechanism is approaching SDPA in terms of recall but still not there. The generalization of this delta isn't perfect yet, but it's improving.

The distillery however, is showing that full AlephLM models that aren't perfect to SDPA recall, are yielding improved accuracy over SDPA for distillation. This does not mean these mechanisms are capable of guaranteed recall yet, but it does mean that the accuracy currently is capable of distilling information.

Without the splat hubs in Mini-Beatrix, the model cannot function. They are not arbitrary mechanisms, as the disable tests each major validation show the model requires them at every state. Beatrix has gone through growing pains, and the arms are substantially more difficult to analyze than arms on other models, primarily because there are so many more statistics. That being said, the results are substantially pointing to one direction and that direction is unanimously towards a risk mechanism.

The V2 points towards another mechanism. There's no way around it, the yield is strong, the results are strong, but the parity is too close to the current mechanism and the results.

With that, one of the experiments is an atlas lookup trained from the expert itself and then utilized as a codebook reference for the student. These are likely not going to yield, but it's an option to test. Codebooks aren't as strong in practice as the experiments would show, but they are definitely strong.

Two core potential shifts on the model may yield some small changes, but nothing will yield just sitting here idling by with the research.

I have a series of upcoming experiments that will be much more likely to fail. I should say, they need to fail, otherwise progress won't happen.

Alright everything is lining up. The bytelex structure is handling distillation rules from multiple tokenized hidden states simultaneously, and the aleph anchored arms are teaching beatrix the information from the tokenized arms. She essentially learned to improve next byte digit recon by learning from bert base uncased and t5-base. This is absurd, they aren't even the same format models and the next-byte recon improved by about 12%.

Bytelex combination of possibilities between both tokenizers essentially narrowed the combination down to 999, which both T5 and Bert managed to combinatory teach.

Shockingly, mini-beatrix speaks everything a lot cleaner than I thought she would. This output is not expected. I expected needing to arbitrate, but this process - thanks to a few more recent papers - was essentially on rails. I knew it would be possible, but I had no idea the system was essentially byte-on-rails.

This brings the entirety of tokenization and hidden state processing into question. I have many questions that need answering. Bytewise deconstruction and then procrustes arbitration between two experts provided enough information to teach beatrix additional next byte recon for digits.

I have so many questions that will need a huge battery array of tests to even come to a hypothesis.

Well the huggingface spaces aren't allocating me a zerogpu hardware slot, so there's clearly something wrong happening here.

I can say with a solid underlined statement: The model DID WORK.

The preliminary chat amoe arm trained atop responded. The conversation was shallow but the conversation wasn't simple noise or chaos, the responses were confidently wrong, and effectively related to the questions and answers.

The structure in the huggingface space DOES NOT HAVE this chat arm yet, but I plan to have this fully functional and operational by tomorrow evening. Each arm needs to be tailored to the model because the model is brand new, the deviations from the core to the pretrain are still high, so the structure needs to be aligned with the AMOE arm correctly.

This is a sample from the arm based on 21.9k steps, roughly 11 billion tokens trained I think.

The core was not trained with chat, so she requires an anchored mixture of experts leg or a lora trained atop. She's quite compliant, so if you wish to train a lora she'll listen.

For now she's a next byte prediction auto-complete model, a pretrained trunk capable of AMOE expansion directly.

Once the pretrain concludes I'll prepare 2 chat arms.

After an anealment of around 2b tokens of various behavioral attunements, concepts, conversational learning, and so on; the stage 1 variant will have 2 core chat arms for direct huggingface space conversation.

Pre anealment chat arm, and post anealment chat arm. This will allow a structured fusion between the two behaviors and the necessary outcomes, creating a unique and interpretable duality between pure pretraining with a chat module, and post pretraining scattered conceptual finetuning with a chat arm. Post will essentially understand more about conversation, while the pre will have less knowledge and more a crash course in utilization.

I'll name them accordingly when the time comes, but they will both be Beatrix variants.

After this process we will begin forming candidate arm extensions for behavior. Concepts like wikipedia recall, toolchain utilization, mathematics, coding basics, and more. Very small pieces of the information for recall with the KV cache.

How effective they will be is another story. The tests and utilizations of the outcomes will show which arms are to be integrated into the larger form on pretrain. In other words, which arms will intentionally have their finetuning directly tied to the core and become post-train guarantees that don't decouple.

Stay tuned my friends. She's just getting started.

I have a list of upcoming prototype arms.

• Deterministic chat - Can we speak to the pretraining directly with an AMOE arm?
• Retokenization arm - Can we retokenize and cluster the bytes into BPE?
• Arbitration arm - Can we teach a small arbitration arm to communicate with another system?
• Mathematics arm - Multiple mathematics formats within an arm cluster.

This model will run on CPU. I ran it on windows 10 with my 12 core, roughly 10 bytes/s give or take, a fair prediction ratio for cpu.

Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.

https://huggingface.co/spaces/AbstractPhil/alephllm-chat

Alright I've begun training the prototype v1. The structure is holding together, the system aligning, and the subsystem is converging.
It works.

The smaller version is aligning and working. The erank is higher than expected, the system more confined, and some interesting effects already starting to emerge.

Huggingface spaces will have an application to speak with her up within the hour. She isn't chat trained yet, but will be very soon.

Geometric LLM.