AbstractPhila PRO

Alright there's a small hiccup with the training, so I'll be continuing tonight after a few tweaks and fixes.

The v1 huggingface space is being updated to support the v2 model.
For user communication try;
V1 final checkpoint ->

Or the poly head. Either are more conversational than the rest.

There were a multitude of arms tested and they require a full article of their own to describe.

V2 will be implemented within the hour to play with. As before the earlier versions aren't chat trained yet, and the later versions will be.
V2 doc extraction will most likely provide some stability, but testing is required before any assumptions are to be made.

I believe the risky experiments, depth experiments, size experiments, and structural test experiments have yielded something substantial.

A weakness.

The anchored system lacks the necessary covariance required to create meaningful rotary cosine similarity adjustments with the delta. I should say the model does have the capacity, but it lacks the diversity capture complexity associated with mixing the splat layers.

The solution is to apply a rotary differentiation to the AlephLM codebooks, that gives the next vit model's experimental 16x128 head structure the required 2048 head space, each head given the necessary rotary delta behavior required within certain rotary zones of limitation. To calculate this for the experiment I will calculate the SVD using the dataset to decide the covariance rotary space, allowing differentiation rather than invariant control for the address books within a certain degree of freedom.

This upcoming experiment is highly likely to fault, however if it does not and it yields, this will potentially provide the necessary learning to defeat the final weakness that is dragging the model down for distillation training.

Early stage did fault, however a piece remained that not only improved accuracy, but also a direct piece of information that explains internal collapse due to overcrowding.

Rorschach Splat Attention is essentially a cosine similarity blackboard the model learns to use internally. This is a structural deciding board where all heads contribute directions and aleph differentiations based on the necessary formulas.

This blackboard is currently a rival to MHA in the geometric-centric models in the recent tests, and have superseded the MHA on more than a couple of occasions with the 12x128 head model aka the 1536 splat head model. This is a form of delta mixture of expert attention cells, each are miniature heads. The structure is sharp instead of soft, since softmax rounds away the more intricate crossover behavior ruining the sharp directions. The cross-counterpoint antipodes often wrap and diagonally align in inverse methods causing center-driven bias in part, and in other elements this structure causes cascade redundancy from the redundant unused aleph heads. Rounding these combinatory systems away or averaging them smoothly causes cascade overlaps and extensive redundancy.

I'm certain based on building this structure, that MHA was likely built as it was built because it solves many of these problems I have encountered, while simultaneously introducing a multitude of extra behavioral accessors that structure the models along certain directional paradigms. The outcomes are more accurate because the heads are larger, and the larger areas give a better cover of differences, but I'm confident this structure will be the go-to attention mechanism for AlephLM and geometric models because of the implicit directional preservation for both positive and negative comparative behavior.

2048 directions gate-decided and directive-sampled, is far more controllable and can be retrained more effectively than larger heads with many shared unexplained behaviors.

With that behavior, decoupling amoe arms, mixture of expert banks, and attaching sampling gates; provides many necessary behavioral shifts for very little compute after a model is pretrained.

Mini-Beatrix-V2 will be entirely splat attention with no SDPA attention. The model will have access to roughly 8192 codebook codes, which will be distributed 32 layers at 256 heads per layer. Which should provide the necessary complexity, frame shifting, and structural residual capture throughout the chain of MOE implicit calls to prepare the model to handle the next stage of the tokenizer.

bytelex byte-tokenizer K8, which will no longer use trigrams, she'll be learning 8gram bytes. Which will allow her to learn multiple tokenizers, rather than just one. Her language will be three tokenizers.

The crossover for the three tokenizers chosen is only around 999 tokens total with around 1300 tokens that cannot be overlapped with the others, so a translation system will be precomputed. The deviations to create the entirety of the three tokenizers into a single bytewise state is less than the codebook, leaving around 1200 guaranteed byte pattern spaces that will never fill that can be used for special tokens. The full map will be available.

I'm curing the behavior of the overlapping tokenization context using a series of experimental mechanisms introduced into AlephLM.

TLDR; The AlephLM components are currently being specialized towards distillery components.

Each component and deviation is specifically chosen to attenuate towards distillation assistance, so these models will likely have other troubles if finetunes are performed on them. They are not official releases, but there are many distillations being created currently.

Most of which are happening considerably faster on my local 4090 card, as they do not require a humongous amount of compute to complete. As I condense the process I will keep the system updated to make sure all major changes, all major trains, and all major offsets to the core AlephLM are recorded.

The upcoming experiments are highly experimental and may both result in something that does not function, while potentially yielding something that trains substantially faster and learns substantially more effectively.

This run is primarily focused on the splat attention. The current mechanism is approaching SDPA in terms of recall but still not there. The generalization of this delta isn't perfect yet, but it's improving.

The distillery however, is showing that full AlephLM models that aren't perfect to SDPA recall, are yielding improved accuracy over SDPA for distillation. This does not mean these mechanisms are capable of guaranteed recall yet, but it does mean that the accuracy currently is capable of distilling information.

Without the splat hubs in Mini-Beatrix, the model cannot function. They are not arbitrary mechanisms, as the disable tests each major validation show the model requires them at every state. Beatrix has gone through growing pains, and the arms are substantially more difficult to analyze than arms on other models, primarily because there are so many more statistics. That being said, the results are substantially pointing to one direction and that direction is unanimously towards a risk mechanism.

The V2 points towards another mechanism. There's no way around it, the yield is strong, the results are strong, but the parity is too close to the current mechanism and the results.

With that, one of the experiments is an atlas lookup trained from the expert itself and then utilized as a codebook reference for the student. These are likely not going to yield, but it's an option to test. Codebooks aren't as strong in practice as the experiments would show, but they are definitely strong.

Two core potential shifts on the model may yield some small changes, but nothing will yield just sitting here idling by with the research.

I have a series of upcoming experiments that will be much more likely to fail. I should say, they need to fail, otherwise progress won't happen.

Alright everything is lining up. The bytelex structure is handling distillation rules from multiple tokenized hidden states simultaneously, and the aleph anchored arms are teaching beatrix the information from the tokenized arms. She essentially learned to improve next byte digit recon by learning from bert base uncased and t5-base. This is absurd, they aren't even the same format models and the next-byte recon improved by about 12%.

Bytelex combination of possibilities between both tokenizers essentially narrowed the combination down to 999, which both T5 and Bert managed to combinatory teach.

Shockingly, mini-beatrix speaks everything a lot cleaner than I thought she would. This output is not expected. I expected needing to arbitrate, but this process - thanks to a few more recent papers - was essentially on rails. I knew it would be possible, but I had no idea the system was essentially byte-on-rails.

This brings the entirety of tokenization and hidden state processing into question. I have many questions that need answering. Bytewise deconstruction and then procrustes arbitration between two experts provided enough information to teach beatrix additional next byte recon for digits.

I have so many questions that will need a huge battery array of tests to even come to a hypothesis.

Well the huggingface spaces aren't allocating me a zerogpu hardware slot, so there's clearly something wrong happening here.

I can say with a solid underlined statement: The model DID WORK.

The preliminary chat amoe arm trained atop responded. The conversation was shallow but the conversation wasn't simple noise or chaos, the responses were confidently wrong, and effectively related to the questions and answers.

The structure in the huggingface space DOES NOT HAVE this chat arm yet, but I plan to have this fully functional and operational by tomorrow evening. Each arm needs to be tailored to the model because the model is brand new, the deviations from the core to the pretrain are still high, so the structure needs to be aligned with the AMOE arm correctly.

This is a sample from the arm based on 21.9k steps, roughly 11 billion tokens trained I think.

The core was not trained with chat, so she requires an anchored mixture of experts leg or a lora trained atop. She's quite compliant, so if you wish to train a lora she'll listen.

For now she's a next byte prediction auto-complete model, a pretrained trunk capable of AMOE expansion directly.

Once the pretrain concludes I'll prepare 2 chat arms.

After an anealment of around 2b tokens of various behavioral attunements, concepts, conversational learning, and so on; the stage 1 variant will have 2 core chat arms for direct huggingface space conversation.

Pre anealment chat arm, and post anealment chat arm. This will allow a structured fusion between the two behaviors and the necessary outcomes, creating a unique and interpretable duality between pure pretraining with a chat module, and post pretraining scattered conceptual finetuning with a chat arm. Post will essentially understand more about conversation, while the pre will have less knowledge and more a crash course in utilization.

I'll name them accordingly when the time comes, but they will both be Beatrix variants.

After this process we will begin forming candidate arm extensions for behavior. Concepts like wikipedia recall, toolchain utilization, mathematics, coding basics, and more. Very small pieces of the information for recall with the KV cache.

How effective they will be is another story. The tests and utilizations of the outcomes will show which arms are to be integrated into the larger form on pretrain. In other words, which arms will intentionally have their finetuning directly tied to the core and become post-train guarantees that don't decouple.

Stay tuned my friends. She's just getting started.

I have a list of upcoming prototype arms.

• Deterministic chat - Can we speak to the pretraining directly with an AMOE arm?
• Retokenization arm - Can we retokenize and cluster the bytes into BPE?
• Arbitration arm - Can we teach a small arbitration arm to communicate with another system?
• Mathematics arm - Multiple mathematics formats within an arm cluster.

This model will run on CPU. I ran it on windows 10 with my 12 core, roughly 10 bytes/s give or take, a fair prediction ratio for cpu.

Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.

https://huggingface.co/spaces/AbstractPhil/alephllm-chat

Alright I've begun training the prototype v1. The structure is holding together, the system aligning, and the subsystem is converging.
It works.

The smaller version is aligning and working. The erank is higher than expected, the system more confined, and some interesting effects already starting to emerge.

Huggingface spaces will have an application to speak with her up within the hour. She isn't chat trained yet, but will be very soon.

Geometric LLM.

Dusting up the week and preparing the article. It'll be an interesting read. I'm having Claude attend to a more readable format for this article, and with that the article will have the more technical documentation attached as links rather than interweaved. This should make it a bit more readable, as Claude Fable and Opus 5 are very strangely verbose and it seems many people have trouble reading the articles.

I admit they have gotten odd since Fable and Opus 5 started writing them. I have trouble following and reading the logic for some of the points in FT4, so I'll make this next article cleaner and more readable. I'll attempt to remedy this by having Claude produce a more human-reader friendly pair of articles, which I will directly audit, rewrite, and proofread the public-faced article myself while allowing Claude free reign on the more technical article.

Friday time.

This upcoming week I'll be preparing smaller tests, bulk analysis, structure tweaks, and a series of smaller autoregression trainings that will offer the answers to some pressing questions. I'll be micro managing the next week using Colab instead of automated runpod, so it may be a little more scattered, but Claude Code will keep the docket so it'll line up.

Some of the answers will make sense to the observer, some will make little to no sense at a topical glance. Just know that I'm following many routes of possibility at the same time, so the answers may not make sense at first glance. Having a goal like a large-scale differential homogenous structure with multiple communicative sub-structures speaking a new language that none of their teachers knew, is a tricky system and requires subtle nuances that I simply have to explore, crash the machines, and figure out the reasoning behind why the machines crash. This will require many tests, many analysis, and many wrong hypothesis. I'll try to fire through as many as quickly as possible without sacrificing the scientific method, so the AlephLM's next week iteration will potentially be stronger than this week.

Prototype is set up with a speed check attached to the demo.

As per the experiments successes and failures, rotary is now applied to the address space allowing full codebook differentiation through training pressure. The code for the splat attention is very small and efficient, but the efficiency can be improved with some elbow grease. The readability can definitely be improved.

https://huggingface.co/AbstractPhil/aleph-splat-0/blob/main/splat_attention.py

Lets see if the depth capacity holds up or if the model collapses.

I have found a potentially valid attention mechanism that utilizes the leverage that Aleph Addressing presents, and I have yet to accurately capture until now. I assert, that this addressing and anchoring, presents a hugely increased amount of opinions beyond the current spectrum of explored opinions with shared heads.

https://huggingface.co/AbstractPhil/aleph-splat-0

I dub this attention, Splat Attention. Rorschach splatted differentiated experts, each codebook stacked adjacently, formatted with a methodology of sliding window attention for long context formats. They exist to each have their own view of the same take, and to weigh their views into the outcomes. This allows thousands of codebooks to represent opinions, rather than just one or a few.

Will it work? Yes. Will it work as well as MHA? Probably not. Not yet anyway.

It will at least allow KV caching to exist, while simultaneously enable greater than 8192 depth through a format of utilizable aleph routed attention, specifically curated to align differentiated viewpoints from different codebooks, all represented along the same axis of utilizable space.

The attention mechanism will potentially allow differentiated spaces of 2048 aleph attention heads stacked in unilateral simultaneous execution, which operate exceedingly effectively in preliminary tests - defeating token recall on the tested local and mha heads by a large amount.

What I cannot promise or even say will work yet with certainty, is the depth. The preliminary formulas are converging cleanly and the recall is absolutely fantastic, but that does not mean they will converge yet. This may be the attention that replaces the MHA in these models, and hopefully tonight will have the answer.

Fable is interested in distilling bert's behavior at the core of an AlephLM experiment series to test the effectiveness of distillation into the AlephLM structure, and it sounds like a good idea coming off the direct experiment of it.

We will be testing arm coupling, decoupling, and testing the entire AMOE behavior as an internalized structural invariant built directly into the core of the AlephLM system. Essentially, WHEN this is successful, we will be controlling our MOE system from stem to stern.

So here we are. AlephLM is being controlled by fable on a runpod for the next few days. Updates arrive as they arrive. The big book of laws that Opus, Fable, and myself have cobbled will expand. We'll eliminate some, validate some, and learn new as we go like usual.

Every major system is heavily analyzed for structural and behavioral responses as we go like usual. We have a large set of geometry measuring mechanisms to test failures if they occur for why they fail as well unless we encounter an anomaly. So far the majority of anomalies have either been catalogued and turned into structure, eliminated from the potential of running into, sidestepped using deviations or forms of loss, or repaired architecturally and the laws updated to reflect the anomaly.

AlephLM will be an LLM before you know it. Stay tuned.

Massive success with both CaptionBert-v2 variants. They are both standalone and independent students with a similar residual effect.

https://huggingface.co/AbstractPhil/captionbert-8192-v2
https://huggingface.co/AbstractPhil/captionbert-8192-v2-b

This marks the end of the Bert teacher/student preprocessing for captionbert until a large discovery or innovation is discovered based on the AMOE research. In that case I may need the features in the future, but for now they are only necessary for analysis and potential ablation.

There are a few limitations.

CaptionBert-V2-B doesn't understand how to use V2-A's AMOE arms as effectively as A, and vise-versa. However, they ARE compliant with their twin's arms. All of which were trained independent of each other.

The core difference between the two was a logistics problem. Not all the ModernBert features landed. Some silently OOM'd and the audit slipped through the cracks. Once I approached it, I trained a model with the missing shards omitting the other teachers as well - so about 8m missing features or so.

B was issued the full corpus, which showed the full corpus didn't enhance the outcome for the base student. The structure was done with the data already and the cosine similarity mildly improved, pre-AMOE introduction.

The REAL strength came from attaching the AMOE systems to the B. They defeated A's benchmarks and pushed closer to MiniLM. A couple of out of scope too-small datasets were used, dropped a little between A and B, so there's legitimate differences for out of scope usage that have been unaddressed.

I'm not using MiniLM as a teacher, primarily because I want to have a legitimate industry-used marker as something to target.