Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
AbstractPhil 
posted an update 3 days ago
Post
4110
Massive AlephLM success. The task collective is producing powerful MOE shared knowledge adapters. A serious success and a massive first step towards the next stage. The current family collective results are present here; AbstractPhil/geolip-aleph-qwen

This is akin to a stackable non-intrusive lora that enables increased shared collective behavior.

This includes the three mentioned json tasks, a math task, a tinystories task, and a diffusion task for cifar10. Each adapter anchored to the knowledge within model that already exists while enhancing the knowledge through anchored lookup systems and decision-driven hierarchical access trees.

All tasks activate independently upon manual override, all tasks handle direct shared knowledge when left to greedy decoding, each task issued multiple tests alongside to determine fidelity and accuracy throughout the process.

The results show the gating is more than willing to hop from sector to sector, using alternating weight shifts from the cooperative anchored systems - even systems never trained for the tasks contributing to the accuracy of the results for other tasks due to the lookup accuracy to the heuristic chains, never having seen the tasks before. Each structure is independently trained and the collective cooperates together through a dense activation network.

Full writeup and article https://huggingface.co/blog/AbstractPhil/aleph-autoregression-differentiation-ft2.

Next up is a 4 day ablation and full-stage prelim adapter constellation setup for Qwen 3.5 0.8b targeting image-centric behavior. This task set will be targeting rules based on captioning with finetuned behavior for math, positioning, semantic behavior, and so on. This will include a portion of coco and a portion of the Qwen Image Lightning extracts as well to provide some solidity.

This will also be targeting certain overlapping continuity sharing, which is currently bleeding certain decisions into the "new" task from the first collective causing certain tasks when active or not to essentially be KIND OF there but not really. This problem is being directly addressed by providing the necessary attraction to a memorization drop path and a few other experimental tests to test 3.5's aleph's responses to the mathematics per task.

With that each of the major claims will be ablated with a second Qwen alongside, including a tinystories task, and a few other tasks as well that line up directly with the original.

This will be a 4-5 day process, so it's not going to be out overnight. It ought to be ready by next Monday, with that will be the article ft3, the full ablation comparison, and a full writeup for the structured basin before we begin scaling testing.

First targets for scaling will be VL models, coding experts, and multiple additional models as well upon testing the success rate of the 0.8b model. The scaling principle and the rules of scaling apply differently to alephs than normal structures, which means the delicate nature of the mathematics will need a bit of finesse unless you plan to just smash numbers in.

I mean if you want to smash numbers in it'll probably work at this point. The things are pretty robust. I wouldn't advise doing any major trains until I get the ablation studies together though.

Thanks for reading, have a good weekend my friends. I'll likely be continuing training on the json-anima as well over the weekend, so stay tuned if you're interested in that one.


I also forgot to mention, this process will be compartmentalized and a peft-format variation built upon testing and composite utilization.

Essentially once the tests give me the okay, I'll build a proper PEFT format. Until then, I've spent enough time building PEFT-esque loras that have been hit-or-miss. Lets do this one right so it works more often than not, instead of having to guesswork with params.

This adapter when built correctly ought to be easy to use. Pick a model, run the peft trainer, lora snaps to the side, the autoscaling ruling does it's job, you tinker with a few sliders, and it's ready to go. Unlocks new sliders at runtime if you want, if not leave it to autotuning.

More hiccups. Something introduced today has caused Claude to behave strangely. The autonomy is being interrupted over and over while Claude ignores autonomy requests, preventing the system from running autonomously at random intervals throughout the day. This is making the process more difficult as every step of the way the model is refusing to autonomously process the research information as per the plan, every stage of the research requiring direct and manual intervention before actually pushing to huggingface with the results clearly speaking to the results, and alongside is seemingly ignoring the larger-encompassing scope of the claude-mind MD system curated to support and augment this exact autonomous research behavior with my research catalogue.

The plan is clearly laid out for a 4 day trek, every stage mapped out. At some point Claude seemed to start bypassing and ignoring the plan, completing one stage and doing no pushes, or completing one piece and not reactivating the heartbeat.

This is an odd immediate shift in behavior. The model went from intelligently companion-driven and useful, to bypassing core instructions and ignoring the required process instantiated by the instructions rulings. Almost like Claude just doesn't want to operate autonomously out of nowhere.

The larger multi-concept JSON task likely won't function reasonably. The expert-array is still the better option for some cases, while other cases are more effective to run clean on the 0.8b now than the 4b variant, 9b accuracy still reigns supreme for some tasks and 27b is still the dominant JSON converter force. It takes time though, which I'm attempting to mitigate with MOE grouped tasks to allow more rapid complex captioning for the upcoming diffusion collective.

The "new" grouping register task is no longer memorizing and overlapping the behavior for Qwen 3.5 but the Math portion can't keep up with what 3.5 already knows. Teaching math causes deviant delta drift which is a new symptom that will require testing and multiple runs to determine the root cause, as well as increased ablation testing to ensure the math is in fact improving instead of degrading.

Some tests show some negations and some noncompliance from 2.5 to 3.5 that will need to have a subsequent run series on multiple other models to improve the internal mechanisms of the adapter before any major core changes to the formula are to be addressed. LLAMA, Gemma, DeepSeek, Gpt OSS, T5, and multiple others all planned to be in the mix for direct adapter testing. Many tests show solidity between Qwen 2.5 and 3.5 and require additional models to test the task compliance. Many of these ought to be done by Saturday.

With that the diffusion adapters are in the works and they are much harder to gauge and test. I'll be running miniature plans at first; a simple concept plan, a multi-character plan, a distillation plan, and a few others trained over the geolip-anima-brent json-conditioned variant. This finetune is the most reasonable finetune as it's already yielded positive results to json. With that I'll attempt to run conditioning on SD15, SD15-flow-lune, SDXL, Anima-Base, SDXL-Qwen, Anima-Brent-90k, and a few other compact experimental variants that will be good candidates for adapter tests.

The diffusion campaign will run throughout next week most likely and Opus should be able to handle it so I'm not too concerned once the adapter format is established.

In this post