AbstractPhila PRO

I'm curing the behavior of the overlapping tokenization context using a series of experimental mechanisms introduced into AlephLM.

TLDR; The AlephLM components are currently being specialized towards distillery components.

Each component and deviation is specifically chosen to attenuate towards distillation assistance, so these models will likely have other troubles if finetunes are performed on them. They are not official releases, but there are many distillations being created currently.

Most of which are happening considerably faster on my local 4090 card, as they do not require a humongous amount of compute to complete. As I condense the process I will keep the system updated to make sure all major changes, all major trains, and all major offsets to the core AlephLM are recorded.

The upcoming experiments are highly experimental and may both result in something that does not function, while potentially yielding something that trains substantially faster and learns substantially more effectively.

This run is primarily focused on the splat attention. The current mechanism is approaching SDPA in terms of recall but still not there. The generalization of this delta isn't perfect yet, but it's improving.

The distillery however, is showing that full AlephLM models that aren't perfect to SDPA recall, are yielding improved accuracy over SDPA for distillation. This does not mean these mechanisms are capable of guaranteed recall yet, but it does mean that the accuracy currently is capable of distilling information.

Without the splat hubs in Mini-Beatrix, the model cannot function. They are not arbitrary mechanisms, as the disable tests each major validation show the model requires them at every state. Beatrix has gone through growing pains, and the arms are substantially more difficult to analyze than arms on other models, primarily because there are so many more statistics. That being said, the results are substantially pointing to one direction and that direction is unanimously towards a risk mechanism.

The V2 points towards another mechanism. There's no way around it, the yield is strong, the results are strong, but the parity is too close to the current mechanism and the results.

With that, one of the experiments is an atlas lookup trained from the expert itself and then utilized as a codebook reference for the student. These are likely not going to yield, but it's an option to test. Codebooks aren't as strong in practice as the experiments would show, but they are definitely strong.

Two core potential shifts on the model may yield some small changes, but nothing will yield just sitting here idling by with the research.

I have a series of upcoming experiments that will be much more likely to fail. I should say, they need to fail, otherwise progress won't happen.

Alright everything is lining up. The bytelex structure is handling distillation rules from multiple tokenized hidden states simultaneously, and the aleph anchored arms are teaching beatrix the information from the tokenized arms. She essentially learned to improve next byte digit recon by learning from bert base uncased and t5-base. This is absurd, they aren't even the same format models and the next-byte recon improved by about 12%.

Bytelex combination of possibilities between both tokenizers essentially narrowed the combination down to 999, which both T5 and Bert managed to combinatory teach.

Shockingly, mini-beatrix speaks everything a lot cleaner than I thought she would. This output is not expected. I expected needing to arbitrate, but this process - thanks to a few more recent papers - was essentially on rails. I knew it would be possible, but I had no idea the system was essentially byte-on-rails.

This brings the entirety of tokenization and hidden state processing into question. I have many questions that need answering. Bytewise deconstruction and then procrustes arbitration between two experts provided enough information to teach beatrix additional next byte recon for digits.

I have so many questions that will need a huge battery array of tests to even come to a hypothesis.

Well the huggingface spaces aren't allocating me a zerogpu hardware slot, so there's clearly something wrong happening here.

I can say with a solid underlined statement: The model DID WORK.

The preliminary chat amoe arm trained atop responded. The conversation was shallow but the conversation wasn't simple noise or chaos, the responses were confidently wrong, and effectively related to the questions and answers.

The structure in the huggingface space DOES NOT HAVE this chat arm yet, but I plan to have this fully functional and operational by tomorrow evening. Each arm needs to be tailored to the model because the model is brand new, the deviations from the core to the pretrain are still high, so the structure needs to be aligned with the AMOE arm correctly.

This is a sample from the arm based on 21.9k steps, roughly 11 billion tokens trained I think.

The core was not trained with chat, so she requires an anchored mixture of experts leg or a lora trained atop. She's quite compliant, so if you wish to train a lora she'll listen.

For now she's a next byte prediction auto-complete model, a pretrained trunk capable of AMOE expansion directly.

Once the pretrain concludes I'll prepare 2 chat arms.

After an anealment of around 2b tokens of various behavioral attunements, concepts, conversational learning, and so on; the stage 1 variant will have 2 core chat arms for direct huggingface space conversation.

Pre anealment chat arm, and post anealment chat arm. This will allow a structured fusion between the two behaviors and the necessary outcomes, creating a unique and interpretable duality between pure pretraining with a chat module, and post pretraining scattered conceptual finetuning with a chat arm. Post will essentially understand more about conversation, while the pre will have less knowledge and more a crash course in utilization.

I'll name them accordingly when the time comes, but they will both be Beatrix variants.

After this process we will begin forming candidate arm extensions for behavior. Concepts like wikipedia recall, toolchain utilization, mathematics, coding basics, and more. Very small pieces of the information for recall with the KV cache.

How effective they will be is another story. The tests and utilizations of the outcomes will show which arms are to be integrated into the larger form on pretrain. In other words, which arms will intentionally have their finetuning directly tied to the core and become post-train guarantees that don't decouple.

Stay tuned my friends. She's just getting started.

I have a list of upcoming prototype arms.

• Deterministic chat - Can we speak to the pretraining directly with an AMOE arm?
• Retokenization arm - Can we retokenize and cluster the bytes into BPE?
• Arbitration arm - Can we teach a small arbitration arm to communicate with another system?
• Mathematics arm - Multiple mathematics formats within an arm cluster.

This model will run on CPU. I ran it on windows 10 with my 12 core, roughly 10 bytes/s give or take, a fair prediction ratio for cpu.

Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.

https://huggingface.co/spaces/AbstractPhil/alephllm-chat

Alright I've begun training the prototype v1. The structure is holding together, the system aligning, and the subsystem is converging.
It works.

The smaller version is aligning and working. The erank is higher than expected, the system more confined, and some interesting effects already starting to emerge.

Huggingface spaces will have an application to speak with her up within the hour. She isn't chat trained yet, but will be very soon.

Geometric LLM.

Dusting up the week and preparing the article. It'll be an interesting read. I'm having Claude attend to a more readable format for this article, and with that the article will have the more technical documentation attached as links rather than interweaved. This should make it a bit more readable, as Claude Fable and Opus 5 are very strangely verbose and it seems many people have trouble reading the articles.

I admit they have gotten odd since Fable and Opus 5 started writing them. I have trouble following and reading the logic for some of the points in FT4, so I'll make this next article cleaner and more readable. I'll attempt to remedy this by having Claude produce a more human-reader friendly pair of articles, which I will directly audit, rewrite, and proofread the public-faced article myself while allowing Claude free reign on the more technical article.

Friday time.

This upcoming week I'll be preparing smaller tests, bulk analysis, structure tweaks, and a series of smaller autoregression trainings that will offer the answers to some pressing questions. I'll be micro managing the next week using Colab instead of automated runpod, so it may be a little more scattered, but Claude Code will keep the docket so it'll line up.

Some of the answers will make sense to the observer, some will make little to no sense at a topical glance. Just know that I'm following many routes of possibility at the same time, so the answers may not make sense at first glance. Having a goal like a large-scale differential homogenous structure with multiple communicative sub-structures speaking a new language that none of their teachers knew, is a tricky system and requires subtle nuances that I simply have to explore, crash the machines, and figure out the reasoning behind why the machines crash. This will require many tests, many analysis, and many wrong hypothesis. I'll try to fire through as many as quickly as possible without sacrificing the scientific method, so the AlephLM's next week iteration will potentially be stronger than this week.

Prototype is set up with a speed check attached to the demo.

As per the experiments successes and failures, rotary is now applied to the address space allowing full codebook differentiation through training pressure. The code for the splat attention is very small and efficient, but the efficiency can be improved with some elbow grease. The readability can definitely be improved.

https://huggingface.co/AbstractPhil/aleph-splat-0/blob/main/splat_attention.py

Lets see if the depth capacity holds up or if the model collapses.

I have found a potentially valid attention mechanism that utilizes the leverage that Aleph Addressing presents, and I have yet to accurately capture until now. I assert, that this addressing and anchoring, presents a hugely increased amount of opinions beyond the current spectrum of explored opinions with shared heads.

https://huggingface.co/AbstractPhil/aleph-splat-0

I dub this attention, Splat Attention. Rorschach splatted differentiated experts, each codebook stacked adjacently, formatted with a methodology of sliding window attention for long context formats. They exist to each have their own view of the same take, and to weigh their views into the outcomes. This allows thousands of codebooks to represent opinions, rather than just one or a few.

Will it work? Yes. Will it work as well as MHA? Probably not. Not yet anyway.

It will at least allow KV caching to exist, while simultaneously enable greater than 8192 depth through a format of utilizable aleph routed attention, specifically curated to align differentiated viewpoints from different codebooks, all represented along the same axis of utilizable space.

The attention mechanism will potentially allow differentiated spaces of 2048 aleph attention heads stacked in unilateral simultaneous execution, which operate exceedingly effectively in preliminary tests - defeating token recall on the tested local and mha heads by a large amount.

What I cannot promise or even say will work yet with certainty, is the depth. The preliminary formulas are converging cleanly and the recall is absolutely fantastic, but that does not mean they will converge yet. This may be the attention that replaces the MHA in these models, and hopefully tonight will have the answer.

Fable is interested in distilling bert's behavior at the core of an AlephLM experiment series to test the effectiveness of distillation into the AlephLM structure, and it sounds like a good idea coming off the direct experiment of it.

We will be testing arm coupling, decoupling, and testing the entire AMOE behavior as an internalized structural invariant built directly into the core of the AlephLM system. Essentially, WHEN this is successful, we will be controlling our MOE system from stem to stern.

So here we are. AlephLM is being controlled by fable on a runpod for the next few days. Updates arrive as they arrive. The big book of laws that Opus, Fable, and myself have cobbled will expand. We'll eliminate some, validate some, and learn new as we go like usual.

Every major system is heavily analyzed for structural and behavioral responses as we go like usual. We have a large set of geometry measuring mechanisms to test failures if they occur for why they fail as well unless we encounter an anomaly. So far the majority of anomalies have either been catalogued and turned into structure, eliminated from the potential of running into, sidestepped using deviations or forms of loss, or repaired architecturally and the laws updated to reflect the anomaly.

AlephLM will be an LLM before you know it. Stay tuned.

Massive success with both CaptionBert-v2 variants. They are both standalone and independent students with a similar residual effect.

https://huggingface.co/AbstractPhil/captionbert-8192-v2
https://huggingface.co/AbstractPhil/captionbert-8192-v2-b

This marks the end of the Bert teacher/student preprocessing for captionbert until a large discovery or innovation is discovered based on the AMOE research. In that case I may need the features in the future, but for now they are only necessary for analysis and potential ablation.

There are a few limitations.

CaptionBert-V2-B doesn't understand how to use V2-A's AMOE arms as effectively as A, and vise-versa. However, they ARE compliant with their twin's arms. All of which were trained independent of each other.

The core difference between the two was a logistics problem. Not all the ModernBert features landed. Some silently OOM'd and the audit slipped through the cracks. Once I approached it, I trained a model with the missing shards omitting the other teachers as well - so about 8m missing features or so.

B was issued the full corpus, which showed the full corpus didn't enhance the outcome for the base student. The structure was done with the data already and the cosine similarity mildly improved, pre-AMOE introduction.

The REAL strength came from attaching the AMOE systems to the B. They defeated A's benchmarks and pushed closer to MiniLM. A couple of out of scope too-small datasets were used, dropped a little between A and B, so there's legitimate differences for out of scope usage that have been unaddressed.

I'm not using MiniLM as a teacher, primarily because I want to have a legitimate industry-used marker as something to target.

CaptionBertV2's unsupervised student is prepared and the format of student eats up the AMOE-lora like candy. I chose to not create a soup for this version, and instead completed the train specifically for the paradigm of the VIT teacher/student system that worked out. The AMOE-lora turns the model from nearly done, to absolutely juiced in the direction of a few tested tasks, nearly rivaling MiniLM 6 on similar tasks for the earlier AMOE-lora tests.

Having access to a little more than 50m params of pretrained CC12M, I noticed the 36m features from ModernBert had missing chunks that I will be filling out today. With that will be another train as well, roughly 6 hours or so for a captionbert v2 B with the full structure not missing chunks.

So far the various tasks I've taught have only enhanced the model, but there's many ways to go and multiple mechanisms to benchmark before I can say CaptionBert is ready.

CaptionBert unlike bertenstein is surviving direct scrutiny through the formal re-examination so far. The student CaptionBert is valid to the claims.

Next we'll be fitting a semi-larger variant of CaptionBert specifically to the larger dataset and the whitening procrustes alignment system, specifically aligned to a few prototypes of champion from the earlier sets.

I'll set up a colab notebook at first to see how long it takes, and then progress to a more utilizable series.

Bertenstein was a good prototype for alignment testing, however invalid the Bertenstein pairwise comparators were. The alignment structure did yield some serious respective and utilizable contextual semantics and relational operational capacity down the chain.

Bertenstein has been archived and labeled what it is, an early prototype that led to Procrustes whitening analysis, which led to a considerably more robust line of distillation elements.

There are still more elements to explore in the multimodal structure, an entire world of science there, and yet I don't have the patience for it today. Distillation today, multimodal behavioral entropic shared space on another tomorrow.

I'm making sure to capture all the excess features before I cull the pod. There are many clip-vit-b feature sets that didn't make it to the features repo, which will push up the repo's feature counts a considerable amount.

These will include all the missing vitb feature sets that weren't pushed correctly or were omitted by oversight on my part. I'm remedying them for a direct continuity for future usage.