Hugging Face's logo
AbstractPhila's picture
🤝 Open to Collab

AbstractPhila PRO

AbstractPhil
AbstractPowered
19 6 27
Lancelotto's profile picture JoeMama123123123's profile picture FanchL's profile picture
·
https://civitai.com/user/AbstractPhila

AI & ML interests

datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.

Recent Activity

updated a model about 1 hour ago
AbstractPhil/alephlm-0
repliedto their post about 8 hours ago
I believe I have a solution for cross-tokenizer chatter and noise, which I've built a prototype repo for this exact tooling dubbed bytelex. https://github.com/AbstractEyes/geolip-bytelex I had a bit of an inspiration recently and built a prototype for a token translation matrix that I called geolip-bytelex, which allows bytewise translation of many different tokenizers into byte format. The goal is to allow comparative distillation from multiple models to simultaneously represent expertise based on input tokens and differentiated teacher/student InfoNCE and MSE training paradigms, while cutting a huge cost of the distillation analysis comparative compute that cross-tokenizer noise will naturally cause when tokenizers are mismatched or incorrect, reducing a large portion of invalidity from the trained systems established by incorrect valuations from the distillations and lora trainings. Bytelex is essentially a byte-wise deconstruction of a tokenizer's state into a preliminary 255 byte language allowing for 10s of thousands of sequences per token to be represented rather than just a few. I'm not the first to try this, however I'm in a unique position due to my creation AlephLM being built entirely by learning it's own lexicon, thus allowing this to be more than experiment and instead a working prototype distillation potential. This can solve a longstanding multi-tokenizer problem that I and many other researchers have been facing, at the cost of setup overhead compute for the preliminary experiments, however the translation matrix I'm planning will potentially solve this problem allowing models to be directly bytewise captured in a more guaranteed methodology through cross-sampled analysis at distillation time in this optimizer state that I'm working out. I've dubbed this distillation loss ByteInfoNCE and the preliminary is showing humongous promise, with that the bytelex is the crux and prototype concept that I'll be expanding and researching further.
updated a model about 18 hours ago
AbstractPhil/clip-vitb-mini-distilled
View all activity

Organizations

DeepGHS's profile picture Blog-explorers's profile picture BangumiBase's profile picture Abstract Powered Research's profile picture