CodeSoft/slm-arena-data
1.65 MB
CodeSoft PRO
CodeSoft
AI & ML interests
Working on small language models
Recent Activity
new activity about 4 hours ago
CodeSoft/SLM-Arena:Open the user inputs updated a bucket about 17 hours ago
CodeSoft/slm-arena-data liked a Space about 21 hours ago
VIDraft/global-llm-leaderboardOrganizations
Posts 6
Post
2232
Sorbet Mini Experimental is out!
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
Post
115
Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
models 7
CodeSoft/sorbet-mini-experimental
Text Generation • 4.92M • Updated • 710 • 1
CodeSoft/sorbet-v2-25m
Text Generation • 25.2M • Updated • 1.32k
CodeSoft/sorbet-25m-gguf
Text Generation • 25.2M • Updated • 103
CodeSoft/sorbet-25m
Text Generation • 25.2M • Updated • 1.24k • 4
CodeSoft/MetaDiffusion-150M-ChatBase
Text Generation • 0.2B • Updated • 988 • 2
CodeSoft/MetaDiffusion-600M-ChatBase
Text Generation • 0.8B • Updated • 594 • 1
CodeSoft/MetaDiffusion-150M-exp
Text Generation • 0.2B • Updated • 432 • 2
datasets 0
None public yet