Quantized Models (GGUF, IQ, Imatrix)
Collection
Various GGUF quantizations of small models. Models with a "checkmark" are personal favorites. An "orange arrow" means it's being uploaded. • 97 items • Updated • 71
Discussions
- General discussion and feedback.
Feedback is always welcome for potential issues with quants and as a way to help the author improve on the next iteration, your comments are appreciated!
SillyTavern
The complete AIO recommended preset:
[SillyTavern Presets]
My GGUF-ARM-Imatrix quants of Captain-Eris-Diogenes_Twilight-V0.420-12B.
⛶ [Expand/hide] Example setup.
Example setup in SillyTavern...
- Downloads last month
- 1,168
Hardware compatibility
Log In to add your hardware
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support