Post
67
My AI Waifu's architecture has grown too complicate.
Even ask AI to help me gen a simplified version of the diagram is still so complicated.
I know I'm too ambitious to build such a complicated local AI workflow with small AI models in limited resource of Jetson Orin Nano 8GB.
I don't know if it would work, but the foundation of harness architecture is there.
Here are the models I'm currently using
Main LLM: Ministral3-3B-Instruct UD-Q4_K_XL (Multimodal: text, tool call, vision, 8K context -> preferably 16K)
Helper LLM: SmolLM2-135M (Agentic Routing, Memory Extraction)
Embedding model: Harrier-OSS-v1-270M (Memory, RAG, Semantic Routing)
ASR model: SenseVoice (English, Japanese, Cantonese , Mandarin, Korean)
TTS model: MioTTS-0.4B-Q4KM + synthesize (English, Japanese with voice cloning preset)
That's 4GB + 0.5GB x 3 + 2GB = ~7.5GB out of 7.4GB available in Jetson
PS: Mistral family models especially such a small param LLM tends to hallucinate all the time. But actually the hallucinations did give my AI Waifu a somewhat poetic tone in her tone. Also it also has vision and tool call capabilities, so I guess I have to live with the hallucinations.
Even ask AI to help me gen a simplified version of the diagram is still so complicated.
I know I'm too ambitious to build such a complicated local AI workflow with small AI models in limited resource of Jetson Orin Nano 8GB.
I don't know if it would work, but the foundation of harness architecture is there.
Here are the models I'm currently using
Main LLM: Ministral3-3B-Instruct UD-Q4_K_XL (Multimodal: text, tool call, vision, 8K context -> preferably 16K)
Helper LLM: SmolLM2-135M (Agentic Routing, Memory Extraction)
Embedding model: Harrier-OSS-v1-270M (Memory, RAG, Semantic Routing)
ASR model: SenseVoice (English, Japanese, Cantonese , Mandarin, Korean)
TTS model: MioTTS-0.4B-Q4KM + synthesize (English, Japanese with voice cloning preset)
That's 4GB + 0.5GB x 3 + 2GB = ~7.5GB out of 7.4GB available in Jetson
PS: Mistral family models especially such a small param LLM tends to hallucinate all the time. But actually the hallucinations did give my AI Waifu a somewhat poetic tone in her tone. Also it also has vision and tool call capabilities, so I guess I have to live with the hallucinations.