Add inference metrics + bench script, tune threads/batch for 2-vCPU HF Space, backoff on readiness polling

Enforce max_tokens upstream, prune request registry, reap dead workers,

fix exceptions

fix error format wrapping now applies to /v1/chat/completions and generation stats

add token generation speed to ui

fix request parsing

Log detailed error bodies for UI failures

Fix 400 for llama.cpp web UI completion requests

Fix web UI chat by adding buffered SSE fallback

Fix Docker build for modular llm-manager

fix build bugs

Refactor the C++ LLM manager into modular components, moves Python modules under python/, and keeps the current control-plane behavior intact. The C++ server now has clearer separation for config, model lifecycle, runtime services, request parsing, HTTP helpers, and server routing, while Docker build/runtime paths were updated to compile multiple C++ files and load Python code from the new package folder

add auth, token policy, queue scheduler, and cancel flow, etc

add new endpoint to cancel all processing prompts

fix encoding

fix proxied response in llm manager

fix routing in llm manager

add cpp server