SoAI

An OpenAI and Anthropic-compatible API in front of llama.cpp, Ollama, vLLM and remote providers, with on-demand backends, queuing, failover and load balancing. Also includes a multi-user WebUI with chat, agents, automations, a RAG knowledge base, GPU monitoring and control.

GitHub

Official source

Start locally with the CLI guide