tokio-prompt-orchestrator

Tokio pipeline for serving LLM requests: bounded five-stage flow with dedup, circuit breakers, retries, rate limits and a dead-letter queue in front of Anthropic, OpenAI, llama.cpp or vLLM. REST, SSE, MCP, TUI.

GitHub

Official source

Start locally with the CLI guide