flyweight

A native C++/CUDA GGUF inference runtime with an OpenAI- and Anthropic-compatible server. Runs Qwen, DeepSeek-V4, Gemma 4, BailingMoE3 and more, with CUDA-graph decode, MoE expert offload, multi-token prediction, quantized KV caches, vision input and a bundled chat UI.

GitHub

Official source

Start locally with the CLI guide