bloomery

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

GitHub

Official source

Start locally with the CLI guide