octobench

Benchmark harness that compares coding agents — not just models — on real, contamination-resistant tasks: 80 fail-to-pass cases from merged OSS PRs, long-run sequences, SWE-bench-Live, and 25+ domain benchmarks. We benchmark our own agent, octomind, under the same rules.

GitHub

Official source

Start locally with the CLI guide