suwalski-agent-eval

Evaluation framework for AI agents: scenarios as YAML in the agent's repo (including simulated users), checks on tool calls, events, citations and text, LLM-as-a-judge, sweeps comparing models and prompts, trends, and regression tracking that tells what got worse after a change. Web app, CLI and Claude Code plugin.

GitHub

Official source

Start locally with the CLI guide