Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

We treat agents like senior engineers, so why evaluate them like junior engineers?

Senior SWE-Bench feature tasks have realistic instructions that read like natural language messages rather than over-specified requirements. To reliably evaluate these tasks, we introduce a validation agent which uses expert-designed recipes to write behavioral tests that adapt to submitted solutions.

Senior SWE-Bench bug tasks reflect tricky user reports and focus on investigation, from starting services to debugging subtle runtime issues. They are sourced from PRs that needed significant runtime investigation to solve (e.g. logs, profiling data, reproduction steps).