Benchmarked on what users feel.
A feature can work correctly and still be hard for people to find, understand, or use. These difficulties can emerge as an app reaches more users and changes over time. Interaction Bench tests whether coding agents can investigate these difficulties from a user’s perspective and fix the app.
- tasks
- open-source apps
- combined GitHub stars
- best pass rate
Pass rate
↑ higher is betterCost
↓ lower is betterCache hit
↑ higher is betterSpeed
↓ lower is betterCompare task pass rates with mean cost or median runtime. Each point represents a model and harness; the line shows the Pareto frontier.
Start with a real task
Each task comes from a real maintenance case. The agent receives the user’s intent, the source code, and an environment without network access.
Investigate and repair
The agent interacts with the app and inspects its code to understand where users struggle, decide what should change, and implement a repair.
Judge the repair
A grading agent interacts with the patched app to judge whether the problem is fixed, using expert-defined behavioral requirements.
Each task pairs a reproducible software problem with reviewed repair requirements and a working example of a fix.
- 01
Investigate the problem
Investigate issue and pull request records, then document and record the problem. - 02
Determine the expected behavior
Use the user’s intention and problem evidence to explain how the application should behave and why. - 03
Run trials and review
Review trial results against the quality criteria, revise the task materials, and check them again.
Each task runs in a fixed version of an open-source application, prepared with the data needed to reproduce the problem.
