What this measures
People use software with different levels of experience and ability, under different conditions. Some usability problems only become apparent during particular tasks, in certain application states, or through specific interactions. To evaluate coding agents’ ability to discover and repair usability problems in real-world scenarios, we review maintenance records, confirm and document each problem, and define verifiable criteria for a repair. We then give an agent the user’s original intention and ask it to find and fix the problem.
To turn usability analysis into repeatable evaluation, we describe each problem in structured natural language following a behavior-driven development (BDD) style. The description connects the user’s intention to the conditions and interactions that expose the problem, then states the observable software behavior a repair should satisfy. A rationale explains why these requirements address the user’s difficulty, while allowed alternatives leave room for different valid repairs.
- Intention
- What the user wants to accomplish.
- Given · When · Observed
- The starting conditions, the interaction, and the behavior that exposes the problem.
- Rationale
- Why the behavior causes difficulty and why the required change fits the product.
- Require · Allowed
- The behavior a repair must satisfy and the acceptable alternatives.
With the patch applied to a clean copy of the application, a grading agent uses the supplied setup scripts to recreate the conditions described in Given.
The agent uses the supplied replay scripts to perform the interaction described in When. It can adapt controls and action paths to the repaired interface while preserving the scenario and interaction being checked.
The agent uses the new recording and execution records to judge whether the observed behavior meets Require, accounting for Allowed alternatives. A task passes only when every case passes. Timeouts and incomplete checks receive no credit.