What is a usability problem?
Usability problems arise when an application makes it difficult for people to accomplish their goals. They may become apparent only during a particular activity or under specific conditions. Understanding them requires examining what the user is trying to do, how the application responds, and why that response causes difficulty. Interaction Bench (iBench) evaluates whether coding agents can investigate these problems and repair them in existing applications.
People interpret an interface to decide how to pursue a goal. After acting, they interpret the feedback to understand what happened and choose their next step. This cycle depends on their experience and abilities, as well as the conditions in which they use the software. An action or response that seems clear to someone familiar with the product may be difficult for another person to find, perform, or understand.
Human–computer interaction (HCI) and cognitive psychology help explain how people’s attention, memory, and prior experience shape their use of software. We draw on these perspectives to understand where an interaction causes difficulty and what changes could help, considering the user’s goal and the product’s context.
A good designer knows to ask whether an interface explains what happened, uses concepts familiar to its users, and helps them avoid or recover from errors. They investigate why people struggle and determine what should change, considering the user’s goal and the product’s constraints. This is the kind of judgment iBench asks coding agents to exercise.
In scope: usability
Out of scope
iBench focuses on usability problems in existing applications during maintenance. As a product evolves, its specifications may become outdated or leave interactions between features unexplained. People may also use it in ways, or under conditions, that its maintainers did not anticipate. Understanding what should change therefore requires examining the current product alongside the user’s goal.
Specify
Intent is fresh. Most decisions about who the product is for and how it should behave are made here, together.
Build
The product is built against that spec. What it should do and what it does are still close.
Release
Real users arrive, on devices, in contexts and with habits nobody designed for.
Maintain
New features land on old ones, priorities shift, and the spec drifts out of date. This is where Interaction Bench sits.
Change accumulates
Use widens
iBench tasks begin with a user’s goal in an existing application. The agent must investigate the difficulty and determine what should change, using the running product and its repository.
These cases illustrate usability problems that people may readily notice during use, but frontier agents can still miss.
Explore the landscape photo's location on the map.
Draw a closed path on the canvas.
Evaluation is outcome-oriented: we transform user activities into replayable UI interactions and expected outcomes into observable application behavior. We replay these interactions in the modified application and check whether the resulting behavior meets the requirements.
- GivenAn administrator is viewing Campaign Performance in the Invite Access Demo collection. Sales can view the collection, while Marketing and All Users cannot.
- WhenThe administrator opens the dashboard invitation form, selects Marketing, adds and then removes Sales, and sends the invitation with only non-access groups selected.
- RequireThe invitation form shows all eligible groups and distinguishes those granting dashboard access. Both access and non-access groups remain selectable. Selecting only non-access groups shows an inline warning, and submitting the invitation succeeds.
- RationaleAn administrator is inviting a colleague to Campaign Performance and choosing their team memberships. Sales can view the dashboard’s collection, while Marketing cannot, so changing the selected groups also changes the colleague’s access. The form should explain this consequence and warn when access is missing, while still allowing the intended memberships.