Github

Task construction

We build repair tasks from interaction problems reported in open-source projects.

Construction process

Browse tasks
  1. 01

    Investigate the problem

    Investigate the issue and pull request to identify the conditions and interactions that trigger the problem. Document the findings and write replay scripts, then run those scripts to produce a video of the problem.

    Problem description and reproduction video.
  2. 02

    Determine the expected behavior

    Experts review the user’s intention alongside the problem description and video, then determine how the application should behave to help the user achieve that goal and why.

    Expected behavior and why it supports the user’s goal.
  3. 03

    Run trials and review

    Run trial evaluations, then review the results and task materials against the quality criteria below. Revise any materials that fall short and repeat the affected checks.

    Tasks that meet the quality criteria.

Quality criteria

Tasks should preserve the original problem and user intent.
The description and recordings must show the reported problem under legitimate application conditions. The user’s goal should provide enough context to investigate without revealing the solution. Reproduction must not create the defect.
Expected behavior should be justified by the user’s goal and product context.
Requirements must describe observable behavior supported by the product and its maintenance records. They must allow other valid solutions and have a demonstrated repair within the task’s software and dependency constraints. The repair’s code and supporting explanations cannot add requirements.
Scores should reflect whether the required behavior was achieved.
Checks must recreate the specified situation and gather fresh evidence for each requirement. Judgments follow the stated requirements and allowed alternatives. Environment, execution, and evidence failures are recorded separately from failures of the application.

Selection funnel

Counts show distinct PRs reaching each stage.

Source screening
Reject reports outside the scope of user-facing interaction problems.
4,356
PRs
Problem and definition review
Reject unreproducible problems, unclear goals, or unsupported expectations.
112
PRs
Maintainer fix validation
Reject when the maintainer’s fix does not resolve the defined problem.
73
PRs
Trial review
Reject unreliable checks or problems readily identified from code alone.
65
PRs
Released tasks
Included in the benchmark.
44
Tasks