Contribute

Install the CLI, then run these commands from the repository root.

Preview

Browse task intentions, rubrics, and reference recordings locally:

Shell

uv run ibench preview --task '<task-id>'

Omit --task to browse all tasks. Preview does not start task environments.

Debug

Start the original application with its reproduction fixtures:

Shell

uv run ibench debug '<task-id>' \
  --revision base --fixture before --output jobs/map-debug

Keep the process running while you inspect the environment or edit the replay scripts. The CLI prints the Docker project and script paths.

Press Ctrl-C to save edited scripts to jobs/map-debug/edited-scripts and remove the session's containers.

Replay

Record the original behavior and reference repair:

Shell

uv run ibench replay '<task-id>' \
  --revision base --fixture before --output jobs/map-before

uv run ibench replay '<task-id>' \
  --revision merged --fixture reference-after --output jobs/map-reference

Add --scripts jobs/map-debug/edited-scripts to replay your edited scripts. Use a new output directory for each debug session or replay. Recordings, logs, and results.json are saved there.

Check

Validate task files before submitting changes:

uv run ibench check .

This checks the release files and task contracts. Use debug and replay to check how the application runs.

Feedback

Report problems with the benchmark, website, or documentation in GitHub Issues.