Run the benchmark

Install

Use Linux with Git, uv, Docker Engine, and Docker Compose v2. Docker must be running and accessible to your user.

git clone https://github.com/eval-sys/ibench.git
cd ibench
uv sync --locked
cp .env.example .env
bash scripts/build-dev-images.sh
uv run ibench check .

The build script creates the local base image required by the task containers. Run these commands from the repository root.

Credentials

Set DEEPSEEK_API_KEY in .env. The CLI loads this file automatically; existing environment variables take precedence.

To use another credential file, add --env-file /path/to/credentials.env before the run or resume subcommand.

Run

Start with one task and one concurrent trial, using DeepSeek Flash for both solving and scoring:

uv run ibench run --output jobs/deepseek-smoke \
  --task photo-map-panning -n 1 \
  --solver-config configs/deepseek-flash.json \
  --score-config configs/deepseek-flash.json

--task selects a task by name or ID, and -n controls concurrency.

Repeat --task to select several tasks, or omit it to run all 44. Task names and IDs are listed in tasks/release.json.

Choose other solver and scorer configs from configs/ as needed. For example, configs/gpt-6-astra-high.json uses OPENAI_API_KEY for the solver.

The run initializes the task environment, runs the solver, and then scores its patch. Results are saved to jobs/deepseek-smoke/results.json. Use a new output directory for each run.

Resume

To continue an interrupted run, use its output directory:

uv run ibench resume jobs/deepseek-smoke