-i fix-git --n-concurrent 1 --jobs-dir /tmp/gym-jobs

```

`bench/post_gym_run.py` posts a completed job's graded result to

`POST /api/v3/gym/runs`. Post only runs whose verifier actually ran: a

score is a claim, and a crashed grader is not a grade.

`bench/post_gym_run.py` posts a completed job's graded result to the Gym.

Without a run id it posts one summary row to `POST /api/v3/gym/runs`. With

`--run-id` (or `OPENAGENTS_GYM_RUN_ID` in the environment) it finalizes a

run that `run-suite.sh` registered against the lifecycle API: it upserts

each trial's final state and patches the run to `graded`, or to `abandoned`

when no verifier ran. Post only runs whose verifier actually ran: a score

is a claim, and a crashed grader is not a grade.

If the dev server binds loopback only (another session started it

without `PHX_LISTEN_ALL`), bridge instead of fighting over the port: run

## Suite runner

`bench/run-suite.sh` runs a suite file through Harbor and posts the result.

`bench/run-suite.sh` runs a suite file through Harbor and reports the run

to the Gym.

```sh

bench/run-suite.sh <suite-file> --model <harbor-model> [options]

Options include `--lane`, `--api-url`, `--jobs-dir`, `--n-concurrent`,

`--timeout-multiplier`, and `--dry-run`. Set `OPENAGENTS_TOKEN` unless the

model starts with `ollama/`.

### Run lifecycle

When you set `OPENAGENTS_TOKEN`, the runner registers the run with

`POST /api/v1/gym/runs/start` before Harbor starts, so `/gym` shows the run

while its trials are still executing. The runner exports

`OPENAGENTS_GYM_RUN_ID` and `OPENAGENTS_GYM_API_URL` — the host-side API

URL, not the container-rewritten one — into the Harbor run, and the adapter

reports each trial to `POST /api/v1/gym/runs/{id}/trials` from the host:

state `running` when the agent phase starts, then again with the thread id

it parses from the coder's `[oa:thread <uuid>]` line in the trial's

`coder.txt`. The coder prints that line in `--plain` mode on both lanes

when it has a server thread; a trial whose coder ran without one is

registered and graded without linkage. A failed trial report never fails

the trial.

After grading, the runner finalizes through `post_gym_run.py --run-id`,

which upserts each trial's final state — `passed`, `failed`, or `ungraded`

when the verifier never ran — and patches the run to `graded`. When no

verifier ran at all, it patches the run to `abandoned` instead: a crashed

grader is not a grade, but it must not stay a forever-running row. When the

suite fails before grading, `run-suite.sh` patches the run to `abandoned`.

Without a token, for example an `ollama/...` dry run, the runner skips

registration and the trials run without live reporting. If registration

fails, the suite still runs and the post-hoc one-shot path posts the graded

result as before.

published npm version. The trajectory is the coder's own ATIF export: the

run pipes `/export` after the instruction and copies the newest export to

the trial's `trajectory.json`.

When `OPENAGENTS_GYM_RUN_ID` is set (`bench/run-suite.sh` registers the run

against the Gym lifecycle API, OpenAgentsInc/openagents#38), the adapter

reports each trial to the run from the host side: state `running` at

agent-phase start, and again after the agent phase with the thread id parsed

from the captured coder output (`coder.txt`, the `[oa:thread <uuid>]` line

the coder prints in `--plain` mode). Reporting never fails a trial: a

refused or unreachable Gym is logged and the trial continues.

"""

import json

import os

import re

import shlex

import sys

import urllib.request

from pathlib import Path

from typing import override

_DEFAULT_API_URL = "http://host.docker.internal:4000"

_EXPORT_DIR = "$HOME/.openagents/exports"

# The coder's --plain thread announcement (OpenAgentsInc/openagents#39). The

# format is a contract between the CLI and this adapter; do not loosen it.

_THREAD_LINE = re.compile(r"\[oa:thread ([0-9a-fA-F-]{36})\]")

class OpenAgentsCoder(BaseInstalledAgent):

    SUPPORTS_ATIF = True

    def _local_lane(self) -> bool:

        return bool(self.model_name) and self.model_name.startswith("ollama/")

    @property

    def _task_name(self) -> str:

        # `logs_dir` is the host-side `<job>/<trial>/agent` directory, and

        # Harbor names the trial directory `<task>__<shortuuid>`. The task

        # half is the upsert key the run's trials endpoint keeps per task.

        trial = Path(self.logs_dir).parent.name

        return trial.rsplit("__", 1)[0] or trial

    def _thread_id_from_log(self) -> str | None:

        """The thread id the coder announced, or None when it ran offline."""

        try:

            text = (Path(self.logs_dir) / "coder.txt").read_text(errors="replace")

        except OSError:

            return None

        match = _THREAD_LINE.search(text)

        return match.group(1) if match else None

    def _report_trial(self, state: str, thread_id: str | None = None) -> None:

        """Report this trial to the registered Gym run, from the host side.

        `run()` executes on the host, so the POST goes to the host-side

        `OPENAGENTS_GYM_API_URL` on `OPENAGENTS_TOKEN`. Reporting must never

        fail the trial: every failure is logged and swallowed.

        """

        run_id = os.environ.get("OPENAGENTS_GYM_RUN_ID", "")

        api_url = os.environ.get("OPENAGENTS_GYM_API_URL", "")

        token = os.environ.get("OPENAGENTS_TOKEN", "")

        if not run_id or not api_url or not token:

            return

        payload: dict[str, str] = {"task": self._task_name, "state": state}

        if thread_id:

            payload["thread_id"] = thread_id

        request = urllib.request.Request(

            f"{api_url}/api/v1/gym/runs/{run_id}/trials",

            data=json.dumps(payload).encode(),

            headers={

                "Content-Type": "application/json",

                "Authorization": f"Bearer {token}",

            },

            method="POST",

        )

        try:

            with urllib.request.urlopen(request, timeout=10):

                pass

        except Exception as error:  # noqa: BLE001 - reporting never fails the trial

            print(f"gym trial report failed for {self._task_name}: {error}", file=sys.stderr)

    @override

    async def install(self, environment: BaseEnvironment) -> None:

        if not _TARBALL.exists():

                "OPENAGENTS_CODER_OLLAMA_HOST", "http://host.docker.internal:11434"

            )

        await self.exec_as_agent(

            environment,

            command=command,

            env=env,

        )

        # The trial exists before the agent phase does anything, and after it

        # the coder's captured output names the thread the session opened —

        # on both lanes now that the local lane reports too (#39). The second

        # report runs in `finally` so a failed agent phase still links its

        # trial before the exception continues to Harbor.

        self._report_trial("running")

        try:

            await self.exec_as_agent(

                environment,

                command=command,

                env=env,

            )

        finally:

            self._report_trial("running", self._thread_id_from_log())

Usage:

    OPENAGENTS_TOKEN=... python3 bench/post_gym_run.py \

      <job-dir> [--api-url http://localhost:4000] [--lane proxy]

      <job-dir> [--api-url http://localhost:4000] [--lane proxy] \

      [--run-id <uuid>]

Reads the job's `result.json` and each trial's ATIF `trajectory.json` (for

token totals), derives the recipe digest from the job's `config.json` plus

the agent tarball digest, and POSTs to `/api/v3/gym/runs`. Idempotent: the

server replays a repeated digest rather than duplicating the row.

the agent tarball digest, and posts to the Gym.

With `--run-id` (or `OPENAGENTS_GYM_RUN_ID` in the environment) this is the

finalize step of a run `bench/run-suite.sh` registered against the lifecycle

API (OpenAgentsInc/openagents#38): each trial's final state is upserted to

`POST /api/v1/gym/runs/{id}/trials` — `passed` or `failed` where the

verifier ran, `ungraded` where it never did — and the run is closed with

`PATCH /api/v1/gym/runs/{id}`. A run whose verifier never ran on any trial

is patched `abandoned`: a crashed grader is not a grade, but it should not

be a forever-running row either. A 409 on the PATCH means another run

already holds the digest; that is a replay, reported and treated as success.

Without a run id, this keeps its original shape: one POST to

`/api/v3/gym/runs`, idempotent by recipe digest. Post only runs whose

verifier actually ran: a score is a claim, and a crashed grader is not a

grade.

"""

from __future__ import annotations

import argparse

import hashlib

import json

import os

import re

import sys

import urllib.error

import urllib.request

from pathlib import Path

# The coder's --plain thread announcement (OpenAgentsInc/openagents#39),

# the same contract the adapter parses live.

_THREAD_LINE = re.compile(r"\[oa:thread ([0-9a-fA-F-]{36})\]")

def sha256_file(path: Path) -> str:

    digest = hashlib.sha256()

    return digest.hexdigest()

def request_json(url: str, token: str, payload: dict, method: str) -> tuple[int, dict]:

    request = urllib.request.Request(

        url,

        data=json.dumps(payload).encode(),

        headers={

            "Content-Type": "application/json",

            "Authorization": f"Bearer {token}",

        },

        method=method,

    )

    with urllib.request.urlopen(request) as response:

        return response.status, json.loads(response.read() or b"{}")

def thread_id_of(trial_dir: Path) -> str | None:

    """The thread id the coder announced in this trial's captured output."""

    coder_log = trial_dir / "agent" / "coder.txt"

    try:

        text = coder_log.read_text(errors="replace")

    except OSError:

        return None

    match = _THREAD_LINE.search(text)

    return match.group(1) if match else None

def main() -> int:

    parser = argparse.ArgumentParser()

    parser.add_argument("job_dir", type=Path)

    parser.add_argument("--api-url", default="http://localhost:4000")

    parser.add_argument("--lane", default="proxy")

    parser.add_argument("--suite", default="terminal-bench@2.0")

    parser.add_argument(

        "--run-id",

        default=os.environ.get("OPENAGENTS_GYM_RUN_ID") or None,

        help="Finalize this registered Gym run instead of the one-shot POST. "

        "Defaults to OPENAGENTS_GYM_RUN_ID when set.",

    )

    arguments = parser.parse_args()

    token = os.environ.get("OPENAGENTS_TOKEN", "")

    config = json.loads((job_dir / "config.json").read_text())

    trials = []

    trial_entries = []

    input_tokens = 0

    output_tokens = 0

    duration_seconds = 0.0

            continue

        trial = json.loads(trial_result.read_text())

        trials.append(trial)

        trial_entries.append((trial, trial_dir))

        execution = trial.get("agent_execution") or {}

        started = execution.get("started_at")

        finished = execution.get("finished_at")

            agent_version = agent.get("version") or agent_version

            model = agent.get("model_name") or model

    def verifier_ran(trial: dict) -> bool:

        return trial.get("verifier_result") is not None

    def passed(trial: dict) -> bool:

        verifier = trial.get("verifier_result") or {}

        rewards = verifier.get("rewards") or trial.get("rewards") or {}

            reward = next(iter(rewards.values()))

        return bool(reward) and float(reward) > 0

    def task_name(trial: dict) -> str:

        return trial.get("task_name") or trial.get("trial_name") or "?"

    def task_key(trial_dir: Path) -> str:

        # The same key the adapter reports live: the task half of Harbor's

        # `<task>__<shortuuid>` trial directory name, so the finalize upserts

        # the rows the live reports created rather than writing new ones.

        return trial_dir.name.rsplit("__", 1)[0] or trial_dir.name

    tasks_total = len(trials)

    tasks_passed = sum(1 for trial in trials if passed(trial))

    ).encode()

    recipe_digest = "harbor:" + hashlib.sha256(recipe_source).hexdigest()

    report = {

        "job_id": result.get("id"),

        "trials": [

            {

                "task": task_name(trial),

                "passed": passed(trial),

                "exception": (trial.get("exception_info") or {}).get("exception_type")

                if trial.get("exception_info")

                else None,

            }

            for trial in trials

        ],

    }

    if arguments.run_id:

        run_url = f"{arguments.api_url}/api/v1/gym/runs/{arguments.run_id}"

        # Each trial's final state, upserted by task. A trial whose verifier

        # never ran stays a claimless `ungraded` rather than a grade.

        for trial, trial_dir in trial_entries:

            state = ("passed" if passed(trial) else "failed") if verifier_ran(trial) else "ungraded"

            trial_payload = {"task": task_key(trial_dir), "state": state}

            thread_id = thread_id_of(trial_dir)

            if thread_id:

                trial_payload["thread_id"] = thread_id

            try:

                request_json(f"{run_url}/trials", token, trial_payload, "POST")

            except (urllib.error.URLError, OSError) as error:

                print(f"trial upsert failed for {task_key(trial_dir)}: {error}", file=sys.stderr)

        if not any(verifier_ran(trial) for trial in trials):

            # A crashed grader is not a grade, and a run nobody will grade

            # should not stay a forever-running row.

            request_json(run_url, token, {"status": "abandoned"}, "PATCH")

            print(f"run {arguments.run_id} abandoned: no trial's verifier ran")

            return 0

        finalize = {

            "status": "graded",

            "tasks_total": tasks_total,

            "tasks_passed": tasks_passed,

            "input_tokens": input_tokens or None,

            "output_tokens": output_tokens or None,

            "duration_seconds": int(duration_seconds) or None,

            "recipe_digest": recipe_digest,

            "report": report,

            "model": model,

            "agent_version": agent_version,

        }

        try:

            status, body = request_json(run_url, token, finalize, "PATCH")

        except urllib.error.HTTPError as error:

            if error.code == 409:

                # Another run already holds this digest: a replay, not a

                # failure. The registered row is closed by the run that owns

                # the digest.

                print(f"409: run {arguments.run_id} replayed an existing digest")

                return 0

            raise

        run = body.get("run") or {}

        print(

            f"{status}: run {run.get('id') or arguments.run_id} "

            f"status={run.get('status')} passed={tasks_passed}/{tasks_total}"

        )

        return 0

    payload = {

        "suite": arguments.suite,

        "agent": "openagents-coder",

        "output_tokens": output_tokens or None,

        "duration_seconds": int(duration_seconds) or None,

        "recipe_digest": recipe_digest,

        "report": {

            "job_id": result.get("id"),

            "trials": [

                {

                    "task": (trial.get("task_name") or trial.get("trial_name") or "?"),

                    "passed": passed(trial),

                    "exception": (trial.get("exception_info") or {}).get("exception_type")

                    if trial.get("exception_info")

                    else None,

                }

                for trial in trials

            ],

        },

        "report": report,

    }

    request = urllib.request.Request(

        f"{arguments.api_url}/api/v3/gym/runs",

        data=json.dumps(payload).encode(),

        headers={

            "Content-Type": "application/json",

            "Authorization": f"Bearer {token}",

        },

        method="POST",

    status, body = request_json(f"{arguments.api_url}/api/v3/gym/runs", token, payload, "POST")

    run = body.get("run") or {}

    print(

        f"{status}: run {run.get('id')} score={run.get('score')} "

        f"replayed={body.get('replayed')}"

    )

    with urllib.request.urlopen(request) as response:

        body = json.loads(response.read())

        run = body.get("run") or {}

        print(

            f"{response.status}: run {run.get('id')} score={run.get('score')} "

            f"replayed={body.get('replayed')}"

        )

    return 0

Environment:

  OPENAGENTS_TOKEN        Required for the Harbor run unless --model starts

                          with ollama/. Required for posting results.

                          with ollama/. Required for posting results and for

                          registering the run against the Gym lifecycle API;

                          without it registration is skipped.

  OPENAGENTS_CODER_API_URL Overrides the coder container API URL.

With OPENAGENTS_TOKEN set, the runner registers the run at

POST <api-url>/api/v1/gym/runs/start before Harbor starts, exports

OPENAGENTS_GYM_RUN_ID and OPENAGENTS_GYM_API_URL into the Harbor run so the

adapter reports each trial live, and finalizes through post_gym_run.py

--run-id. A suite that fails before grading patches the run to abandoned.

Examples:

  bench/run-suite.sh bench/suites/tb2-cross-section.txt \

    --model openai/gpt-5.6-luna --lane proxy --n-concurrent 2

  echo "$url" | sed -E 's#(https?://)(localhost|127\.0\.0\.1)(:[0-9]+)?#\1host.docker.internal\3#'

}

# Register the run against the Gym lifecycle API and print its id

# (OpenAgentsInc/openagents#38). Prints nothing on failure: the suite still

# runs, and the post-hoc one-shot path still posts the graded result.

register_gym_run() {

  python3 - "$API_URL" "$SUITE_NAME" "$CATALOG_MODEL" "$LANE" "$1" <<'PY'

import json, os, sys, urllib.request

api_url, suite, model, lane, tasks_total = sys.argv[1:6]

payload = {

    "suite": suite,

    "agent": "openagents-coder",

    "model": model,

    "lane": lane,

    "tasks_total": int(tasks_total),

}

request = urllib.request.Request(

    f"{api_url}/api/v1/gym/runs/start",

    data=json.dumps(payload).encode(),

    headers={

        "Content-Type": "application/json",

        "Authorization": f"Bearer {os.environ.get('OPENAGENTS_TOKEN', '')}",

    },

    method="POST",

)

try:

    with urllib.request.urlopen(request, timeout=15) as response:

        body = json.loads(response.read() or b"{}")

        run_id = (body.get("run") or {}).get("id")

        if run_id:

            print(run_id)

except Exception as error:

    print(f"gym run registration failed: {error}", file=sys.stderr)

PY

}

# Close a registered run without grades. A crashed suite is not a grade, but

# it should not be a forever-running row on /gym either.

abandon_gym_run() {

  [ -n "${GYM_RUN_ID:-}" ] || return 0

  log "Marking Gym run $GYM_RUN_ID abandoned..."

  python3 - "$API_URL" "$GYM_RUN_ID" <<'PY'

import json, os, sys, urllib.request

api_url, run_id = sys.argv[1:3]

request = urllib.request.Request(

    f"{api_url}/api/v1/gym/runs/{run_id}",

    data=json.dumps({"status": "abandoned"}).encode(),

    headers={

        "Content-Type": "application/json",

        "Authorization": f"Bearer {os.environ.get('OPENAGENTS_TOKEN', '')}",

    },

    method="PATCH",

)

try:

    with urllib.request.urlopen(request, timeout=15):

        pass

except Exception as error:

    print(f"gym run abandon failed: {error}", file=sys.stderr)

PY

}

# Argument defaults

SUITE_FILE=""

MODEL=""

SUITE_NAME="$(basename "$SUITE_FILE" .txt)"

TIMESTAMP="$(date +%Y%m%d-%H%M%S)"

# The catalog name the run registers under: Harbor spells models

# provider/name, the catalog id is the name, and an ollama/ model is the

# coder's `ollama:<name>` local-lane shape — the same mapping the adapter

# applies.

if [[ "$MODEL" == ollama/* ]]; then

  CATALOG_MODEL="ollama:${MODEL#ollama/}"

else

  CATALOG_MODEL="${MODEL#*/}"

fi

if [ -z "$JOBS_DIR" ]; then

  JOBS_DIR="/tmp/gym-jobs-${TIMESTAMP}"

fi

  echo "[dry-run] Pack command:"

  echo "[dry-run]   (cd $(printf '%q' packages/openagents-cli) && pnpm build && pnpm pack --pack-destination $(printf '%q' ../../bench))"

  echo

  if [ -n "$OPENAGENTS_TOKEN" ]; then

    echo "[dry-run] Register command:"

    echo "[dry-run]   POST $API_URL/api/v1/gym/runs/start {\"suite\": \"$SUITE_NAME\", \"agent\": \"openagents-coder\", \"model\": \"$CATALOG_MODEL\", \"lane\": \"$LANE\", \"tasks_total\": ${#TASKS[@]}}"

    echo "[dry-run]   (exports OPENAGENTS_GYM_RUN_ID=<run-id> and OPENAGENTS_GYM_API_URL=$API_URL into the Harbor run)"

  else

    echo "[dry-run] Register command: skipped (OPENAGENTS_TOKEN is not set)"

  fi

  GYM_ENV_ARGS=()

  RUN_ID_SUFFIX=""

  if [ -n "$OPENAGENTS_TOKEN" ]; then

    GYM_ENV_ARGS=("OPENAGENTS_GYM_RUN_ID=<run-id>" "OPENAGENTS_GYM_API_URL=$API_URL")

    RUN_ID_SUFFIX=" --run-id '<run-id>'"

  fi

  echo

  echo "[dry-run] Harbor command:"

  echo -n "[dry-run]   "

  printf '%q ' \

    "PYTHONPATH=$BENCH_DIR" \

    "OPENAGENTS_TOKEN=$TOKEN_DISPLAY" \

    "OPENAGENTS_CODER_API_URL=$CANDIDATE_CODER_API_URL" \

    ${GYM_ENV_ARGS[@]+"${GYM_ENV_ARGS[@]}"} \

    harbor run \

    "${HARBOR_ARGS[@]}"

  echo

  echo

  echo "[dry-run] Post command (after locating the job directory under $JOBS_DIR):"

  echo "[dry-run]   python3 $(printf '%q' "$BENCH_DIR/post_gym_run.py") <job-dir> --api-url $(printf '%q' "$API_URL") --lane $(printf '%q' "$LANE") --suite $(printf '%q' "$SUITE_NAME")"

  echo "[dry-run]   python3 $(printf '%q' "$BENCH_DIR/post_gym_run.py") <job-dir> --api-url $(printf '%q' "$API_URL") --lane $(printf '%q' "$LANE") --suite $(printf '%q' "$SUITE_NAME")$RUN_ID_SUFFIX"

  if [ -n "$OPENAGENTS_TOKEN" ]; then

    echo

    echo "[dry-run] On a suite failure before grading:"

    echo "[dry-run]   PATCH $API_URL/api/v1/gym/runs/<run-id> {\"status\": \"abandoned\"}"

  fi

  exit 0

fi

  exit 1

fi

# Register the run so /gym shows it while the trials are still executing.

# Without a token (ollama dry runs) there is nothing to register against;

# a failed registration degrades to the post-hoc one-shot path.

GYM_RUN_ID=""

if [ -n "$OPENAGENTS_TOKEN" ]; then

  log "Registering Gym run at $API_URL..."

  GYM_RUN_ID="$(register_gym_run "${#TASKS[@]}" || true)"

  if [ -n "$GYM_RUN_ID" ]; then

    log "Gym run id: $GYM_RUN_ID"

  else

    log "Gym run registration failed; continuing without live run reporting."

  fi

else

  log "OPENAGENTS_TOKEN is not set; skipping Gym run registration."

fi

log "Running Harbor suite: $SUITE_NAME (${#TASKS[@]} tasks)..."

(

if ! (

  export PYTHONPATH="$BENCH_DIR"

  export OPENAGENTS_TOKEN

  if [ -n "${OPENAGENTS_CODER_API_URL:-}" ]; then

  else

    export OPENAGENTS_CODER_API_URL="$CANDIDATE_CODER_API_URL"

  fi

  if [ -n "$GYM_RUN_ID" ]; then

    # The adapter reports trials host-side, so it takes the host api url,

    # not the container-rewritten OPENAGENTS_CODER_API_URL.

    export OPENAGENTS_GYM_RUN_ID="$GYM_RUN_ID"

    export OPENAGENTS_GYM_API_URL="$API_URL"

  fi

  harbor run "${HARBOR_ARGS[@]}"

)

); then

  log "Harbor run failed."

  abandon_gym_run

  exit 1

fi

# Locate the completed job directory. Harbor creates a child directory under

# --jobs-dir, but if it writes directly into the directory, use that.

if [ -z "$JOB_DIR" ] || [ ! -d "$JOB_DIR" ]; then

  log "Could not locate Harbor job directory under $JOBS_DIR"

  abandon_gym_run

  exit 1

fi

log "Harbor job directory: $JOB_DIR"

# Post the graded result.

# Post the graded result. With a registered run this is the finalize step:

# post_gym_run.py upserts each trial's final state and patches the run to

# graded, or to abandoned when no verifier ran.

if [ -z "$OPENAGENTS_TOKEN" ]; then

  log "OPENAGENTS_TOKEN is not set; skipping result post."

  log "To post later, run:"

  exit 0

fi

RUN_ID_ARGS=()

if [ -n "$GYM_RUN_ID" ]; then

  RUN_ID_ARGS=(--run-id "$GYM_RUN_ID")

fi

log "Posting result to $API_URL..."

python3 "$BENCH_DIR/post_gym_run.py" "$JOB_DIR" --api-url "$API_URL" --lane "$LANE" --suite "$SUITE_NAME"

if ! python3 "$BENCH_DIR/post_gym_run.py" "$JOB_DIR" --api-url "$API_URL" --lane "$LANE" --suite "$SUITE_NAME" ${RUN_ID_ARGS[@]+"${RUN_ID_ARGS[@]}"}; then

  log "Result post failed."

  abandon_gym_run

  exit 1

fi

log "Suite run complete: $SUITE_NAME"