pier

Pier is a Harbor-compatible framework for evaluating coding agents in sandboxed environments. It reads Harbor's task format and runs trials against it.

pier run -p path/to/task --agent claude-code --env modal

Pier is a fork. We wanted a smaller, more opinionated base to build on. On top of Harbor, Pier adds:

• Installed agents in air-gapped tasks (allow_internet = false). When the agent runs inside the sandbox (Claude Code, Codex, etc.), both the install step and the inference call need the network. Pier lets agents declare their install scripts and a network allowlist, which docker and modal environments honor when setting up the sandbox.
• Augmented ATIF v1.7. Strict one step per API turn, strict reasoning vs agent message separation, no fabricated assistant text, peak_context_tokens, summarization_count, llm_call_count, real upstream timestamps.
• A chat-style trajectory viewer (pier view).
• pier critique run for inspecting completed trials with a fresh agent in a fresh sandbox.

• Task format: Harbor-compatible.
• Environments: docker, modal. Per-agent install specs and network allowlists are honored on both, so installed agents work under allow_internet = false.
• Agents: nop, oracle, antigravity-sdk, claude-code, codex, cursor-cli, gemini-cli, opencode, mini-swe-agent. All emit augmented ATIF v1.7.
• Datasets: local Harbor-format task directories via -p / --path.
• CLI: pier run, pier job, pier view, pier critique run, pier check / pier analyze (vendored from Harbor)

Pier does not currently resolve or download Harbor registry datasets directly.

uv tool install datacurve-pier
# or
pip install datacurve-pier

export ANTHROPIC_API_KEY=...
pier run -p path/to/task --agent claude-code --env modal --env-file .env

Run a local dataset, optionally a deterministic random subset:

pier run -p path/to/dataset --agent claude-code --env modal
pier run -p path/to/dataset --n-tasks 10 --sample-seed 0

To use a Harbor registry dataset, download it with Harbor first, then point Pier at it:

uv run --directory ~/code/harbor harbor download swebenchpro -o ~/code/pier/datasets
uv run pier run -p datasets/swebenchpro --n-tasks 10 --sample-seed 0

Trials land under jobs/<timestamp_or_name>/<trial_id>/. See pier run --help, pier job --help, pier critique --help, and pier view --help for everything else.

For environments without mounted logs, Pier attempts to copy /logs/verifier/ into the trial's verifier/ directory even when verification fails or times out. A failed copy is logged without replacing the original execution error.

Use agent.model_name for trial metadata, agent.env for runtime env vars, and agent-specific kwargs for tool config. Pier's network allowlist also reads URLs out of those configs (Codex config_toml, OpenCode opencode_config, mini-swe config_yaml), so any base URL you set is allowlisted without code changes.

A few things we've learned plumbing this through Respan and OpenRouter:

Claude Code routes through the Anthropic face from Respan. Plan mode is disabled by default (--disallowedTools EnterPlanMode).

- name: claude-code
  model_name: claude-opus-4-7
  env:
    ANTHROPIC_AUTH_TOKEN: ${RESPAN_API_KEY}
    ANTHROPIC_BASE_URL: https://endpoint.respan.ai/api/anthropic
    ANTHROPIC_CUSTOM_HEADERS: "X-Respan-Route-Provider: vertex_ai"
  kwargs:
    reasoning_effort: max

Codex needs a [model_providers.<name>] block with wire_api = "responses" (not WebSockets, which Codex defaults to and Respan doesn't speak).

- name: codex
  model_name: openai/gpt-5.5
  env: { RESPAN_API_KEY: ${RESPAN_API_KEY} }
  kwargs:
    config_toml: |
      model_provider = "respan"
      [model_providers.respan]
      name = "Respan Gateway"
      base_url = "https://endpoint.respan.ai/api/"
      wire_api = "responses"
      env_key = "RESPAN_API_KEY"
    reasoning_effort: xhigh

Gemini CLI:

- name: gemini-cli
  model_name: gemini/gemini-3.1-pro-preview
  env:
    GEMINI_API_KEY: ${RESPAN_API_KEY}
    GOOGLE_GENERATIVE_AI_API_KEY: ${RESPAN_API_KEY}
    GEMINI_API_BASE: https://endpoint.respan.ai/api/google/vertexai/v1beta
    GOOGLE_GEMINI_BASE_URL: https://endpoint.respan.ai/api/google/vertexai/

Antigravity SDK runs Google's Python SDK with its platform-specific local harness in an isolated Python 3.12 environment with hash-verified, fully locked dependencies. It supports Pier skills, stdio and streamable-HTTP MCP servers, and live ATIF checkpoints. reasoning_effort accepts minimal, low, medium, or high; None uses medium. SSE MCP servers are not supported by google-antigravity 0.1.9.

- name: antigravity-sdk
  model_name: google/gemini-3.6-flash
  env:
    GEMINI_API_KEY: ${GEMINI_API_KEY}
  kwargs:
    reasoning_effort: high

Cursor CLI uses the installed cursor-agent binary, so it fits the same inside-the-sandbox path as Claude Code, Codex, Gemini CLI, and OpenCode. Use cursor/composer-2.5 for Composer 2.5 trial metadata and pass CURSOR_API_KEY through your env file.

- name: cursor-cli
  model_name: cursor/composer-2.5
  env:
    CURSOR_API_KEY: ${CURSOR_API_KEY}

OpenCode uses opencode_config to add unknown providers or override known ones. To redirect Google to Respan, override just options.baseURL; to add a fully custom provider, use opencode_config.provider.<name> with the npm package, options, and models.

mini-swe-agent picks a native adapter from the model-name prefix: openai/... → litellm_response (OpenAI Responses end-to-end), openrouter/... → openrouter (BYOK costs from cost_details.upstream_inference_cost), everything else → LiteLLM auto.

ATIF conversion retains responses rejected by the agent's format validation as agent steps, with their raw content, reasoning, usage, and validation feedback. These responses remain counted once in the token totals.

mini-swe-agent token counts and costs preserve missing values as null and recorded zeros as 0. Per-field totals require usage from every agent turn, trial context, and completed trial. A reported run cost can supply the total when per-turn costs are missing. Retrying a trial updates usage completeness along with its contribution. The viewer leaves uncached input unknown when cache usage is missing, and averages still use the trials that reported each metric.

For Gemini 3 via mini-swe-agent/LiteLLM, omitting reasoning_effort uses the Gemini API default high/dynamic thinking level, but it does not request readable thought summaries. Set kwargs.reasoning_effort: high explicitly when you want LiteLLM to send includeThoughts and preserve returned summaries as reasoning content.

- name: mini-swe-agent
  model_name: openrouter/qwen/qwen3.6-plus
  env: { OPENROUTER_API_KEY: ${OPENROUTER_API_KEY} }
  kwargs:
    set_cache_control: default_end