Skip to content

fix(api_fetch): accept REST-natural shapes for url / params / body - #2

Open
AdarshGoel2001 wants to merge 1 commit into
zapier:mainfrom
AdarshGoel2001:fix/api-fetch-url-query
Open

fix(api_fetch): accept REST-natural shapes for url / params / body#2
AdarshGoel2001 wants to merge 1 commit into
zapier:mainfrom
AdarshGoel2001:fix/api-fetch-url-query

Conversation

@AdarshGoel2001

@AdarshGoel2001 AdarshGoel2001 commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

api_fetch's URL/argument plumbing currently produces silent failures when
agents use the natural REST/curl idiom rather than the bench's strict
"separated URL + JSON-stringified body" convention. Two related symptoms,
one root cause (the tool surface advertises a narrower contract than the
underlying impl actually supports). This PR widens the contract on both
the URL side and the params/body side, so the bench evaluates whether the
agent can do the workflow
, not whether the agent has memorised the
formatting convention
.

Why this matters

The "URL goes here, params go separately, body is a JSON-stringified
string" convention is bench-specific scaffolding — it appears in the
api_fetch tool description and in api_search results, but is otherwise
absent from pretraining. The natural shapes (query inlined in URL; body
as a plain object) dominate REST docs, curl examples, fetch/axios usage,
SDK tutorials, and so on. Frontier models pick up the convention from the
tool description and hold it; smaller / cheaper models default to the
natural shape and slip.

When that happens the failure modes are silent and demoralising — the
agent has typically already done the multi-hop research correctly and only
fails on the write call. From the maintainer's perspective the bench
stops measuring agent capability and starts measuring API-protocol pedantry.

What changes

1. Query strings inlined in the URL

Before: api_fetch("GET", ".../messages?q=from:Jordan") 404s because the
trailing ?q=… survives _url_to_internal_path's prefix-stripping and
the per-app route handlers match by exact path string.

After: a new _merge_url_query_into_params(url, params) helper lifts any
inlined query string out of the URL into the parsed params dict before
routing. URL-decoded; repeated keys preserved as a list; caller-supplied
params win on conflict.

Pre-fix observation (claude-haiku-4-5 via Claude Code subscription harness):
  api_fetch GET https://gmail.googleapis.com/gmail/v1/users/me/messages?q=…
    �� {"error":{"code":404,"message":"No handler for GET ...?q=…"}}
  api_fetch GET https://*.salesforce.com/services/data/v60/query?q=SELECT…
    → {"error":"Missing query parameter 'q'"}

2. Dict-shaped params and body

Before: api_fetch(..., params: Optional[str], body: Optional[str])
LLM-facing tool wrappers (OpenAI tool schema, FastMCP, etc.) advertise a
string-only schema. Loose validators (chat-completions APIs) usually
serialize dicts through to _coerce_to_dict invisibly. Strict validators
(MCP stdio clients, e.g. Claude Code) reject the call client-side before
it reaches api_fetch — the agent never sees the result, eventually
gives up with "I'm experiencing a technical issue with JSON body
parameter formatting."

After: annotations widened to Union[dict, str, None]. The advertised
schema becomes anyOf: [object, string, null]. The impl is unchanged
because _coerce_to_dict already accepts both shapes. The documented
"happy path" stays JSON-stringified — api_search results still model
that shape, no prompt-side tightening — so models trained against the
previous tool description continue to work unchanged. This is purely
tolerance-side.

Pre-fix observation (same harness):
  api_fetch PATCH .../sobjects/Contact/003001  body={"Phone":"+1-555-0101"}
    → silently rejected at the MCP client boundary; never logged,
      never executed, never surfaced as an error to the agent

Tests

  • TestMergeUrlQueryIntoParams — 6 unit tests (no-query passthrough,
    query-only-in-URL, URL-decoding, query+params merge, conflict
    resolution, repeated keys).
  • TestApiFetchAcceptsQueryInUrl — 2 end-to-end regression guards
    (Gmail messages list, Salesforce SOQL query) confirming both call
    styles route equivalently.
  • TestApiFetchAcceptsRestNaturalShapes — 3 tests covering dict
    params, dict body, and a signature-shape regression test that
    fails fast if someone tightens the annotations back to Optional[str]
    (so the strict-MCP-client path doesn't silently break again).

Full existing suite (638 tests under tests/) passes. Pre-existing
unrelated import errors in tests/tools/test_*.py (which import
automationbench.tools.salesforce rather than the actual
automationbench.tools.zapier.salesforce) are present on main and
unaffected by this change.

Non-goals

  • No change to the documented happy path. Tool descriptions, api_search
    result shapes, and _coerce_to_dict's behaviour are all unchanged.
    Strings remain the preferred form.
  • Doesn't widen method or url, only the two args where the dict-vs-string
    ambiguity actually surfaces in observed agent behaviour.

Why a single PR

Both halves are the same family of fix: an LLM-facing schema/contract
narrower than the impl actually supports, causing silent rejection of
the natural agent-output shape. They share the test file, the same
underlying observation (smaller models slip on bench-specific convention),
and the same justification. Reviewing them together keeps the framing
coherent.

Smaller / cheaper agentic models routinely produce the natural REST/curl
idiom — query parameters inlined into the URL, and request bodies as plain
objects — instead of the bench's strict "URL goes here, params/body go
separately, body is a JSON-stringified string" convention. The convention
is bench-specific scaffolding the model picks up only from the tool
description; pretraining is dominated by the natural shape. Frontier models
hold the convention; smaller models slip.

The failure mode is silent and expensive in two ways:

(1) Inlined query strings: the trailing "?…" survives `_url_to_internal_path`'s
    prefix-stripping, the per-app route handlers (which match by exact path
    string) return 404 / "Missing query parameter", and the agent — having
    already done the multi-hop research — gives up.

      api_fetch GET .../gmail/v1/users/me/messages?q=from:Jordan
        → {"error":{"code":404,"message":"No handler for GET ...?q=..."}}
      api_fetch GET .../services/data/v60/query?q=SELECT+...
        → {"error":"Missing query parameter 'q'"}

(2) Dict-shaped params/body: the type annotations `Optional[str]` advertise a
    string-only schema to LLM-facing tool wrappers (OpenAI tool schema,
    FastMCP, etc.). Loose validators (chat-completions APIs) usually serialize
    dicts through to `_coerce_to_dict` invisibly; strict validators (MCP stdio
    clients, e.g. Claude Code) reject the call client-side before it ever
    reaches `api_fetch`. The agent never sees the result, never learns, and
    eventually emits "I'm experiencing a technical issue with JSON body
    parameter formatting" — a polished giving-up signal indistinguishable
    from genuine inability.

      api_fetch PATCH .../sobjects/Contact/003001 body={"Phone":"+1-555-0101"}
        → silently rejected at MCP boundary; never logged, never executed

Both behaviors observed running `claude-haiku-4-5` against the public task
set via Claude Code's subscription harness. The agent had successfully
retrieved all the required data each time but kept hitting these routing /
validation walls on the write-side calls.

Fix:

- `_merge_url_query_into_params(url, params)`: lifts any inlined query string
  out of the URL into the parsed params dict before routing. URL-decodes
  values, preserves repeated keys as lists, caller-supplied params win on
  conflict. Cleaned URL flows through the existing routing.

- Type annotations on `api_fetch.params` and `api_fetch.body` widened from
  `Optional[str]` to `Union[dict, str, None]`. The advertised schema becomes
  `anyOf: [object, string, null]`. Implementation needs no change because
  `_coerce_to_dict` already accepts both shapes; the existing convention
  (JSON-stringified strings) remains the documented preferred form so models
  trained against the previous tool description continue to work unchanged.

Tests:

- `TestMergeUrlQueryIntoParams` — 6 unit tests (no-query passthrough,
  query-only-in-URL, URL-decoding, query+params merge, conflict resolution,
  repeated keys).
- `TestApiFetchAcceptsQueryInUrl` — 2 end-to-end regression guards
  (Gmail messages list, Salesforce SOQL query) confirming both call styles
  route equivalently.
- `TestApiFetchAcceptsRestNaturalShapes` — 3 tests covering dict-shaped
  params / dict-shaped body / a signature-shape regression that fails fast
  if the type annotations get tightened back.

Full existing suite (638 tests under tests/, excluding pre-existing
unrelated `tests/tools/` import errors on main) still passes.

Non-goals:

- The documented "happy path" stays JSON-stringified — `api_search` results
  still model that shape; no docstring or prompt-side wording tightening.
  This is purely about tolerating the natural shape so silent-rejection
  failure modes go away.
@AdarshGoel2001
AdarshGoel2001 force-pushed the fix/api-fetch-url-query branch from d7d98bf to 61fc39a Compare April 29, 2026 12:12
@AdarshGoel2001 AdarshGoel2001 changed the title fix(api_fetch): accept query strings inlined in URL Apr 29, 2026
@AdarshGoel2001
AdarshGoel2001 force-pushed the fix/api-fetch-url-query branch from 5fe2779 to 61fc39a Compare May 4, 2026 16:58
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 22, 2026
v2 (576 tasks, 121 solvers) is the follow-up to v1 (831 tasks, 155 solvers).
v1's critical defect was a single-service monoculture (14 salesforce-only
types, 1 of 10 services). v2 fixes it via coupled count<->service sampling,
capped gap-fill floors, shuffled gap-fill selection, and a per-variant
service-coverage gate.

Key v2 results vs v1:
- Services touched: 1 -> 11 (all eligible services represented)
- Distinct assertion types: 14 -> 41
- Type-diversity max: 7 -> 11 (cross-service tail restored; 32 tasks now
  in the 8-11 bin vs 0 in v1)
- Entity max: 45 -> 50 (far tail restored; matches original)
- Per-variant service gate: 0 rejections (approach zapier#2 sufficient, zapier#3 not
  needed)
- salesforce_collection_count_equals universality: 94% -> 74%

Remaining gap: within-service type vocabulary still narrow (41 vs 123) —
v2 reaches all services but draws a thin slice of each service's registered
types. Next leverage point.

Co-Authored-By: Claude <noreply@anthropic.com>
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 23, 2026
…er#2)

zapier#1 — apply_correct sync contract (prompts.py):
Add a "keep build_initial_state/apply_correct/build_assertions in SYNC"
section to the solver prompt + strengthen the apply_correct docstring.
Targets the zapier#1 yield killer: assertions that "do not hold after
apply_correct" because the LLM invented identifiers (recipient_profile_id
'pf-0001', channel '#sales-enterprise', spreadsheet_id 'sheet_A1') only
in build_assertions, never seeded in initial_state nor produced by
apply_correct. Rules: every assertion identifier must be seeded or
created; apply_correct must perform the exact action each assertion type
checks; derive all identifiers from solve(seed). This also re-covers
canva (collateral damage of the linkedin/slack apply_correct bugs).

zapier#2 — run-wide unused-type tracking (prompts.py + solver_generator.py):
Add a used_types set threaded live through every solver generation
(main loop, resume, gap-fill, orphans). After each accepted solver, its
assertion types accumulate. The solver prompt now shows a "PRIORITIZE
these UNEXPLORED registered types" block — assigned-service types no
accepted solver has used yet, rarest-first — alongside the full menu.
v3 plateaued at 54 of 123 registered types because the LLM reused
workhorses despite seeing the full menu; this makes virgin types
explicit and prioritized.

Co-Authored-By: Claude <noreply@anthropic.com>
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 23, 2026
v4 revealed the generic sync hint helped linkedin (0 failures) but didn't
generalize: apply_correct failures spread to salesforce_field_equals (14),
gmail_*_body_contains (17), slack_channel_exists (12), zoom_meeting_field_equals
(6), and a field-name bug — salesforce_task_not_exists_for used related_to_id
instead of what_id (handler reads None, assertion silently fails).

Make the hint concrete and pattern-specific, citing the exact v4 failure shapes:
- IDENTIFIER PROVENANCE: record_id/meeting_id/etc must be seeded or created,
  never invented (e.g. OPP-5501 that doesn't exist in initial_state).
- EXACT FIELD NAMES: copy verbatim from examples; call out known traps
  (what_id not related_to_id; object/object_type not record_type;
  channel_id/user_id not channel/user).
- apply_correct MUST REALIZE EACH ASSERTION, with per-type mutations:
  *_exists -> APPEND the record to the WorldState list; *_field_equals ->
  SET the field on the existing record; *_sent_to_with_body_contains ->
  append a message whose body contains the text; *_draft_exists -> create
  the draft; *_not_exists -> don't create it; *_count -> match the length.

This targets zapier#2's side effect: the unused-types nudge pushes the LLM into
rarer types, whose apply_correct semantics the generic hint didn't cover.

Co-Authored-By: Claude <noreply@anthropic.com>
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 23, 2026
v4/v5 showed the prompt-hint approach to zapier#1 (apply_correct sync) and the
solver-prompt nudge for zapier#2 (unused-types) fight each other: zapier#2 pushes the
LLM into rarer types whose apply_correct semantics zapier#1's hint can't enforce,
depressing yield (v5: 0 validations in 5 solvers). Revert prompts.py and
solver_generator.py to the v3 state (commit 9331b1a) — the last good run
(104 solvers, 54 types, 0 slack errors).

The two goals will be re-attacked with structural mechanisms that don't
depress the main fleet's yield:
- zapier#2: type-level gap-fill BACKSTOP (runs after the main loop, forces virgin
  types only in extra solvers — not in the main 160).
- zapier#1: solve(seed)->plan contract + a locator-consistency validator gate
  (hard, actionable error instead of a soft hint).

The v3 report + v3_metrics.py are preserved (they were additive).

Co-Authored-By: Claude <noreply@anthropic.com>
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 23, 2026
…utral)

Replaces the failed v4/v5 prompt-nudge approach with two structural
mechanisms that don't depress the main fleet's yield.

zapier#1 — solve->plan contract + locator-consistency validator gate:
- Contract (prompts.py): solve(seed) is now the SINGLE SOURCE OF TRUTH
  carrying canonical record ids/emails/names; build_initial_state,
  apply_correct, build_assertions all derive ids from solve(seed)'s
  return, never hardcoding an id in one function that isn't in the others.
- Validator gate (validator.py): _check_locators runs after apply_correct
  and before the assertion check. For positive existence types, verifies
  the assertion's record-locator (record_id/meeting_id/channel name/to/
  what_id/profile_id/spreadsheet_id) is present in the world (or in
  solve(seed)'s output). Invented locators (the dominant v4/v5 failure:
  'OPP-5501' that doesn't exist) are rejected with an ACTIONABLE error
  naming the missing id + the valid ids present. Negative (*_not_exists)
  types are skipped (their locator is intentionally absent). Hand-mapped
  for ~22 high-frequency failing types; others fall through to the
  generic self-consistency check.

zapier#2 — type-level gap-fill backstop (yield-neutral):
- New _type_gap_fill phase in _ensure_service_coverage, runs AFTER the
  main 160-solver fleet + service gap-fill. Computes registered types no
  accepted solver emitted, and generates EXTRA solvers (new slots) with
  the virgin type(s) FORCED via type_target.
- The main fleet runs with NO type forcing (full v3 yield) — only the
  backstop forces types, so vocabulary widens without the yield collapse
  the v4/v5 solver-prompt nudge caused.
- type_target threaded through solver_user_prompt (MANDATORY type block
  with concrete examples) + a per-solver validator gate that the forced
  type fires in >=1 sample variant. Rarest types first; capped slot
  budget; types that can't be wired after retries stay unused (logged).

Co-Authored-By: Claude <noreply@anthropic.com>
BetaGoal added a commit to Red-Hat-AI-Innovation-Team/AutomationBench that referenced this pull request Aug 23, 2026
…ckstop

v6's first 5 solvers all failed (0/5 validated, 10 does-not-hold) vs v3's
4/5 validated at the same point. Root cause: the solve->plan contract
docstrings I added (solve as 'SINGLE SOURCE OF TRUTH', detailed apply_correct
realization rules) made the contract prompt longer and pushed the LLM toward
over-engineered solvers (state-machine transition maps, etc.) with more
logic-bug surface area — depressing main-fleet yield below v3.

Revert the solve/build_initial_state/apply_correct/build_assertions docstrings
back to v3's short versions. The main-fleet prompt is now byte-identical to
v3 (the only prompts.py diff vs 9331b1a is the type_target block, which is
inactive for the main fleet — type_target defaults to empty).

Kept (these don't touch main-fleet generation behavior):
- zapier#1's locator-consistency validator gate (validator.py): catches invented
  record-locator IDs with actionable errors. A pure check, doesn't change
  what the LLM generates — only enriches failure feedback.
- zapier#2's type-gap-fill backstop (solver_generator.py): runs AFTER the main
  fleet, forces virgin types only in extra solvers.

So v7 = v3 main-fleet yield + the two structural backstops.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant