Skip to content

test: Add MCP 2026-07-28 integration and conformance coverage #1132

Description

@MQ37

Part of #1128. Depends on #1140. Must merge and pass before the hosted stateless route reaches production.

Scope

  • Extend tests/integration/suite.ts with the dated transport value 2026-07-28.
  • Add a v2 SDK Client using automatic version negotiation.
  • Run protocol-neutral tool, prompt, resource, structured-output, payment, and error scenarios against the new transport.
  • Keep session lifecycle, Tasks, restore, standalone SSE, and termination scenarios restricted to legacy transports using the existing it.runIf and it.skipIf patterns.
  • Pin the official @modelcontextprotocol/conformance package to an exact compatible version as a development dependency.
  • Add a repeatable package command that runs the conformance server suite against the local development server.

Implementation decisions

  • Packages, pinned exact: @modelcontextprotocol/client@2.0.0-beta.5 (must match the pinned server beta) and @modelcontextprotocol/conformance@0.2.0-alpha.9 (the latest 0.1.16 does not know 2026-07-28; newer alphas are blocked by pnpm minimumReleaseAge until they age past 3 days).
  • Client typing: union, no adapter. createClientFn returns Client | StatelessClient (v1 @modelcontextprotocol/sdk client union v2 @modelcontextprotocol/client client). The suite only calls callTool, listTools, listPrompts, readResource, and close, all callable through the union. The client.experimental.tasks call sites (legacy-only Tasks cases) need an explicit narrow to the v1 client — add one guard helper and use it in those describe blocks. If result-type unions cause heavy assertion friction, the fallback is a v1-shaped facade over the v2 client in tests/helpers.ts; both hide behind the same createClientFn signature.
  • Follow-up (test: Migrate integration suite to the v2 SDK client #1169): migrate the whole suite to the v2 client (it negotiates both protocol eras via mode: 'auto') once v2 leaves beta, removing the union and the v1 client from the suite.
  • v2 client auth: verify the v2 StreamableHTTPClientTransport can send Authorization: Bearer; if awkward, use the ?token= query parameter the server already accepts.

Conformance command

The package script must execute the equivalent of:

pnpm exec conformance server --url <dev-server-url> --suite all --spec-version 2026-07-28

The script owns the full lifecycle: build, start the dev server (PORT=3001, default config), wait for ready, run the suite, tear down, and propagate the exit code — the same command locally and in CI. The runner has no header flag, so the Apify token rides the URL: --url "http://127.0.0.1:3001/?token=$APIFY_TOKEN".

Record explicit expected exclusions per scenario name with a one-line reason each, in three categories:

  1. Not applicable — scenarios requiring the reference fixture server's tools/resources/prompts by name (test_simple_text, test://static-text, input-required-result-*, x-mcp-header, ...).
  2. Unsupported extensions — Tasks, subscriptions/listen.
  3. Upstream SDK bugs — known at 2.0.0-beta.5: server/discover nests serverInfo under _meta, and a request missing clientInfo is not rejected. Link each to an upstream modelcontextprotocol/typescript-sdk issue.

Do not hide unexpected failures behind a blanket expected-failures file. The command must fail on any failure not in the list and on any listed scenario that unexpectedly passes, so entries get pruned when SDK bumps fix them.

Test sequence

  1. Build the package because the MCP probes require compiled output.
  2. Start the development server (a single endpoint serves both protocol revisions; there is no per-revision toggle).
  3. Run the v2 integration client against the 2026-07-28 route.
  4. Run the official conformance suite against that same endpoint.
  5. Run mcpc against @stdio to verify legacy behavior.

mcpc is only the legacy smoke test because it cannot negotiate 2026-07-28. It does not replace the v2 client or conformance runner.

CI

The integration dimension runs where the suite already runs: _integration_tests.yaml, with APIFY_TEST_USER_API_TOKEN and the existing fork-PR skip. Add the conformance command as a step in the same workflow, reusing the token from the environment. Use the default server config (including the apify/rag-web-browser preload). Note: the dev server rebuilds the facade per stateless request (~0.6–2.2s); if conformance runs ever flap on timeouts, the lever is a slim ?tools= config for the conformance target only.

Internal repo impact

Internal issue apify/apify-mcp-server-internal#677 adds the same dated transport dimension to the hosted contract suite, using the identical transport value 2026-07-28 so the two suites stay aligned. Both suites must pass before apify/apify-mcp-server-internal#676 is released.

Verification checklist

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-aiIssues owned by the AI team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions