Add tests for markdown serving based on user agents

[conico974]

Implement tests to verify that the application serves markdown content correctly based on different user agents, ensuring proper handling for AI agents and traditional bots.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3e3b0bf

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

Summary of the deployments:

Version	URL	Status
Vercel	https://gitbook-v2-97as8pvhl-gitbook.vercel.app	✅
Cloudflare	https://2e1d44a7-gitbook-open-v2-preview.gitbook.workers.dev	✅

Test content

Site	2v	2c
GitBook	https://gitbook-v2-97as8pvhl-gitbook.vercel.app/url/gitbook.com/docs	https://2e1d44a7-gitbook-open-v2-preview.gitbook.workers.dev/url/gitbook.com/docs
E2E	https://gitbook-v2-97as8pvhl-gitbook.vercel.app/url/gitbook.gitbook.io/test-gitbook-open	https://2e1d44a7-gitbook-open-v2-preview.gitbook.workers.dev/url/gitbook.gitbook.io/test-gitbook-open

[argos-ci]

The latest updates on your projects. Learn more about Argos notifications ↗︎

Build	Status	Details	Updated (UTC)
customers-v2-cloudflare (Inspect)	✅ No changes detected	-	Apr 30, 2026, 7:07 AM
customers-v2-vercel (Inspect)	✅ No changes detected	-	Apr 30, 2026, 7:05 AM
v2-cloudflare (Inspect)	⚠️ Changes detected (Review)	6 changed	Apr 30, 2026, 7:13 AM
v2-vercel (Inspect)	⚠️ Changes detected (Review)	6 changed	Apr 30, 2026, 7:12 AM

[SamyPesse]

Approving but move the tests

[SamyPesse]

why put these tests in the tests for robots.txt 🤔 ?

markdown.test.ts would be better