fix: keep add event stream alive

[wachiravit-thitagran]

Warning

This pull request and its code were generated by AI (OpenAI Codex). Please review the implementation and tests carefully before merging.

Problem

POST /api/v1/add can spend longer than Cloudflare's default 125-second Proxy Read Timeout processing an uploaded document without emitting another Server-Sent Events (SSE) body chunk. Cloudflare defines this timeout between successive reads from the origin, so an otherwise healthy long-running ingestion can be terminated if the stream stays silent for too long.

An interrupted stream may appear in Chromium-based clients after the response headers have already returned HTTP 200 as:

net::ERR_HTTP2_PROTOCOL_ERROR 200 (OK)

This is an observed symptom, not proof by itself that Cloudflare is the root cause. Confirming attribution requires comparing the proxied request with a direct-origin request and checking Cloudflare logs, a CF-Ray/HAR, or a browser NetLog.

Why this approach

The HTML Standard recommends sending an SSE comment line (a line beginning with :) roughly every 15 seconds to protect event streams from proxy timeouts.

This fix therefore emits:

: ping

every 15 seconds while a file is being processed. If Cloudflare forwards each SSE chunk normally, the interval stays well below its 125-second read timeout and is expected to prevent this specific timeout.

This condition matters:

• A normal proxied custom hostname or named Cloudflare Tunnel supports SSE.
• A Quick Tunnel on *.trycloudflare.com does not support SSE and may buffer text/event-stream; heartbeat comments cannot solve that configuration. Use a named tunnel or another SSE-capable route instead.
• This change does not address origin crashes, malformed HTTP/2 frames, compression/content-length errors, or request-upload stalls before the streaming response starts.

What changed

• Run each streamed add operation as an asynchronous task.
• Emit the valid SSE comment : ping\n\n every 15 seconds while processing remains in progress.
• Preserve the existing uploaded, file_start, file_done, final, error, and done event contract.
• Cancel an unfinished task when the client closes the stream.
• Add a regression test that simulates slow processing, verifies the configured interval remains within 15-30 seconds, observes the keepalive comment, and confirms the final events are still delivered.

SSE clients ignore comment-only frames, so the frontend event payload contract is unchanged. Non-streaming requests are also unchanged.

Validation

Application-level validation:

• pytest -q tests/test_api.py tests/test_file_size.py — 158 passed
• mypy openkb/api_helpers.py openkb/api.py — passed
• ruff check . — passed
• ruff format --check . — passed
• git diff --check — passed

The automated test confirms that Starlette emits the keepalive chunks and subsequently completes the SSE stream. It is not an end-to-end Cloudflare integration test.

References

• Cloudflare connection limits
• Cloudflare Proxy Read Timeout setting
• HTML Standard: Server-Sent Events authoring notes
• Cloudflare Tunnel setup and Quick Tunnel limitations
• Cloudflare troubleshooting and NetLog/HAR guidance