Skip to content

fix(observability): retry recording upload connection failures - #2350

Open
rosetta-livekit-bot[bot] wants to merge 1 commit into
claude/quirky-galileo-W2Ccnfrom
hurdler-palmy-frayed
Open

fix(observability): retry recording upload connection failures#2350
rosetta-livekit-bot[bot] wants to merge 1 commit into
claude/quirky-galileo-W2Ccnfrom
hurdler-palmy-frayed

Conversation

@rosetta-livekit-bot

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Ports livekit/agents#6977.

Summary

  • retry DNS, connection, and connection-timeout failures before an upload request is transmitted
  • keep RetryInfo response retries and cap both retry paths at three retries with jittered connection backoff
  • scope 900-second total and 30-second connection timeouts to session recording uploads
  • avoid retrying TLS failures, total timeouts, and post-connect disconnects
  • add a patch changeset for @livekit/agents

Testing

  • pnpm test -- agents (49 files, 728 passed, 2 skipped)
  • pnpm build:agents
  • pnpm --filter @livekit/agents lint
  • pnpm exec prettier --check "agents/src/**/*.{ts,tsx,md,json}"
Source diff coverage
  • Adapted: livekit-agents/livekit/agents/telemetry/traces.py -> agents/src/telemetry/traces.ts. Implemented the same retry and timeout policy using Node form-data/HTTPS request lifecycle events: pre-connect coded network failures are retryable, TLS and post-connect failures are not, and total/connect timers are local to this upload.
  • Not applicable: livekit-agents/livekit/agents/utils/http_context.py. agents-js has no shared aiohttp-style session or global 900-second timeout to restore; the parity outcome is the upload-local timeout implementation above.
  • Not applicable: tests/test_http_context_helper.py. There is no target shared HTTP context counterpart; timeout scoping is covered in the telemetry upload tests.
  • Adapted: tests/test_recording.py -> agents/src/telemetry/traces.test.ts. Ported every added behavior: extended upload timeout configuration, connection failure retries, RetryInfo retries, non-RetryInfo failure, no retry for total timeout/post-send disconnect/TLS failure, and retry exhaustion.

Ported from livekit/agents#6977

Original PR description

Recording/session report uploads can fail before request transmission. Retry DNS, connection, and connection-timeout failures, plus responses that explicitly include protobuf RetryInfo. Do not retry total timeouts or post-send disconnects because the upload POST is not idempotent.

Keep the 900-second timeout on this upload only. Other users of the shared HTTP session use aiohttp defaults again.

Addresses AGT-3354

Initial prompt and agent context

Model: GPT-5.6

do we have retry when recording upload failed due to timeout?

we should add retry at http level?

yes please. then we can restore the original default timeout and only use 900s timeout for uploading too, right?

okay, create a PR please. Uploads refer to the recording/session report upload, right?

sg, we should honor RetryInfo, but I don't think we set them properly today. yeah, limit the retry scope as well.

@changeset-bot

changeset-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f0735cb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 28 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

0 participants