Skip to content

python(feat): expose include_received_at in get_data - #754

Draft
wei-qlu wants to merge 2 commits into
mainfrom
python/expose-ingest-time
Draft

python(feat): expose include_received_at in get_data#754
wei-qlu wants to merge 2 commits into
mainfrom
python/expose-ingest-time

Conversation

@wei-qlu

@wei-qlu wei-qlu commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Exposes GetDataRequest.include_received_at field

  • client.channels.get_data(...), get_data_as_arrow(...), and Channel.data(...) now take include_received_at
  • When set, each returned DataFrame gains a "<channel_name>.sift_received_at" column with the time each point was received by Sift, NaT for points without one
  • Not available for enum or bitfield channels (their protos carry no extras field)
  • Received-at data is never cached, so these calls always fetch from the server

Hardening for the new nullable column in _merge_pages:

  • Row-level dedup on duplicate timestamps instead of per-column groupby.last(), which skips nulls and could pair a fresh value with a stale row's received-at
  • Explicit datetime64[ns, UTC] dtype so an all-NaT page doesn't degrade the merged column to object dtype

Test plan

  • Extras parsing: prefixed column, NaT for unset points, foreign extras ignored, length-mismatch guard, real bitfield/enum payloads unaffected
  • Request flag set vs unset on the proto, cache bypass (read and write)
  • Page merge with duplicate timestamps and all-NaT pages
  • Flag forwarding pinned at each layer: Channel.data, ChannelsAPI.get_data, low-level client
image
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Python docs preview: https://sift-stack.github.io/sift/python/pr-754/

Deployed from f44c5bb. The link may take up to a minute to become live as GitHub Pages propagates.

@wei-qlu
wei-qlu force-pushed the python/expose-ingest-time branch from fdb91f4 to 9b4f340 Compare August 26, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant