[FLINK-40422][python] Add slicing APIs to DataFrame API - #29033
[FLINK-40422][python] Add slicing APIs to DataFrame API#29033Milesian111 wants to merge 3 commits into
Conversation
Add limit, offset, and head as lazy DataFrame transformations backed by the existing Table fetch and offset APIs. Validate that n is a non-negative integer and delegate head to limit to keep their behavior consistent. Document ordering semantics and add tests for delegation, pagination, zero-length slices, invalid arguments, laziness, and deterministic batch results. Generated-by: Codex GPT-5
| Keep at most the first ``n`` rows. | ||
|
|
||
| This is a lazy transformation. Row order is deterministic only when the underlying table | ||
| has an explicit ordering. |
There was a problem hiding this comment.
I suggest explaining or referencing what lazy transformation means here.
Can we expose orderBy as well?
nit: I assume that changes to the underlying table content could make the row order not deterministic.
There was a problem hiding this comment.
Thanks for the suggestions.
I will clarify that a lazy transformation only builds and returns a new
DataFrame plan without starting a Flink job. Execution is triggered later by
an action such as collect() or to_pandas().
Ordering support is already tracked separately by FLINK-40421. FLIP-591
specifies DataFrame.sort(), which will delegate to the existing
Table.order_by() API. To avoid introducing two overlapping public
DataFrame APIs and to keep this PR scoped to FLINK-40422, I suggest leaving
sorting support to FLINK-40421. Once implemented, users will be able to write:
df.sort("ts", descending=True).limit(100)
I will also reword the ordering note to clarify that, without explicit
sorting, the selected rows and their order are unspecified. Changes to the
underlying table content may naturally change the result, and fully stable
ordering also requires ordering keys that uniquely determine row order.
Would clarifying the lazy semantics and referencing FLINK-40421 address the
concern?
There was a problem hiding this comment.
I have pushed a follow-up commit that clarifies the lazy transformation
semantics and the ordering behavior for limit, offset, and head.
Sorting support remains tracked separately by FLINK-40421, so I did not add a
separate DataFrame.order_by() API to this PR.
Generated-by: Codex GPT-5
The Table API requires offset to be combined with a finite fetch for executable plans. Keep the pagination coverage while removing the unsupported standalone offset execution test. Generated-by: Codex GPT-5
854df45 to
10fc884
Compare
What is the purpose of the change
This pull request implements FLINK-40422 by adding row-slicing transformations to the PyFlink DataFrame API.
The new APIs reuse the existing Table API operations and remain lazy:
DataFrame.limit(n)delegates toTable.fetch(n).DataFrame.offset(n)delegates toTable.offset(n).DataFrame.head(n)delegates toDataFrame.limit(n).offsetandlimitcan be composed for pagination. The documentation clarifies that row order is deterministic only when the underlying table has an explicit ordering.Brief change log
DataFrame.limit,DataFrame.offset, andDataFrame.headAPIs.nto be a non-negative integer.headas a delegating wrapper aroundlimit.offset(a).limit(b)headandlimitVerifying this change
This change added tests and can be verified as follows:
headdelegation, pagination composition, zero-length slices, and invalid arguments.limit,offset,offset(a).limit(b), equivalence betweenheadandlimit, andn == 0.Local execution of the batch integration tests was blocked by pre-existing invalid Java planner artifacts in the local build environment. The added batch tests are included for verification in CI.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): yesDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Codex GPT-5