[FLINK-40169][table] Add target option to the EARLY_FIRE hint by weiqingy · Pull Request #28827 · apache/flink

weiqingy · 2026-07-27T02:44:23Z

Part of the FLIP-497 implementation stack under umbrella FLINK-36953. Landing order:

Step	Sub-task	Scope
PR-1a	FLINK-40167	EARLY_FIRE hint surface + option validation (#28353, merged)
PR-1b	FLINK-40168	Thread the hint into the interval join (#28796, merged)
PR-2 (this PR)	FLINK-40169	`target` option
PR-3	FLINK-40170	Update-producing changelog mode + insert-only guard
PR-4	FLINK-40171	Runtime early-fire emit + retraction
PR-5	FLINK-40172	Processing-time early fire on an event-time join
PR-6	FLINK-40173	State restore coverage
PR-7	FLINK-40174	User-facing documentation

What is the purpose of the change

Adds an optional target option to the EARLY_FIRE hint so a hint can be explicitly scoped to a single operator kind. Only interval_join is accepted today, and an omitted target means interval_join, so existing hints keep their meaning. This is a forward-compatibility guard: it keeps a bare EARLY_FIRE hint from silently expanding its scope if other operators honor the hint in the future.

Brief change log

Add EarlyFireJoinHintOptions.TARGET (optional stringType) and the INTERVAL_JOIN constant.
The EARLY_FIRE KV option checker validates target against the supported set. Any other value fails planning.
StreamPhysicalIntervalJoinRule applies the hint only when it targets the interval join, and leaves a hint aimed at another operator kind untouched. That rule-level check is redundant with validation today, since an unsupported value already fails planning. It is there so that a hint aimed at a future operator kind is ignored by this rule rather than misapplied.

Verifying this change

This change added tests and can be verified as follows:

EarlyFireJoinHintTest: an explicit target='interval_join' still threads earlyFireDelay/earlyFireTimeMode into the exec plan, and an unsupported target value fails planning with a message naming the supported set.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes (a new option on the @PublicEvolving EarlyFireJoinHintOptions)
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes (an option on the FLIP-497 hint)
If yes, how is the feature documented? in the documentation PR at the end of this stack (PR-7)

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code (Anthropic)

flinkbot · 2026-07-27T02:50:56Z

CI report:

7e6ff3a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

weiqingy · 2026-07-27T03:06:37Z

Hi @RocMarshal, opening PR-2 of the FLIP-497 stack early as a draft so it's queued behind #28796.

Since it's stacked, the commit list and diff here also carry #28796's commit for now. Once that one is merged I'll rebase this onto master, leaving only the target option change, and take it out of draft for review.

Thanks!

Add an optional `target` option to the EARLY_FIRE join hint. Only `interval_join` is accepted today; any other value fails planning. An omitted target is equivalent to `interval_join`, so existing hints keep their meaning. `target` scopes the hint to a single operator kind. The interval-join rule consumes the hint only when it targets the interval join and leaves a hint aimed at another operator kind untouched, so an untargeted hint never silently expands its scope.

weiqingy · 2026-08-02T22:18:43Z

Hi @RocMarshal, #28796 is merged, so this one is rebased onto master and out of draft.

One thing worth your eye: the rule-level target check in StreamPhysicalIntervalJoinRule is unreachable from SQL today, since the hint checker already rejects any value other than interval_join. I kept it because it is the piece that makes a hint aimed at a future operator kind get ignored by the interval join rather than misapplied, which is the forward-compatibility contract from the FLIP. Happy to drop it if you would rather not carry unreachable code.

PTAL when you have a moment. Thanks!

weiqingy · 2026-08-03T01:28:02Z

The one failing check looks unrelated to this PR.

test_ci core: RescaleTimelineITCase.testRescaleTerminatedByJobCancelled timed out with java.util.concurrent.TimeoutException: Condition was not met within 10000 ms at RescaleTimelineITCase.java:698, called from :325. The module tally was Tests run: 405, Failures: 0, Errors: 1, Skipped: 2, and that single error is what failed the build. Azure build 77673.

Why it cannot be this change: the PR touches 5 files, all under flink-table/flink-table-api-java and flink-table/flink-table-planner, and neither module is in the core stage's reactor. tools/ci/stage.sh lists no table module in MODULES_CORE, and the core stage runs -pl $MODULES_CORE -am, which pulls in upstream dependencies only. The stage that does cover these files, test_ci table, passed in the same build, as did compile_ci and all four e2e legs.

The failure also looks like a timing flake rather than a real regression. The test races a rescale, a cancelJob, and the scheduler's async history recording, then polls for all of it inside a fixed 10000 ms budget with no latch or barrier. The sibling test 20 lines above already uses 60000 ms for the same kind of wait, with the comment "Generous budget: on a loaded CI leg the unblock-to-finish window can itself exceed 10s".

Retriggering CI to confirm.

weiqingy · 2026-08-03T20:16:09Z

Still red on test_ci core (module flink-runtime), but on a different test than last time. Both legs ran the same tree, since the retrigger commit was empty:

77673: RescaleTimelineITCase.testRescaleTerminatedByJobCancelled, TimeoutException: Condition was not met within 10000 ms at RescaleTimelineITCase.java:698 from :325.
77684: DefaultSchedulerTest.testStoppingCheckpointSchedulerAfterExecutionGraphFinished, expected: null but was: org.apache.flink.runtime.checkpoint.CheckpointCoordinator@1e6cec20 at DefaultSchedulerTest.java:761 from :742. Module tally Tests run: 10124, Failures: 1, Errors: 0, Skipped: 381.

Same source tree, two different failures, so this does not look like a regression from this PR. The PR touches 5 files, all under flink-table/flink-table-api-java and flink-table/flink-table-planner, and neither module is in the core stage's reactor: tools/ci/stage.sh lists no table module in MODULES_CORE, and the core stage runs -pl $MODULES_CORE -am, which pulls in upstream dependencies only. test_ci table, the stage that does cover these files, passed in both builds.

The clearest evidence is on master itself. Build 77188 is a scheduled refs/heads/master run from 2026-07-17 (sourceVersion 80f0587c774), ten days before this PR was opened, and its core log carries the byte-identical signature:

Run 1: RescaleTimelineITCase.testRescaleTerminatedByJobCancelled:324 » AssumptionViolated
Run 2: ...:325->waitUntilConditionWithTimeout:698 » Timeout Condition was not met within 10000 ms.

Same test, same two line numbers, on a commit that does not contain this change.

Both failures also occur outside this PR. I grepped the failed core legs on the PR and master/release mirrors back to early March:

RescaleTimelineITCase.testRescaleTerminatedByJobCancelled fails with the identical two-run signature (run 1 AssumptionViolated, run 2 timeout at the same :325 and :698) on master builds 77188, 75390, 75252, 74704, on release-2.3 builds 76078, 74591, and on the unrelated table-only PR [FLINK-39859][table] Upgrade Calcite to 1.41.0 #28316 (build 76482). FLINK-40150 tracks the assumeTrue plus cluster-rebuild pattern behind the run-1 skip.
DefaultSchedulerTest is a recurring flake at the class level: other methods in it fail on master builds 73422, 74617, 75618, on release-2.3 build 76434, and on the unrelated table-only PR [FLINK-39399] Fix integer overflow in HyperLogLogPlusPlus causing APPROX_COUNT_DISTINCT undercount #27895 (build 73888). I did not find this specific method failing anywhere else, so I would not call it a known flake yet. The leg runs test classes concurrently under CPU contention, which fits the pattern.

One caveat on those counts: only builds whose final result is failed can be enumerated, so a leg that failed and was later re-run green is invisible. The occurrences above are a lower bound.

For what it is worth, flink-runtime/pom.xml has no flink-table dependency at all, so there is no code path from this diff to the failing module.

RocMarshal · 2026-08-07T01:13:35Z

+        // target scopes the hint to one operator kind: this rule applies it only when it targets
+        // the interval join, and leaves a hint aimed at any other operator kind untouched.
+        String target = conf.get(EarlyFireJoinHintOptions.TARGET);
+        if (target != null && !EarlyFireJoinHintOptions.INTERVAL_JOIN.equals(target)) {


Suggested change

if (target != null && !EarlyFireJoinHintOptions.INTERVAL_JOIN.equals(target)) {

if (!EarlyFireJoinHintOptions.INTERVAL_JOIN.equals(target)) {

Thanks for the review. Good catch. I had to add one more piece to make it work.

target had no default, so when someone leaves it out, conf.get returns null and !INTERVAL_JOIN.equals(null) is true. The hint would get thrown away for the common EARLY_FIRE('delay'='5s') form. Six tests catch it, and in two of them the validation errors stop firing, so a bad hint would plan fine instead of failing.

So I gave target a default of interval_join. It is never null now, your line works as written, and the checker in FlinkHintStrategies drops its null check too. Pushed in 559ab5d.

Sorry and thanks @weiqingy for your clarify.

Given that this involves changing a config default, I'm unsure if it conflicts with the ratified FLIP.

That said, the original implementation remains perfectly reasonable to me. If altering this default goes against the design outlined in the FLIP, please revert to the original approach. Thanks.

Sounds good. I reverted it in 7e6ff3a.

Thanks @weiqingy

…_join Declaring TARGET with a default of interval_join puts the "an omitted target means interval_join" contract in the option itself, so the rule and the hint checker both reduce to a plain equality check instead of each re-deriving the contract from a null test.

Copilot

Pull request overview

This PR extends the EARLY_FIRE SQL join hint (part of the FLIP-497 implementation stack) by introducing a target option that scopes the hint to a specific operator kind. Today, only interval_join is accepted (and remains the default), preserving existing behavior while preventing a bare EARLY_FIRE hint from unintentionally expanding scope if additional operators support it in the future.

Changes:

Add EarlyFireJoinHintOptions.TARGET (defaulting to interval_join) and the INTERVAL_JOIN constant in the @PublicEvolving hint options.
Validate target in the EARLY_FIRE KV option checker to reject unsupported values at planning time.
Ensure StreamPhysicalIntervalJoinRule only applies the hint when target=interval_join, plus add/extend planner tests to cover explicit and unsupported targets.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/hints/stream/EarlyFireJoinHintTest.xml	Adds golden-plan coverage for an explicit `target='interval_join'` EARLY_FIRE hint.
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/hints/stream/EarlyFireJoinHintTest.java	Adds test cases for unsupported `target` validation and explicit `interval_join` targeting.
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalIntervalJoinRule.java	Applies EARLY_FIRE only when the hint targets interval joins.
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/hint/FlinkHintStrategies.java	Extends EARLY_FIRE hint option validation to check `target`.
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/EarlyFireJoinHintOptions.java	Introduces the `TARGET` option and `INTERVAL_JOIN` constant in the public hint-option surface.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Restore the explicit null handling for an omitted target so the option surface stays as designed: target is optional with no default, and both the hint checker and the interval-join rule treat an absent target as interval_join. Scope the INTERVAL_JOIN javadoc to the operator kind supported today, since target exists so that a future operator kind can opt in explicitly.

RocMarshal

Given the lack of further responses over the past days,
we will proceed with merging this PR to keep things moving. If there are additional suggestions, please feel free to submit a hotfix or follow-up PR.

Thanks!

weiqingy force-pushed the FLINK-36953-pr2-target branch from 0ef163a to 3b14d56 Compare August 2, 2026 22:18

weiqingy marked this pull request as ready for review August 2, 2026 22:18

weiqingy mentioned this pull request Aug 2, 2026

[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join #28877

Merged

1 task

[FLINK-40169][table] Trigger CI re-run

a29e102

[FLINK-40169][table] Trigger CI re-run

dcd7a45

RocMarshal reviewed Aug 7, 2026

View reviewed changes

RocMarshal requested a lite review from Copilot August 7, 2026 10:39

Copilot started reviewing on behalf of RocMarshal August 7, 2026 10:40 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Comment thread ...table-api-java/src/main/java/org/apache/flink/table/api/config/EarlyFireJoinHintOptions.java Outdated

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 7, 2026

RocMarshal approved these changes Aug 8, 2026

View reviewed changes

RocMarshal merged commit 75d978e into apache:master Aug 8, 2026

This was referenced Aug 10, 2026

[FLINK-40171][table-runtime] Emit and retract early-fire results in the interval join operator #28952

Merged

[FLINK-40172][table-runtime] Support processing-time early fire on a row-time interval join #28953

Open

weiqingy mentioned this pull request Aug 30, 2026

[FLINK-40173][table-planner] Add restore coverage for early-fire interval join #29045

Draft

1 task

Navigation Menu

Uh oh!

[FLINK-40169][table] Add target option to the EARLY_FIRE hint - #28827

[FLINK-40169][table] Add target option to the EARLY_FIRE hint#28827
RocMarshal merged 5 commits into
apache:masterfrom
weiqingy:FLINK-36953-pr2-target

weiqingy commented Jul 27, 2026 •

edited

Loading

flinkbot commented Jul 27, 2026 •

edited

Loading

weiqingy commented Jul 27, 2026

weiqingy commented Aug 2, 2026 •

edited

Loading

weiqingy commented Aug 3, 2026

weiqingy commented Aug 3, 2026

RocMarshal Aug 7, 2026

weiqingy Aug 7, 2026

RocMarshal Aug 7, 2026

weiqingy Aug 7, 2026

RocMarshal Aug 8, 2026

Copilot AI left a comment

Uh oh!

RocMarshal left a comment

Labels

4 participants

	if (target != null && !EarlyFireJoinHintOptions.INTERVAL_JOIN.equals(target)) {
	if (!EarlyFireJoinHintOptions.INTERVAL_JOIN.equals(target)) {

Uh oh!

Conversation

weiqingy commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

flinkbot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

weiqingy commented Jul 27, 2026

weiqingy commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

weiqingy commented Aug 3, 2026

weiqingy commented Aug 3, 2026

RocMarshal Aug 7, 2026

Choose a reason for hiding this comment

weiqingy Aug 7, 2026

Choose a reason for hiding this comment

RocMarshal Aug 7, 2026

Choose a reason for hiding this comment

weiqingy Aug 7, 2026

Choose a reason for hiding this comment

RocMarshal Aug 8, 2026

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Labels

4 participants

weiqingy commented Jul 27, 2026 •

edited

Loading

flinkbot commented Jul 27, 2026 •

edited

Loading

weiqingy commented Aug 2, 2026 •

edited

Loading