Clarifying merge queue behavior with "Only merge non-failing pull requests" disabled

[timothycobrien]

Select Topic Area

Question

Body

I'm trying to understand exactly how merge queue handles failures when "Only merge non-failing pull requests" is disabled, particularly around group formation and PR ejection.
My understanding:

With the setting disabled, only the last PR in a group needs to pass for the group to merge
This is useful for flaky tests since a passing "last PR" validates the combined changeset

What I'm unclear on:

Question 1: When does a failing PR get ejected? I observed that a failing PR was removed from the queue after I manually removed the only PR behind it. This suggests "keep failing PRs around as long as there's a later PR that might pass." Is that accurate?

Question 2: How do groups interact with ejection? If max group size is 3:

Group A: PR #1, #2, #3
Group B: PR #4, #5, #6

If all 3 PRs in Group A fail, does the queue remove PR #1 from Group A once it's full? Or does it wait for Group B to also fail as long as those checks are still running?

The "Failing CI" section only describes behavior when "Only merge non-failing pull requests" is enabled. What happens when it's disabled and the last PR in the queue fails? Does it eventually get ejected, or does it persist as long as new PRs keep joining?

Question 3: Same scenario but PR #1 fails, PR #2 passes, PR #3 fails. What happens and when?

Trying to find the right balance between tolerating flaky tests and ensuring real bugs get ejected in a reasonable timeframe.

[vanhuy2005]

Hi @timothycobrien,

Here is my answer based on your scenarios

The Core Logic
With this setting disabled, the Merge Queue operates on a "Last PR Wins" logic. It assumes that if Base + PR #1 + PR #2 passes CI, then the resulting state is valid, regardless of whether Base + PR #1 (on its own) was failing.

Answer to Question 1: When does a failing PR get ejected?
Your observation is correct.

• Scenario: PR Welcome to GitHub product feedback #1 (Failing) is followed by PR Feature Request: Select a column when adding an issue to a project #2 (Pending/Passing).
• Behavior: The queue creates a temporary branch for PR Feature Request: Select a column when adding an issue to a project #2 that includes PR Welcome to GitHub product feedback #1. As long as PR Feature Request: Select a column when adding an issue to a project #2 is in the queue, the system is hoping that the check for PR #2 will pass. If it does, both Welcome to GitHub product feedback #1 and Feature Request: Select a column when adding an issue to a project #2 merge.
• Why it ejected when you removed Feature Request: Select a column when adding an issue to a project #2: Once PR Feature Request: Select a column when adding an issue to a project #2 was gone, PR Welcome to GitHub product feedback #1 became the "Last PR" in that group. Since the "Last PR" failed, the queue deemed the state invalid and ejected Welcome to GitHub product feedback #1.

Answer to Question 2: Group interactions (Group A vs. Group B)
The queue processes groups sequentially regarding success/failure.

• If Group A (#1, #2, #3) is running, the queue checks the status of the tip of that group (PR Feature request: support release details #3).
• If PR Feature request: support release details #3 fails: The entire Group A is considered failed because the "final state" of that group is broken.
• Interaction with Group B: The queue will not wait for Group B to fail. Since Group B is built on top of Group A, and Group A is now invalid, Group B is effectively checking against a bad base.
• The Outcome: The queue will likely halt Group B, and then attempt to split/bisect Group A (e.g., try merging just #1 and #2) to see if a valid state can be found within that subgroup.

Answer to Question 3: Mixed bag (#1 fails, #2 passes, #3 fails)
Assuming they are in the same merge group [#1, #2, #3]:

1. The CI runs on the tip of the group (PR Feature request: support release details #3). It fails.
2. The group is marked as failed.
3. The queue attempts to bisect/retry to salvage valid PRs. It will likely attempt to run a group of [#1, #2].
4. The Result: Since #2 passes (and validates the state including #1), the new subgroup [#1, #2] turns green.
5. Merge: #1 and #2 are merged into the base branch.
6. Ejection: #3 is identified as the culprit (since it failed when attached to the valid #1+#2 state) and is ejected.

Summary
To balance flaky tests and real bugs

• Disabled (Your setting): Great for high-velocity repos with flaky tests. It assumes "if the build passes eventually, we are good." Risk: You might merge a PR that breaks a specific feature, but somehow doesn't break the build when combined with the next PR (rare, but possible).
• Enabled: Strict safety. Every single PR must be green on its own.

Hope this clarifies and help you something!