Aggs: Add cancellation checks to FilterByFilter aggregator#130452
Merged
ivancea merged 8 commits intoelastic:mainfrom Jul 7, 2025
Merged
Aggs: Add cancellation checks to FilterByFilter aggregator#130452ivancea merged 8 commits intoelastic:mainfrom
ivancea merged 8 commits intoelastic:mainfrom
Conversation
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Collaborator
|
Hi @ivancea, I've created a changelog YAML for you. |
Collaborator
|
Hi @ivancea, I've updated the changelog YAML for you. |
nik9000
reviewed
Jul 2, 2025
| * As CancellableBulkScorer does a minimum of 4096 docs per batch, this number must be low to avoid long test times. | ||
| * </p> | ||
| */ | ||
| private static final long SLEEP_SCRIPT_MS = 1; |
Member
There was a problem hiding this comment.
Could you use a pair of Semaphors like so:
private static final Semaphore scriptRunPermits = new Semaphore(0);
private static final Semaphore cancelRunPermits = new Semaphore(0);
...
In the test:
cancelRunPermits.acquire();
client().cancelTheRequest();
scriptRunPermits.release(Integer.MAX_VALUE);
...
In the script:
cancelRunPermits.release(1);
scriptRunPermits.acquire();
I'm sure there's a simpler way to do it. But this'd block the cancel task until the the script starts. Then block the script until the cancel has finished.
Contributor
Author
There was a problem hiding this comment.
Done! It looks far better, faster and consistent with a Repeat(1000)
nik9000
approved these changes
Jul 7, 2025
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 7, 2025
…30452) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 7, 2025
…30452) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 7, 2025
…30452) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 7, 2025
…30452) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
This was referenced Jul 7, 2025
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 7, 2025
…130736) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 7, 2025
…130745) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 7, 2025
…130746) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 7, 2025
…130744) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 7, 2025
…130743) By default, the `FilterByFilterAggregator` (Used by the `"filter"` and `"filters"` aggs) was using the `DefaultBulkScorer` (From Lucene), which has no cancellation mechanism. This PR wraps it into a `CancellableBulkScorer`, which instead calls the inner scorer with ranges, and checks cancellation between them. This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 9, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 9, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 9, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 9, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
ivancea
added a commit
to ivancea/elasticsearch
that referenced
this pull request
Jul 9, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jul 17, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jul 17, 2025
Fixes elastic#130770 Both a bigger thread pool and not draining the semaphore permits were leading to failing tests sometimes because of blocked threads (Too many threads searching would end up draining all permits in parallel, and getting stuck). The test was added in elastic#130452
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By default, the
FilterByFilterAggregator(Used by the"filter"and"filters"aggs) was using theDefaultBulkScorer(From Lucene), which has no cancellation mechanism.This PR wraps it into a
CancellableBulkScorer, which instead calls the inner scorer with ranges, and checks cancellation between them.This should solve cases of long-running tasks using these aggregators not being cancelled, or greatly reduce the time they take after cancellation.