ESQL: introduce a pre-mapping logical plan processing step#121260
Merged
bpintea merged 32 commits intoelastic:mainfrom Jan 31, 2025
Merged
ESQL: introduce a pre-mapping logical plan processing step#121260bpintea merged 32 commits intoelastic:mainfrom
bpintea merged 32 commits intoelastic:mainfrom
Conversation
This adds a processing step that runs rules which can modify an optimized LogicalPlan by executing async calls to the services avaialble to ESQL's transport query action. The rules need to implement a new interface (`MappingPreProcessor`). The resulting plan won't be further optimized logically, but mapped and further processed as a physical plan.
bpintea
added
>refactoring
auto-backport
costin
approved these changes
Jan 30, 2025
...ain/java/org/elasticsearch/xpack/esql/expression/function/fulltext/QueryBuilderResolver.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/util/Holder.java
Outdated
Show resolved
Hide resolved
...ain/java/org/elasticsearch/xpack/esql/expression/function/fulltext/QueryBuilderResolver.java
Outdated
Show resolved
Hide resolved
...ain/java/org/elasticsearch/xpack/esql/expression/function/fulltext/QueryBuilderResolver.java
Outdated
Show resolved
Hide resolved
| return indexNames; | ||
| } | ||
|
|
||
| private record FullTextFunctionsRewritable(LogicalPlan plan) implements Rewriteable<QueryBuilderResolver.FullTextFunctionsRewritable> { |
Member
There was a problem hiding this comment.
Nit:
While this pattern (of using record) works, it's somewhat misleading - instead use a closure to encapsulate the method directly at the calling site to reference the plan:
Rewriteable<QueryBuilderResolver.FullTextFunctionsRewritable> closure = c -> {
...
}
Contributor
Author
There was a problem hiding this comment.
A nice proposal, but I'd like to keep it separated, just to maintain the legibility of that method (which would otherwise grow quite a bit).
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/premapper/PreMapper.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
Outdated
Show resolved
Hide resolved
…lastic#121318) AggregateMetricDouble should be behind a feature flag but on release builds it was getting added when it should not have been. This commit addresses that bug.
…=reference/snapshot-restore/apis/get-snapshot-api/line_751} elastic#121345
…lastic#121325) Assertion was using reference equality on two boxed longs. So assertion could produce false positives. Change to Objects.equals to check value and avoid null check.
Sometimes there are multiple warning.
…eIT test {yaml=search.vectors/42_knn_search_int4_flat/Vector similarity with filter only} elastic#121350
* Integrate watsonx reranking to inference api * Add api_version to the watsonx api call * Fix the return_doc option * Add top_n parameter to task_settings * Add truncate_input_tokens parameter to task_settings * Add test for IbmWatonxRankedResponseEntity * Add test for IbmWatonxRankedRequestEntity * Add test for IbmWatonxRankedRequest * [CI] Auto commit changes from spotless * Add changelog * Fix transport version * Add test for IbmWatsonxService * Remove canHandleStreamingResponses * Add requireNonNull for modelId and projectId * Remove maxInputToken method * Convert all optionals to required * [CI] Auto commit changes from spotless * Set minimal_supported version to be ML_INFERENCE_IBM_WATSONX_RERANK_ADDED * Remove extraction of unused fields from IbmWatsonxRerankServiceSettings * Add space * Add space --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
bpintea
added
auto-merge-without-approval
Collaborator
💔 Backport failed
You can use sqren/backport to manually backport by running |
bpintea
added a commit
to bpintea/elasticsearch
that referenced
this pull request
Jan 31, 2025
…21260) This adds a pre-mapping logical plan processing step, occurring after the logical optimisation, but before mapping it to a physical plan. This step can perform async actions, if needed, and involves using a new `TransportActionServices` record with all available services. Furthermore, the query rewriting step part of the `FullTextFunction`s planning (occurring on the coordinator only) is refactored a bit to update the queries in-place. The verification done by `Match` and `Term` involving checking on the argument type is also now pulled back from post-optimisation to post-analysis. Their respective tests are moved accordingly as well. (cherry picked from commit 0393e56)
6 tasks
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jan 31, 2025
…121382) This adds a pre-mapping logical plan processing step, occurring after the logical optimisation, but before mapping it to a physical plan. This step can perform async actions, if needed, and involves using a new `TransportActionServices` record with all available services. Furthermore, the query rewriting step part of the `FullTextFunction`s planning (occurring on the coordinator only) is refactored a bit to update the queries in-place. The verification done by `Match` and `Term` involving checking on the argument type is also now pulled back from post-optimisation to post-analysis. Their respective tests are moved accordingly as well. (cherry picked from commit 0393e56)
This was referenced Mar 25, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a pre-mapping logical plan processing step, occurring after the logical optimisation, but before mapping it to a physical plan. This step can perform async actions, if needed, and involves using a new
TransportActionServicesrecord with all available services.Furthermore, the query rewriting step part of the
FullTextFunctions planning (occurring on the coordinator only) is refactored a bit to update the queries in-place.The verification done by
MatchandTerminvolving checking on the argument type is also now pulled back from post-optimisation to post-analysis. Their respective tests are moved accordingly as well.