ESQL: Push down StartsWith and EndsWith functions to Lucene#123381
ESQL: Push down StartsWith and EndsWith functions to Lucene#123381ivancea merged 10 commits intoelastic:mainfrom
Conversation
|
Hi @ivancea, I've created a changelog YAML for you. |
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
astefan
left a comment
There was a problem hiding this comment.
This is a good start in terms of testing this, but the tests are the bare minimum.
Let's make this functionality more robust by having stronger tests, please. For CSV (notice also the _index metadata which is a feature of pushable attributes):
from employees | where starts_with(first_name, "A") | keep first_name, last_name | where ends_with(last_name, "g")from employees* metadata _index | where starts_with(first_name::keyword, "A") | keep first_name, last_name, _index | where ends_with(last_name::keyword, "g") and starts_with(_index, "e")from e* metadata _index | where starts_with(first_name::keyword, "A") | keep first_name, last_name, languages.long, _index | where ends_with(last_name::keyword, "g") and starts_with(_index, "e")from e* metadata _index | where starts_with(first_name::keyword, "A") | keep first_name, last_name, languages.long, _index | where ends_with(last_name::keyword, "g") and ends_with(_index, "e")from employees metadata _index | where starts_with(first_name, "A") | keep first_name, last_name, languages.long, _index | where ends_with(_index, "s") and (ends_with(last_name, "e") or ends_with(last_name, "m")from employees metadata _index | where starts_with(first_name, "A") | keep first_name, last_name, languages.long, _index | where ends_with(_index, "s") and not (ends_with(last_name, "e") or ends_with(last_name, "m")
For unit testing, please look at PhysicalPlanOptimizerTests.testPushAndInequalitiesFilter and make sure that the EsQueryExec that gets created includes the actual wildcard query in its proper form. Please, also use a query with two statements so we can see the bool query getting created as well.
|
@astefan Just added some extra CSV tests with some combinations, with and without _index and wildcards. |
There was a problem hiding this comment.
Please, add as a javadoc the expected physicalplan tree. I know some methods in this class do not have this, but I (or we) try to have this consistent throughout the class. Thanks.
astefan
left a comment
There was a problem hiding this comment.
LGTM with a minor comment regarding the unit test.
…123381) Fixes elastic#123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in elastic#123398, which I kept separated as it changes many files
…123381) Fixes elastic#123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in elastic#123398, which I kept separated as it changes many files
…#124583) Fixes #123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in #123398, which I kept separated as it changes many files
…#124582) Fixes #123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in #123398, which I kept separated as it changes many files
…123381) Fixes elastic#123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in elastic#123398, which I kept separated as it changes many files
…123381) Fixes elastic#123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in elastic#123398, which I kept separated as it changes many files
…123381) Fixes elastic#123067 Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in elastic#123398, which I kept separated as it changes many files
|
Also backporting to 8.18 in #124871 |
Fixes #123067
Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike
This, like some other functions, needs a FoldContext. I'm using the static one for this here, but it's fixed in #123398, which I kept separated as it changes many files