ES|QL - Full text functions accept null as field parameter by carlosdelest · Pull Request #137430 · elastic/elasticsearch

carlosdelest · 2025-10-31T10:03:08Z

Full text functions that take a field as a parameter should allow null as a field parameter. This is necessary as the field may not be present on an index mapping, and the local optimizer replaces it by null.

We allow null as a field in FullTextFunctions, meaning that they are nullable and will be replaced by NULL.

This PR also refactors full text functions that depend on a single field (MATCH, MATCH_PHRASE, KNN) to use a common superclass (SingleFieldFullTextFunction) that contains the common field logic.

… and add the FoldNull rule to LocalLogicalPlanOptimizer

…down to an index that doesn't contain the field

elasticsearchmachine · 2025-10-31T10:03:34Z

Hi @carlosdelest, I've created a changelog YAML for you.

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

…-functions-accept-null # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizerTests.java

carlosdelest · 2025-10-31T10:04:53Z

...ugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/Match.java

@@ -300,39 +286,7 @@ public final void writeTo(StreamOutput out) throws IOException {

    @Override
    protected TypeResolution resolveParams() {
-        return resolveField().and(resolveQuery())


Refactoring - all of these are moved up to SingleFieldFullTextFunction

carlosdelest · 2025-10-31T10:05:57Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    }
+
+    @Override
+    public boolean foldable() {


This is what changed on this PR besides all the refactoring. When the FTF field is null, the FTF is foldable to NULL, and nullable.

carlosdelest · 2025-10-31T11:32:32Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

                new InferNonNullAggConstraint(),
-                new ReplaceDateTruncBucketWithRoundTo()
+                new ReplaceDateTruncBucketWithRoundTo(),
+                new FoldNull()


@alex-spies I added foldNull to local logical optimization, so full text functions can be folded to null when null has been replaced. Does it look OK to you?

I think this should be beneficial, yes. Not sure why we didn't have it earlier.
Do you think you could please add one/some LocalLogicalPlanOptimizerTests with maybe some "easy" other function(s) (string or math?) to check that they locally now get folded too. This should add a tiny bit of performance to other functions too.

the FoldNull rule is in the localOperators batch, that is applied after the localRewrites batch. Is it too late to wait for it to be executed in localOperators? I wonder why we need a duplicate of it here?

Oh, I see - I didn't understand that this rule was already used as part of localOperators(). Removing it, thanks @fang-xing-esql !

carlosdelest · 2025-10-31T14:34:59Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

+            """);
+
+        // Limit[1000[INTEGER],false,false]
+        Limit limit = as(plan, Limit.class);


@fang-xing-esql , now FTFs can be pushed down - when a field does not exist in the index, it will become NULL instead.

Does this make sense from the union all perspective?

I found a potential issue here and I'll provide a reproducer.

The query in the test failed with VerificationException because the full text function does not apply to one of the subquery(branch), the field referenced in the full text function is not an index field, it doesn't throw VerificationException in this PR, however I run into this error when the query is executed. Shall we still throw VerificationException for this kind of situations? Or prune the entire branch when the field does not exist in that branch, a related comment is here.

+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d { "query": "from sample_data_1, (from books) metadata _score | WHERE author:\"Bradbury\" | sort _score desc, author" } ' { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Unrecognized class type class org.elasticsearch.xpack.esql.core.expression.Literal" } ], "type" : "illegal_argument_exception", "reason" : "Unrecognized class type class org.elasticsearch.xpack.esql.core.expression.Literal" }, "status" : 400 }

The mapping and data a like below.

curl -u elastic:password -X POST "localhost:9200/books/_doc?pretty" -H 'Content-Type: application/json' -d' {"name": "Snow Crash", "author": "Neal Stephenson", "release_date": "1992-06-01", "page_count": 470, "*name": "*Snow Crash"} ' curl -u elastic:password -X POST "localhost:9200/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d' { "index" : { "_index" : "books" } } {"name": "Revelation Space", "author": "Alastair Reynolds", "release_date": "2000-03-15", "page_count": 585, "*name": "*Revelation Space"} { "index" : { "_index" : "books" } } {"name": "1984", "author": "George Orwell", "release_date": "1985-06-01", "page_count": 328, "*name": "*1984"} { "index" : { "_index" : "books" } } {"name": "Fahrenheit 451", "author": "Ray Bradbury", "release_date": "1953-10-15", "page_count": 227, "*name": "*Fahrenheit 451"} { "index" : { "_index" : "books" } } {"name": "Brave New World", "author": "Aldous Huxley", "release_date": "1932-06-01", "page_count": 268, "*name": "*Brave New World"} { "index" : { "_index" : "books" } } {"name": "The Handmaids Tale", "author": "Margaret Atwood", "release_date": "1985-06-01", "page_count": 311, "*name": "*The Handmaids Tale"} ' curl -u elastic:password -X PUT "localhost:9200/sample_data_1?pretty" -H 'Content-Type: application/json' -d' { "settings" : {"mode" : "lookup"}, "mappings": { "properties": { "client.ip": {"type": "ip"}, "message": {"type": "text"}, "test_date": {"type": "date"}, "event.duration": {"type": "long"}, "mv": {"type": "long"} } } } ' curl -X PUT "localhost:9200/sample_data_1/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d' {"index": {}} {"@timestamp": "2023-10-23T12:15:03.360Z", "client.ip": "172.21.2.162", "message": "Connected to 10.1.0.3", "event.duration": 3450233, "test_date": "1985-11-21T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T12:27:28.948Z", "client.ip": "172.21.2.113", "message": "Connected to 10.1.0.2", "event.duration": 2764889, "test_date": "1985-11-22T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:33:34.937Z", "client.ip": "172.21.0.5", "message": "Disconnected", "event.duration": 1232382, "test_date": "1985-11-23T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:51:54.732Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 725448, "test_date": "1985-11-24T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:52:55.015Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 8268153, "test_date": "1985-11-25T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:53:55.832Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 5033755, "test_date": "1985-11-26T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:55:01.543Z", "client.ip": "172.21.3.15", "message": "Connected to 10.1.0.1", "event.duration": 1756467, "test_date": "1985-11-27T00:00:00Z", "mv": [1.0, 2.0]} '

Thanks for spotting this! This was indeed a problem with returning Literal.NULL instead of null when folding.

I fixed that in bcf5d4f, and added CSV tests in a9dec3d to double check.

elasticsearchmachine · 2025-10-31T14:35:33Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

Co-authored-by: Bogdan Pintea <sig11@mailbox.org>

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

fang-xing-esql

Thank you @carlosdelest, and for taking into account subqueries as well! Agreed with @bogdan, it will be great to add some additional tests(verify the logical plan and some CsvTests to validate query results) that have these full text functions with null as input field. If the full text functions are used in filters, the PruneFilters rule might be able to replace its children plans with a LocalRelation, that is a sign that the filter on full text function + null fields is applied successfully.

fang-xing-esql · 2025-11-04T19:55:35Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    @Override
+    public Object fold(FoldContext ctx) {
+        // We only fold when the field is null (it's not present in the mapping), so we return null
+        return Literal.NULL;


I wonder if we should just return a null here, instead of wrapping a Literal around it? A lot of implementations of fold return a plain null.

Yes, this is what caused the error you mentioned 🤦

carlosdelest · 2025-11-05T09:28:27Z

I think some simple CSV-spec tests would be good to have too, just to check that they work with literals, centrally too. (MATCH(null, ..), ..| EVAL x = null | WHERE MATCH(x, ...)).

@bpintea makes sense, added CSV tests in a9dec3d and more optimization tests for null in 0c4293c

fang-xing-esql

Thank you @carlosdelest, LGTM! A couple of suggestions:

I came across a similar issue that a field cannot be found on the data node, although it exists in the index mapping once before. It was because there are more than 10 indices on the cluster, and our batch execution model split them into multiple batches, by default 10 indices per batch. When the batch that does not have that index/field is executed, ReplaceFieldWithConstantOrNull converts the field to null. I didn't spend much time to investigate how the original/reported issue related to this PR happened, however this is likely one of the causes, and it will be great that if we have a test that covers it, an example can be found here - the test creates more than 10 indices to trigger the batch executions, and it exercise the code path from a different perspective, it will be nice to have this coverage.
subquery is under snapshot, adding the test-release label help us ensure the change(relevant to subquery) is not exposed to a release build.

fang-xing-esql · 2025-11-05T13:53:40Z

...sql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizerTests.java

 import static org.hamcrest.Matchers.startsWith;

-//@TestLogging(value = "org.elasticsearch.xpack.esql:TRACE", reason = "debug")
+@TestLogging(value = "org.elasticsearch.xpack.esql:TRACE", reason = "debug")


Don't forget to comment out the debug log before merging.

… field

carlosdelest · 2025-11-06T09:10:14Z

the test creates more than 10 indices to trigger the batch executions, and it exercise the code path from a different perspective, it will be nice to have this coverage.

@fang-xing-esql ah interesting - did a test to exercise that in 7a6538d

carlosdelest · 2025-11-06T13:21:13Z

@fang-xing-esql I added the test-release tag - failures are due to HistogramPercentileTests, which is unrelated to this change.

fang-xing-esql

Thanks for adding the test, @carlosdelest. That failed release test seems to be related to a BWC issue. As long as the release tests for full-text functions and subqueries are passing, we’re good. This also reminds me to double-check the BWC of the new test(Rest).

fang-xing-esql · 2025-11-06T14:21:51Z

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

        assertThat(answer.get("values"), equalTo(List.of(List.of("#"), List.of("foo#bar"))));
    }

+    public void testMatchFunctionAcrossMultipleIndicesWithMissingField() throws IOException {


Do we need a capability here? Will this test fail on older versions? It also reminds me whether this fix need to be backported to some prior versions.

IIUC, RestEsqlSpecIT is not a bwc test - even for multi_node, it does not use previous ES versions. Is that correct?

Re backporting - this includes some refactoring for FTFs that I'd like to check to understand the effort needed before commiting.

carlosdelest · 2025-11-06T16:14:48Z

Thanks for adding the test, @carlosdelest. That failed release test seems to be related to a BWC issue. As long as the release tests for full-text functions and subqueries are passing, we’re good. This also reminds me to double-check the BWC of the new test(Rest).

@fang-xing-esql I believe this is not due to bwc for the feature. All tests are failing for the HistogramPercentileTests due to:

java.lang.IllegalArgumentException: remote node at version [9213000] doesn't understand data type [EXPONENTIAL_HISTOGRAM]

That is not related to this change, but to a new data type being added to snapshots that is failing for release tests.

fang-xing-esql

Thanks for double checking!

BASE=34e3417158280c446b132b753cf6894105f8ade7 HEAD=d42c91bbb6c4cc0ecdba0d2fa9fb08109546018a Branch=main

…37430)

carlosdelest · 2025-11-11T15:59:36Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2
✅	9.1

Questions ?

Please refer to the Backport tool documentation

elasticsearchmachine added 10 commits October 29, 2025 18:43

Remove null check for fields in FTFs

1364419

First version - allow nulls as params, implement nullable() in Match,…

bd00de1

… and add the FoldNull rule to LocalLogicalPlanOptimizer

Introduce base class for Match and MatchPhrase

20b202d

Make Knn a subclass of SingleFieldFullTextFunction, refactor

c7bda0f

Refactor

7f5f5c6

Fix PushDownFilterAndLimitIntoUnionAllTests, now match can be pushed …

8fa9556

…down to an index that doesn't contain the field

Spotless

bd6e99d

Refactor

8ce4820

Fix error tests

1d98c74

Add check for single unique signature

ec81181

carlosdelest added >bug :Analytics/ES|QL AKA ESQL :Search Relevance/ES|QL Search functionality in ES|QL v9.3.0 labels Oct 31, 2025

Update docs/changelog/137430.yaml

336b33c

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Oct 31, 2025

elasticsearchmachine added 5 commits October 31, 2025 11:51

Fix tests and add null cases

959cac7

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

21daf3d

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Add test cases

b737b03

Fix tests

f68b569

carlosdelest commented Oct 31, 2025

View reviewed changes

carlosdelest marked this pull request as ready for review October 31, 2025 14:35

elasticsearchmachine added the Team:Search - Relevance The Search organization Search Relevance team label Oct 31, 2025

elasticsearchmachine and others added 8 commits November 4, 2025 16:58

Remove FoldNull as it is already applied in localOperators()

2436a22

Apply suggestion from @bpintea

012378b

Co-authored-by: Bogdan Pintea <sig11@mailbox.org>

Apply suggestion from @bpintea

30a8319

Co-authored-by: Bogdan Pintea <sig11@mailbox.org>

Apply suggestion from @bpintea

569b084

Co-authored-by: Bogdan Pintea <sig11@mailbox.org>

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

04b7dd9

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Feedback from @bpintea

ad811e5

Add null test cases for Knn and MatchPhrase

845cd35

Merge branch 'main' into bugfix/esql-full-text-functions-accept-null

48fa7c1

fang-xing-esql reviewed Nov 4, 2025

View reviewed changes

elasticsearchmachine added 3 commits November 5, 2025 09:46

Fix push down with unexisting field

bcf5d4f

Add CSV tests

a9dec3d

Add more field nullability tests for logical optimizations

0c4293c

carlosdelest requested a review from fang-xing-esql November 5, 2025 09:30

fang-xing-esql reviewed Nov 5, 2025

View reviewed changes

elasticsearchmachine added 2 commits November 6, 2025 10:03

Add test for multiple index batches that contain / do not contain the…

7a6538d

… field

Add test for multiple index batches that contain / do not contain the…

a044519

… field

carlosdelest added the test-release Trigger CI checks against release build label Nov 6, 2025

Merge branch 'main' into bugfix/esql-full-text-functions-accept-null

d42c91b

fang-xing-esql reviewed Nov 6, 2025

View reviewed changes

fang-xing-esql approved these changes Nov 6, 2025

View reviewed changes

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Nov 6, 2025

Mirror upstream elastic#137430 as single snapshot commit for AI review

4d8b095

BASE=34e3417158280c446b132b753cf6894105f8ade7 HEAD=d42c91bbb6c4cc0ecdba0d2fa9fb08109546018a Branch=main

carlosdelest merged commit f969d7d into elastic:main Nov 7, 2025
33 of 36 checks passed

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Nov 7, 2025

Mirror upstream elastic#137430 as single snapshot commit for AI review

af647d3

BASE=34e3417158280c446b132b753cf6894105f8ade7 HEAD=d42c91bbb6c4cc0ecdba0d2fa9fb08109546018a Branch=main

Kubik42 pushed a commit to Kubik42/elasticsearch that referenced this pull request Nov 10, 2025

ES|QL - Full text functions accept null as field parameter (elastic#1…

a7a56ff

…37430)

carlosdelest mentioned this pull request Nov 11, 2025

[9.2] ES|QL - Full text functions accept null as field parameter (#137430) #137914

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES|QL - Full text functions accept null as field parameter#137430

ES|QL - Full text functions accept null as field parameter#137430
carlosdelest merged 30 commits intoelastic:mainfrom
carlosdelest:bugfix/esql-full-text-functions-accept-null

carlosdelest commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

carlosdelest Oct 31, 2025

carlosdelest Oct 31, 2025

carlosdelest Oct 31, 2025

bpintea Nov 4, 2025 •

edited

Loading

fang-xing-esql Nov 4, 2025

carlosdelest Nov 4, 2025

carlosdelest Oct 31, 2025

fang-xing-esql Nov 4, 2025

fang-xing-esql Nov 4, 2025 •

edited

Loading

carlosdelest Nov 5, 2025

elasticsearchmachine commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

fang-xing-esql left a comment

fang-xing-esql Nov 4, 2025

carlosdelest Nov 5, 2025

carlosdelest commented Nov 5, 2025 •

edited

Loading

fang-xing-esql left a comment

fang-xing-esql Nov 5, 2025

carlosdelest commented Nov 6, 2025

carlosdelest commented Nov 6, 2025

fang-xing-esql left a comment

fang-xing-esql Nov 6, 2025

carlosdelest Nov 6, 2025

carlosdelest commented Nov 6, 2025

fang-xing-esql left a comment

Uh oh!

carlosdelest commented Nov 11, 2025 •

edited

Loading

Labels

5 participants

Conversation

carlosdelest commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpintea Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fang-xing-esql Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

elasticsearchmachine commented Oct 31, 2025

fang-xing-esql left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Nov 6, 2025

carlosdelest commented Nov 6, 2025

fang-xing-esql left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Nov 6, 2025

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 All backports created successfully

Questions ?

Labels

5 participants

bpintea Nov 4, 2025 •

edited

Loading

fang-xing-esql Nov 4, 2025 •

edited

Loading

carlosdelest commented Nov 5, 2025 •

edited

Loading

carlosdelest commented Nov 11, 2025 •

edited

Loading