ES|QL: Add TRANGE ES|QL function by leontyevdv · Pull Request #136441 · elastic/elasticsearch

leontyevdv · 2025-10-11T00:02:58Z

Add a new ES|QL function that filters @timestamp values for the given time range. It transforms into a fileter and is pushed down to Lucene.

What is currently supported

Explicit start and end times:

TRANGE("2024-05-12T12:00:00", "2024-05-12T15:30:00") - [explicit start; explicit end]
TRANGE(1715504400000, 1715517000000) - [explicit start in millis; explicit end in millis]

Offset from now:

TRANGE(1h) - [now - 1h; now] - supports time_duration (1h, 1min, etc) and period (1day, 1month, etc.)

What is not supported and requires consideration

Explicit start time:

TRANGE("2024-05-12T12:00:00") - [explicit start; now]
TRANGE(1715504400000) - [explicit start in millis; now]

Modifier for an explicit start time (start time with offset):

TRANGE("2024-05-12T12:00:00", 1h) - [explicit start; start + 1h] - supports time_duration (1h, 1min, etc) and period (1day, 1month, etc.)
TRANGE("2024-05-12T12:00:00", -1h) - [explicit start; start - 1h] - supports time_duration (1h, 1min, etc) and period (1day, 1month, etc.)
TRANGE(1715504400000, 1h) - [explicit start in millis; start + 1h] - supports time_duration (1h, 1min, etc) and period (1day, 1month, etc.)
TRANGE(1715504400000, -1h) - [explicit start in millis; start - 1h] - supports time_duration (1h, 1min, etc) and period (1day, 1month, etc.)

TRANGE(1w + 1h, 1w)

Closes #135599

@timestamp

Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

elasticsearchmachine · 2025-10-11T00:03:23Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-11T00:03:24Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2025-10-11T00:03:46Z

Hi @leontyevdv, I've created a changelog YAML for you.

github-actions · 2025-10-11T00:05:37Z

🔍 Preview links for changed docs

github-actions · 2025-10-11T00:05:38Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

@timestamp

Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

dnhatn · 2025-10-12T16:15:43Z

Thank you, Dima! Is there a reason we don't translate TRANGE to GreaterThanEquals() or BinaryLogicOperation(GreaterThanEquals, LessThanOrEquals)?

leontyevdv · 2025-10-14T10:16:22Z

Thank you, Dima! Is there a reason we don't translate TRANGE to GreaterThanEquals() or BinaryLogicOperation(GreaterThanEquals, LessThanOrEquals)?

Hi @dnhatn ! Thank you for the review! Initially, I implemented TRANGE as a surrogate and struggled with the tests for quite a while. The issue was with the assertion ("Duplicate name ids are not allowed in layouts"): see Layout. Then, inspired by the StartsWith function, I rewrote it using TranslationAware.SingleValueTranslationAware directly.

I have the surrogate version in my stash. We can take a look at it together. Wdyt?

Fix tests. Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries-rate.csv-spec # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Replace asQuery by a surrogate Closes elastic#135599

Fix tests Closes elastic#135599

Fix constructor visibility Closes elastic#135599

Fix tests Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

dnhatn · 2025-10-17T16:20:08Z

I'll postpone the TRANGE(1w + 1h, 1w) one. The function will guarantee the order and types of the arguments and will be failing with the wrong input.

Yes, this case needs more thought!

Remove wrong csv test Closes elastic#135599

bpintea

Looking good.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/Configuration.java

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries-rate.csv-spec

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/FieldNameUtils.java

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/Configuration.java

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

...in/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java

dnhatn

I left two questions, but this looks good. Thanks Dima!

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

Address PR review comments Closes elastic#135599

dnhatn

Great work, Dima! Thank you for all the iterations. Could you please wait for Bogdan’s approval before merging?

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

Address PR review comments Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Address PR review comments Closes elastic#135599

Add Verifier tests Closes elastic#135599

bpintea

Left a set of notes, but LGTM otherwise.

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

bpintea · 2025-10-21T16:49:32Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

+        Literal timestampLiteral = as(Alias.unwrap(eval.fields().getFirst()), Literal.class);
+        long expectedTimestamp = DateUtils.asDateTimeWithNanos(timestampValue, DateUtils.UTC).toInstant().toEpochMilli();
+        assertThat(timestampLiteral.fold(FoldContext.small()), equalTo(expectedTimestamp));
+    }


We should continue with the inspection of the plan here: what happens here is that since timestampValue is within the intervals passed to TRANGE, the WHERE should fold away (it's always true). The test should confirm that. We usually inspect the plan all the way down to the source (just like in the null folding case, though there's compacter, cause it's folded down to just one node).

Optionally, you could also add a test that folds to false.

BTW, you could add a similar test as a CSV spec, to also check the results. But optional.

Thanks Bogdan! I added inspection all the way down to the source, also added a test which tests folding to false.

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

bpintea · 2025-10-21T17:17:20Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

+        }
+
+        if (rangeStart.isAfter(rangeEnd)) {
+            throw new InvalidArgumentException("TRANGE rangeStart time [{}] must be before rangeEnd time [{}]", rangeStart, rangeEnd);


Can we add a test for this condition?
Ideally for the one above too.

Done! Added to TRangeErrorTests

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

Address PR review's comments Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

Add more error tests Closes elastic#135599

bpintea

Added some notes on instantiating the function. Otherwise all good.

bpintea · 2025-10-22T12:38:59Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java

+        this(source, new UnresolvedAttribute(source, MetadataAttribute.TIMESTAMP_FIELD), first, second, configuration);
+    }
+
+    public TRange(Source source, Expression timestamp, Expression first, Expression second, Configuration configuration) {


This should be protected, as then the public one can be called unanbigously directly from the function registry.

Changed this and removed the bic() method from the function registry.

bpintea · 2025-10-22T12:40:51Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

                def(MonthName.class, MonthName::new, "month_name"),
-                def(Now.class, Now::new, "now") },
+                def(Now.class, Now::new, "now"),
+                def(TRange.class, bic(TRange::new), "trange") },


Having one public c'tor in TRange will allow this direct call, since we have already a BinaryConfigurationAwareBuilder. The bic method can then be removed.

Suggested change

def(TRange.class, bic(TRange::new), "trange") },

def(TRange.class, TRange::new, "trange") },

Make constructor protected Closes elastic#135599

Make constructor public Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

Add docs Closes elastic#135599

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

bpintea · 2025-10-23T10:49:58Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

                def(MonthName.class, MonthName::new, "month_name"),
-                def(Now.class, Now::new, "now") },
+                def(Now.class, Now::new, "now"),
+                def(TRange.class, bic(TRange::new), "trange") },


@leontyevdv, hopefully this will make the call unambiguous and also have EsqlNodeSubclassTests tests pass.

Suggested change

def(TRange.class, bic(TRange::new), "trange") },

def(TRange.class, (BinaryConfigurationAwareBuilder<TRange>)TRange::new, "trange") },

@timestamp

* ES|QL: Add TRANGE ES|QL function Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

b89f498

Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

leontyevdv requested a review from dnhatn October 11, 2025 00:02

leontyevdv self-assigned this Oct 11, 2025

leontyevdv added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :StorageEngine/TSDB You know, for Metrics :Analytics/ES|QL AKA ESQL Team:StorageEngine :StorageEngine/ES|QL Timeseries / metrics / PromQL / logsdb capabilities in ES|QL labels Oct 11, 2025

elasticsearchmachine added the v9.3.0 label Oct 11, 2025

Update docs/changelog/136441.yaml

6d8cc81

leontyevdv requested a review from a team October 11, 2025 00:10

ES|QL: Add TRANGE ES|QL function

063f72b

Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

leontyevdv added 9 commits October 14, 2025 15:13

ES|QL: Add TRANGE ES|QL function

e42b1ca

Fix tests. Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

f4c694f

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries-rate.csv-spec # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

ES|QL: Add TRANGE ES|QL function

5642817

Replace asQuery by a surrogate Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

736194e

Replace asQuery by a surrogate Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

ea9e431

Fix tests Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

e572f55

Fix constructor visibility Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

ec90dc5

ES|QL: Add TRANGE ES|QL function

9b0aa07

Fix tests Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

af3c2d1

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

leontyevdv requested a review from bpintea October 15, 2025 15:51

ES|QL: Add TRANGE ES|QL function

051d0d0

Remove wrong csv test Closes elastic#135599

bpintea reviewed Oct 17, 2025

View reviewed changes

dnhatn reviewed Oct 17, 2025

View reviewed changes

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java Outdated Show resolved Hide resolved

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java Outdated Show resolved Hide resolved

leontyevdv added 2 commits October 20, 2025 13:30

ES|QL: Add TRANGE ES|QL function

74cf80b

Address PR review comments Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

e2f74bb

Address PR review comments Closes elastic#135599

dnhatn approved these changes Oct 20, 2025

View reviewed changes

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/TRange.java Outdated Show resolved Hide resolved

leontyevdv added 3 commits October 21, 2025 13:53

ES|QL: Add TRANGE ES|QL function

19c5d70

Address PR review comments Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

3e1848e

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

ES|QL: Add TRANGE ES|QL function

98518c4

Address PR review comments Closes elastic#135599

leontyevdv requested a review from bpintea October 21, 2025 13:07

ES|QL: Add TRANGE ES|QL function

5566598

Add Verifier tests Closes elastic#135599

bpintea approved these changes Oct 21, 2025

View reviewed changes

leontyevdv added 3 commits October 22, 2025 13:23

ES|QL: Add TRANGE ES|QL function

d94afe2

Address PR review's comments Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

b3b9d16

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

ES|QL: Add TRANGE ES|QL function

44e2d9f

Add more error tests Closes elastic#135599

bpintea approved these changes Oct 22, 2025

View reviewed changes

leontyevdv and others added 8 commits October 22, 2025 14:51

ES|QL: Add TRANGE ES|QL function

2d606d6

Make constructor protected Closes elastic#135599

ES|QL: Add TRANGE ES|QL function

58948b6

Make constructor public Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

92a9591

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

[CI] Auto commit changes from spotless

bf71cd1

ES|QL: Add TRANGE ES|QL function

16612b0

Add docs Closes elastic#135599

Merge branch 'main' into feature/esql-trange-function

ddc4a54

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Merge branch 'main' into feature/esql-trange-function

0917f77

Merge branch 'main' into feature/esql-trange-function

bb326e1

bpintea reviewed Oct 23, 2025

View reviewed changes

leontyevdv added 2 commits October 23, 2025 12:56

Merge branch 'main' into feature/esql-trange-function

c2a76e8

Merge branch 'main' into feature/esql-trange-function

60670fa

leontyevdv merged commit d0555e5 into elastic:main Oct 23, 2025
34 checks passed

fzowl pushed a commit to voyage-ai/elasticsearch that referenced this pull request Nov 3, 2025

ES|QL: Add TRANGE ES|QL function (elastic#136441)

04919a7

* ES|QL: Add TRANGE ES|QL function Add a new ES|QL function that filters @timestamp values for the given time range. Closes elastic#135599

	def(TRange.class, bic(TRange::new), "trange") },
	def(TRange.class, TRange::new, "trange") },

	def(TRange.class, bic(TRange::new), "trange") },
	def(TRange.class, (BinaryConfigurationAwareBuilder<TRange>)TRange::new, "trange") },

Conversation

leontyevdv commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is currently supported

What is not supported and requires consideration

elasticsearchmachine commented Oct 11, 2025

elasticsearchmachine commented Oct 11, 2025

elasticsearchmachine commented Oct 11, 2025

github-actions bot commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

github-actions bot commented Oct 11, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

dnhatn commented Oct 12, 2025

leontyevdv commented Oct 14, 2025

dnhatn commented Oct 17, 2025

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bpintea Oct 21, 2025

Choose a reason for hiding this comment

leontyevdv Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bpintea Oct 21, 2025

Choose a reason for hiding this comment

leontyevdv Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

bpintea Oct 22, 2025

Choose a reason for hiding this comment

leontyevdv Oct 22, 2025

Choose a reason for hiding this comment

bpintea Oct 22, 2025

Choose a reason for hiding this comment

bpintea Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Labels

5 participants

leontyevdv commented Oct 11, 2025 •

edited

Loading

github-actions bot commented Oct 11, 2025 •

edited

Loading

leontyevdv Oct 22, 2025 •

edited

Loading