Skip to content

ESQL: Track memory in evaluators (#133392)#133593

Merged
nik9000 merged 2 commits intoelastic:8.19from
nik9000:esql_track_evaluators_8_19
Aug 27, 2025
Merged

ESQL: Track memory in evaluators (#133392)#133593
nik9000 merged 2 commits intoelastic:8.19from
nik9000:esql_track_evaluators_8_19

Conversation

@nik9000
Copy link
Member

@nik9000 nik9000 commented Aug 26, 2025

If you write very very large ESQL queries you can spend a lot of memory on the expression evaluators themselves. You can certainly do it in real life, but our tests do something like:

FROM foo
| EVAL a0001 = n + 1
| EVAL a0002 = a0001 + 1
| EVAL a0003 = a0002 + 1
...
| EVAL a5000 = a4999 + 1
| STATS MAX(a5000)

Each evaluator costs like 200 bytes a pop. For thousands of evaluators this adds up. So! We have to track it.

Nhat had suggested charging a flat 200 bytes a pop. I thought about it and decided that it'd be pretty easy to get the actual size. Most of the evaluators are generated and it's a fairly small generated change to pick that up. So I did.

We do build the evaluators before we cost them, but that's fine because they are very very small. So long as we account for them, I think it's safe.
@nik9000
ESQL: Track memory in evaluators (elastic#133392)
e87bd19 
If you write very very large ESQL queries you can spend a lot of memory
on the expression evaluators themselves. You can certainly do it in real
life, but our tests do something like:
```
FROM foo
| EVAL a0001 = n + 1
| EVAL a0002 = a0001 + 1
| EVAL a0003 = a0002 + 1
...
| EVAL a5000 = a4999 + 1
| STATS MAX(a5000)
```

Each evaluator costs like 200 bytes a pop. For thousands of evaluators
this adds up. So! We have to track it.

Nhat had suggested charging a flat 200 bytes a pop. I thought about it
and decided that it'd be pretty easy to get the actual size. Most of the
evaluators are generated and it's a fairly small generated change to
pick that up. So I did.

We *do* build the evaluators before we cost them, but that's fine
because they are very very small. So long as we account for them, I
think it's safe.
@github-actions
Copy link
Contributor

github-actions bot commented Aug 26, 2025

Documentation preview:
@nik9000 nik9000 merged commit 74d0c72 into elastic:8.19 Aug 27, 2025
22 checks passed
sarog pushed a commit to portsbuild/elasticsearch that referenced this pull request Sep 11, 2025
@nik9000 @sarog
ESQL: Track memory in evaluators (elastic#133392) (elastic#133593)
0aaa32e 
If you write very very large ESQL queries you can spend a lot of memory
on the expression evaluators themselves. You can certainly do it in real
life, but our tests do something like:
```
FROM foo
| EVAL a0001 = n + 1
| EVAL a0002 = a0001 + 1
| EVAL a0003 = a0002 + 1
...
| EVAL a5000 = a4999 + 1
| STATS MAX(a5000)
```

Each evaluator costs like 200 bytes a pop. For thousands of evaluators
this adds up. So! We have to track it.

Nhat had suggested charging a flat 200 bytes a pop. I thought about it
and decided that it'd be pretty easy to get the actual size. Most of the
evaluators are generated and it's a fairly small generated change to
pick that up. So I did.

We *do* build the evaluators before we cost them, but that's fine
because they are very very small. So long as we account for them, I
think it's safe.
sarog pushed a commit to portsbuild/elasticsearch that referenced this pull request Sep 19, 2025
@nik9000 @sarog
ESQL: Track memory in evaluators (elastic#133392) (elastic#133593)
148ca7f 
If you write very very large ESQL queries you can spend a lot of memory
on the expression evaluators themselves. You can certainly do it in real
life, but our tests do something like:
```
FROM foo
| EVAL a0001 = n + 1
| EVAL a0002 = a0001 + 1
| EVAL a0003 = a0002 + 1
...
| EVAL a5000 = a4999 + 1
| STATS MAX(a5000)
```

Each evaluator costs like 200 bytes a pop. For thousands of evaluators
this adds up. So! We have to track it.

Nhat had suggested charging a flat 200 bytes a pop. I thought about it
and decided that it'd be pretty easy to get the actual size. Most of the
evaluators are generated and it's a fairly small generated change to
pick that up. So I did.

We *do* build the evaluators before we cost them, but that's fine
because they are very very small. So long as we account for them, I
think it's safe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment