ES|QL: add local optimizations for constant_keyword

[luigidellaquila]

Adding local planning optimizations for constant field values (eg. constant_keyword).

The rule tries to get the value at planning time and replaces it with a literal, avoiding field extraction and allowing to trigger further optimizations.

[luigidellaquila]

After discussing it with @alex-spies, we decided to simplify it a bit (to the cost of some minor optimizations) and diverge a bit from what we do in ReplaceMissingFieldWithNull.
In particular, now we avoid to reuse NameIDs, that is potentially dangerous in the long term

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

With the following test data, things do not seem to work as they do on main:

test1

{
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "f1": {
        "type": "keyword"
      }
    }
  }
}

test2

{
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "f1": {
        "type": "long"
      }
    }
  }
}

test3

{
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "f1": {
        "type": "constant_keyword"
      }
    }
  }
}

{"index":{"_index":"test1","_id":1}}
{"f1":"foo"}
{"index":{"_index":"test2","_id":1}}
{"f1":123}
{"index":{"_index":"test3","_id":1}}
{"bar":"baz"}
{"index":{"_index":"test3","_id":2}}
{"f1":"abc"}
{"index":{"_index":"test3","_id":3}}
{"f1":"foo"}

FROM test*,-test2 METADATA _index, _id | where f1 == "abc"

Results in

      bar      |  bar.keyword  |      f1       |    _index     |      _id      
---------------+---------------+---------------+---------------+---------------
null           |null           |foo            |test1          |1              
baz            |baz            |abc            |test3          |1              
null           |null           |abc            |test3          |2

[elasticsearchmachine]

Hi @luigidellaquila, I've created a changelog YAML for you.

[luigidellaquila]

My bad, a missing else.
Before moving forward I'll add more tests, especially with mixed types (eg. text and constant_keyword)

[nik9000]

I'm good with it. It'd be nice if there were an easier way to get the constant value, but this'll do. If everyone else is good, I'm good.

[nik9000]

I believe we tend to use BytesRefs encoded as utf-8.

[astefan]

My bad, a missing else. Before moving forward I'll add more tests, especially with mixed types (eg. text and constant_keyword)

No worries.
Next time, the reviewers' job would be less challenging and more swift if the PR would be ready for review with the tests you mention above. Much better for both parties if there is no need to double-check (with additional tests from reviewers) the validity of the solution. Thanks you :-).

[astefan]

Please, provide a more explanatory comment. I am looking for "reusing NameIDs is dangerous" explanation.

[astefan]

Also, having an eval doing the value "replacement" is not about performance, but about making use of other mechanisms that already exists in the optimizer to naturally "move"/"flow" the EVAL through the Nodes tree (like constant folding, moving literals on the right hand side of boolean expressions etc).

[astefan]

Thinking about this some more... this rule and ReplaceMissingFieldWithNull are, in essence, doing the same thing.

The only essential aspect that is different between them is
localLogicalOptimizerContext.searchStats().exists(f.fieldName())

and

localLogicalOptimizerContext.searchStats().constantValue(attribute.name())

Meaning, the "constant" is either null or is a value coming from a constant_keyword field. Try to see if you can abstract away this logic and either:

• have two rules that use common code and the only thing different is the searchStats() check
• have only one rule that does the null and constant check at the same time.

[luigidellaquila]

Thanks for the feedback @astefan, I unified the two rules.
TBH I was a bit dubious, mostly for two reasons: one is that all the comments in the old code seem to discourage using that pattern, and suggest possible future refactorings; the other one is that the old rule had a negative logic (see if a field needs to be retained, otherwise replace it with null), while the new one had a positive logic (see if a field is constant, and in case replace it).
Anyway, having a single rule will reduce the cost of optimization (we traverse the tree only once), and future refactorings will involve only a single class, so I'm good with this.
Please have a look if you have a chance.

[astefan]

I am ok with the idea behind the changes, but I believe the resulting code is not so nice.
For example, the transformExpressionsOnlyUp here is not clearly showing the two (distinct) use cases, ie what has to do shouldBeRetained with constants (the conditional links them together, but one has nothing to do with the other). I've tried being more explicit by splitting the conditional and adding comments, but I still don't feel good about it. The changes do show that the logic is split and things were just merged, is not a natural way to follow the code.

[luigidellaquila]

Thanks @astefan, I included your suggestions.
I agree with you that the code is not very natural and that it practically does two things at once; I like the idea of having a single rule just because it's more efficient.
IMHO the real problem is this part; the rule was already doing two things at once, ie. replacing fields with constants in EVALs and WHEREs and adding a new EVAL + PROJECT masking fields using the same NameIDs.
A way to make things simpler (and potentially safer) is to just remove the latter, but it's something we could do in the future, if needed.

[astefan]

LGTM

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 127549