Add checks that optimizers do not modify the layout

[julian-elastic]

Add verification that the optimizers do not modify the number of attributes and the attribute datatype.
We add special handling for Lookup Join, by checking EsQueryExec esQueryExec && esQueryExec.indexMode() == LOOKUP and another special handling for ProjectAwayColumns.ALL_FIELDS_PROJECTED

Closes #125576

[elasticsearchmachine]

Hi @julian-elastic, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Hi @julian-elastic, I've updated the changelog YAML for you.

[alex-spies]

Looks good! Mostly minor remarks.

The only non-minor remark is on tests: Can we add tests that shows that the verification triggers when optimization changes the plan's output?

One way to test this is using an inherited optimizer instance where we add a final batch which adds a wrong projection on top, which adds/removes some attributes + changes the data types. We could place such a test into the logical and physical optimizer tests, each.

[julian-elastic]

@alex-spies This is ready for final review, I addressed the code review feedback. I added the UTs to the LocalLogical and LocalPhysical plan optimizer UTs. I am not sure there is any benefit adding them to the Local and Physical plan optimizer UTs as the code is shared.

[ivancea]

LGTM! One question about the lookup condition

[ivancea]

nit: Can the superclass receive a generic with the type of plan?
It's not very important now, but lines like optimizedPlan.collectFirstChildren(EnrichExec.class::isInstance); should fail if you write Enrich.class by mistake (For example). But as it's a QueryPlan, it won't now

[alex-spies]

Looks super good! However, can we merge #129745 first, so the fix to ProjectAwayColumns is in a separate PR (which we'll backport more aggressively)?

Other than that this LGTM!

[alex-spies]

Looking good, let's !