Skip to content

ES|QL support (#194)#199

Merged
mashhurs merged 1 commit intologstash-plugins:3.xfrom
mashhurs:esql-support-3.x
Jul 21, 2025
Merged

ES|QL support (#194)#199
mashhurs merged 1 commit intologstash-plugins:3.xfrom
mashhurs:esql-support-3.x

Conversation

@mashhurs
Copy link
Contributor

@mashhurs mashhurs commented Jul 18, 2025

Cherry pick of 5e3c464

  • ESQL and DSL executors are introduced. param can accept ES|QL query shape now. is introduced for initial step but needs team's feedback. DSL logics moved into DSL executors.

  • Apply suggestions from code review

Separate DSL and ESQL interface in the client.

  • Rebase against upstream main after target support added. Separate unit test for DSL. Address comments: do not save ES version in client, add apply target method in executors, set to target if target is defined, docs update.

  • Introduce query_type option which accepts dsl or esql to define a query shape. Remove multi-depth nested named_params and keep only top-level query_params which aligns with placeholder structure in the ES|QL.

  • Separate event referenced and static valued fields at initialization of the ESQL executor.

  • query_params now supports both Array and Hash types.

  • Add tech preview section under ESQL.

  • Place the query results based on the target specified. If not specified, first result will be set to event's top level.

  • Apply suggestions from code review

Doc corrections.

  • ES|QL result mapping to event doc correction.

  • Integration tests to run with credentials enabled and SSL configs.


(cherry picked from commit 5e3c464)

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

@mashhurs
Copy link
Contributor Author

I have bit-by-bit compared this change (especially lib/logstash/filters/elasticsearch.rb) to #194 and seems identical!

Failed CI step (INTEGRATION=true SNAPSHOT=true LOG_LEVEL=info ELASTIC_STACK_VERSION=8.future) is as expected:

#3 [elasticsearch internal] load metadata for docker.elastic.co/elasticsearch/elasticsearch:8.future
#3 ERROR: docker.elastic.co/elasticsearch/elasticsearch:8.future: not found
#4 [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future
#4 ERROR: docker.elastic.co/logstash/logstash:8.future: not found
------
 > [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future:
@mashhurs mashhurs requested a review from jsvd July 18, 2025 17:56
* ESQL and DSL executors are introduced.  param can accept ES|QL query shape now.  is introduced for initial step but needs team's feedback. DSL logics moved into DSL executors.

* Apply suggestions from code review

Separate DSL and ESQL interface in the client.

Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>

* Rebase against upstream main after target support added. Separate unit test for DSL. Address comments: do not save ES version in client, add apply target method in executors, set to target if target is defined, docs update.

Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>

* Introduce query_type option which accepts dsl  or esql to define a query shape. Remove multi-depth nested named_params and keep only top-level query_params which aligns with placeholder structure in the ES|QL.

* Separate event referenced and static valued fields at initialization of the ESQL executor.

* query_params now supports both Array and Hash types.

* Add tech preview section under ESQL.

* Place the query results based on the target specified. If not specified, first result will be set to event's top level.

* Apply suggestions from code review

Doc corrections.

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>

* ES|QL result mapping to event doc correction.

* Integration tests to run with credentials enabled and SSL configs.

---------

Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit 5e3c464)
@mashhurs
Copy link
Contributor Author

I have bit-by-bit compared this change (especially lib/logstash/filters/elasticsearch.rb) to #194 and seems identical!

Failed CI step (INTEGRATION=true SNAPSHOT=true LOG_LEVEL=info ELASTIC_STACK_VERSION=8.future) is as expected:

#3 [elasticsearch internal] load metadata for docker.elastic.co/elasticsearch/elasticsearch:8.future
#3 ERROR: docker.elastic.co/elasticsearch/elasticsearch:8.future: not found
#4 [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future
#4 ERROR: docker.elastic.co/logstash/logstash:8.future: not found
------
 > [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future:

Solved by #200

@mashhurs mashhurs merged commit 3964b91 into logstash-plugins:3.x Jul 21, 2025
3 checks passed
@mashhurs mashhurs deleted the esql-support-3.x branch July 21, 2025 16:15
@mashhurs
Copy link
Contributor Author

Published with v3.19.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants