ESQL: Implement `network_direction` function by MattAlp · Pull Request #136133 · elastic/elasticsearch

MattAlp · 2025-10-07T20:56:43Z

Implements the network_direction(source_ip, destination_ip, networks) function based on the processor of the same name
Addresses half of ESQL: Support community_id function #133943
This PR required addressing a small codegen (Poet) bug in BlockArgument.java in order to create a functioning evaluator.

…ction (no tests)

…tils tests

…ion tests

elasticsearchmachine · 2025-10-07T20:57:08Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-07T20:57:09Z

Pinging @elastic/kibana-esql (ES|QL-ui)

elasticsearchmachine · 2025-10-07T21:03:10Z

Hi @MattAlp, I've created a changelog YAML for you.

github-actions · 2025-10-07T21:08:18Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/network_direction.md

github-actions · 2025-10-07T21:08:19Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

ldematte · 2025-10-08T06:33:57Z

server/src/main/java/org/elasticsearch/common/network/NetworkDirectionUtils.java

+import java.util.Arrays;
+import java.util.List;
+
+public class NetworkDirectionUtils {


Despite being an utility, this class seems very specific to the esql function; it is not a general-purpose network utility. Is it possible to avoid adding it to server?

After exploring alternatives, like placing it within libs, it seems that the dependency on CIDRUtils and InetAddresses means that moving this elsewhere would require a larger refactor/org of the networking-related utility classes.

x-pack/plugin/esql/qa/testFixtures/src/main/resources/ip.csv-spec

...t/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirectionTests.java

...c/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirection.java

mouhc1ne · 2025-10-10T19:24:11Z

I don't know if there is such a thing as aliases in ESQL scalar functions, but if there was, would be nice to only have to write netdir instead of the full network_direction.

mouhc1ne · 2025-10-10T21:13:42Z

...gin/esql/compute/gen/src/main/java/org/elasticsearch/compute/gen/argument/BlockArgument.java

    @Override
    public String paramName(boolean blockStyle) {
-        return name + (blockStyle ? "Block" : "Vector");
+        return name + "Block";


Are you sure this is needed?

I ran ./gradlew :x-pack:plugin:esql:compileJava locally on this repo, with and without this change, and I don't see any generated classes change. I think it isn't needed.

I believe that's because I've since changed the signature and annotations on the process method, which no longer triggers the faulty codegen.

In order to reproduce, you should be able to check out the earlier commit 3129e57, revert this change, and see what happens when you attempt to compile.

Nik should be able to confirm when he's back - as I understand, BlockArguments can't be represented as Vectors by definition, which is why the reproducer will create valid control flow but mistakenly refer to a ${name}Vector on one line while referencing the ${name}Block earlier.

I made this change in my local copy, then ran the gradle command from earlier. The result is captured in the screenshot.

Looks like the semantics are broken since addRawVector method is called and passed a valuesBlock instead of valuesVector. Not that it matters a lot, I mean compiler is happy, and method is no-op anyways so param can be called anything. It just breaks the semantics, that's all. You're right that vectors don't mean much for the block arg class, but at least the current way both the method and its param name are in sync.

@nik9000 or @dnhatn may have an opinion on this.

That screenshot looks fine - though I had just asked @mouhc1ne about removing this method entirely because it's not called. Let me have a look at the rest of it.

OK! I understand now. @MattAlp does indeed need this change. We don't have any other functions that'd bump into this. But, it does, indeed, make a lame parameter name on those methods. But, like 20 minutes ago, I requested skipping generating these methods anyway. So I guess that's fine.

MattAlp · 2025-10-14T15:22:33Z

I don't know if there is such a thing as aliases in ESQL scalar functions, but if there was, would be nice to only have to write netdir instead of the full network_direction.

It looks like there is, https://github.com/mattalp/elasticsearch/blob/24d8f307acd7ed2e249774899a94cd0135464b16/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java#L457

...in/java/org/elasticsearch/xpack/esql/expression/function/scalar/ScalarFunctionWritables.java

...c/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirection.java

...t/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirectionTests.java

nik9000 · 2025-10-21T21:37:32Z

...gin/esql/compute/gen/src/main/java/org/elasticsearch/compute/gen/argument/BlockArgument.java

    @Override
    public String paramName(boolean blockStyle) {
-        return name + (blockStyle ? "Block" : "Vector");
+        return name + "Block";


OK! I understand now. @MattAlp does indeed need this change. We don't have any other functions that'd bump into this. But, it does, indeed, make a lame parameter name on those methods. But, like 20 minutes ago, I requested skipping generating these methods anyway. So I guess that's fine.

Replicates the network direction enrichment processor's functionality --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

…#250894) Closes elastic/streams-program#569 ## Description This PR introduces a `network_direction` processor to Streamlang which calculates the network direction given a source IP address, destination IP address, and a list of internal networks. It uses the [network_direction](https://www.elastic.co/docs/reference/enrich-processor/network-direction-processor#supported-named-network-ranges) processor for ingest pipeline transpilation and it uses the new [NETWORK_DIRECTION](elastic/elasticsearch#136133) function for ES|QL transpilation. ## Demo https://github.com/user-attachments/assets/5e9807ad-37e5-4e7b-8b00-7df8b318f2cb --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…elastic#250894) Closes elastic/streams-program#569 ## Description This PR introduces a `network_direction` processor to Streamlang which calculates the network direction given a source IP address, destination IP address, and a list of internal networks. It uses the [network_direction](https://www.elastic.co/docs/reference/enrich-processor/network-direction-processor#supported-named-network-ranges) processor for ingest pipeline transpilation and it uses the new [NETWORK_DIRECTION](elastic/elasticsearch#136133) function for ES|QL transpilation. ## Demo https://github.com/user-attachments/assets/5e9807ad-37e5-4e7b-8b00-7df8b318f2cb --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

MattAlp added 15 commits October 2, 2025 14:21

Good place to checkpoint

9637798

Fixed codegen bug, completed first pass of Network Direction ESQL fun…

3129e57

…ction (no tests)

Further refactoring back to keyword return type

13fe69d

Duplicate tests from NetworkDirectionProcessor into NetworkDirectionu…

07b6c8f

…tils tests

Change how extraction of IP from BytesRef is performed

319552e

Copied over NetworkDirectionUtils tests to the NetworkDirection funct…

1fd1e51

…ion tests

Add remaining test types

08647b2

Address the buffer re-use bug; get tests to pass

c4196a2

Remove to-dos following code review

295931a

Extend csv tests and register as an example

584de17

Docs and CSV test, singular

1502b71

Tests work great now

737eae0

Change up some of the CSV tests

b80ef49

Change up some of the CSV tests pt. 2

5e24512

Spotless gets it done

f98b9e0

MattAlp requested a review from a team as a code owner October 7, 2025 20:56

MattAlp added >enhancement :Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI labels Oct 7, 2025

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 7, 2025

elasticsearchmachine added the v9.3.0 label Oct 7, 2025

Update docs/changelog/136133.yaml

0e7ba55

Merge branch 'main' into network_direction_esql_function

e1b1e67

[CI] Auto commit changes from spotless

463cd79

ldematte reviewed Oct 8, 2025

View reviewed changes

mouhc1ne reviewed Oct 10, 2025

View reviewed changes

x-pack/plugin/esql/qa/testFixtures/src/main/resources/ip.csv-spec Show resolved Hide resolved

mouhc1ne reviewed Oct 10, 2025

View reviewed changes

...t/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirectionTests.java Outdated Show resolved Hide resolved

mouhc1ne reviewed Oct 10, 2025

View reviewed changes

...c/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirection.java Show resolved Hide resolved

mouhc1ne reviewed Oct 10, 2025

View reviewed changes

MattAlp and others added 6 commits October 14, 2025 10:01

Create test case creaion helper method

095106c

[CI] Auto commit changes from spotless

8c400af

Change caps for netdir func

925d793

Update NetworkDirection exception handling & add new CSV tests

c17a755

Correct usage of forbidden APIs

dab3921

Add netdir alias for network_direction ESQL function

24d8f30

MattAlp requested a review from mouhc1ne October 14, 2025 15:32

Merge branch 'main' into network_direction_esql_function

a766c77

MattAlp requested a review from nik9000 October 14, 2025 15:41

elasticsearchmachine and others added 2 commits October 14, 2025 15:43

[CI] Auto commit changes from spotless

6a34869

Stragglers

18e39e6

mouhc1ne reviewed Oct 21, 2025

View reviewed changes

...in/java/org/elasticsearch/xpack/esql/expression/function/scalar/ScalarFunctionWritables.java Show resolved Hide resolved

mouhc1ne reviewed Oct 21, 2025

View reviewed changes

...c/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/ip/NetworkDirection.java Show resolved Hide resolved

MattAlp and others added 2 commits October 21, 2025 13:13

Specify that function is in preview mode

ec37f64

Merge branch 'main' into network_direction_esql_function

9d125a1

nik9000 approved these changes Oct 21, 2025

View reviewed changes

Change import

756fc5a

afoucret mentioned this pull request Oct 22, 2025

Fix OperatorTestCase small input size #136982

Merged

Merge branch 'main' into network_direction_esql_function

432ab46

MattAlp merged commit fc7ae8e into elastic:main Oct 22, 2025
34 checks passed

couvq mentioned this pull request Jan 29, 2026

[Observability:Streams][Streamlang] Add a network direction processor elastic/kibana#250894

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESQL: Implement `network_direction` function#136133

ESQL: Implement `network_direction` function#136133
MattAlp merged 35 commits intoelastic:mainfrom
MattAlp:network_direction_esql_function

MattAlp commented Oct 7, 2025 •

edited

Loading

elasticsearchmachine commented Oct 7, 2025

elasticsearchmachine commented Oct 7, 2025

elasticsearchmachine commented Oct 7, 2025

github-actions bot commented Oct 7, 2025 •

edited

Loading

github-actions bot commented Oct 7, 2025

When to use applies_to tags:

What NOT to do:

ldematte Oct 8, 2025

MattAlp Oct 10, 2025

Uh oh!

Uh oh!

Uh oh!

mouhc1ne commented Oct 10, 2025

mouhc1ne Oct 10, 2025

MattAlp Oct 14, 2025

mouhc1ne Oct 21, 2025

mouhc1ne Oct 21, 2025

nik9000 Oct 21, 2025

nik9000 Oct 21, 2025

MattAlp commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

nik9000 Oct 21, 2025

Uh oh!

Labels

5 participants

Conversation

MattAlp commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticsearchmachine commented Oct 7, 2025

elasticsearchmachine commented Oct 7, 2025

elasticsearchmachine commented Oct 7, 2025

github-actions bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

github-actions bot commented Oct 7, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mouhc1ne commented Oct 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MattAlp commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Labels

5 participants

MattAlp commented Oct 7, 2025 •

edited

Loading

github-actions bot commented Oct 7, 2025 •

edited

Loading