[Check Point] Process the packets field in SecureXL format by ilyannn · Pull Request #16235 · elastic/integrations

ilyannn · 2025-12-03T18:55:41Z

Proposed commit message

The specification at https://support.checkpoint.com/results/sk/sk144192 describes the packets field in the Security Gateway - SecureXL Fields section as the tuple of

Source IP address
Source Port
Destination IP address
Destination Port
Protocol Number

The current pipeline, however, parses the packets field as an integer, handling only the format described in the Security Gateway - Firewall Fields section. When this field comes as defined in the SecureXL section – a string describing the dropped packets – the current pipeline fails.

In practice the tuple can also contain an additional member denoting the network interface.

This PR adds the parsing to the pipeline. The result is an array of json objects, each structured according to the ECS, that is:

"packets_dropped" => [{
    "source": {
        "ip": ...
        "port": ...
    },
    "destination": {
        "ip": ...
        "port": ...
    },
    "network": {
        "iana_number": ...
    },
    "interface": {
        "name": ...
    }
}, ...]

We handle both the case when the interface name is present (as in practice) and when it's not (as per specification). The added group is defined as nested to preserve the relationships within each tuple.

The parsing is implemented as a Painless script; alternative approaches were tried, but were not ultimately successful.

Screenshot

From the specification:

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
~~[ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices~~

Author's Checklist

Add test cases for the new functionality
elastic-package check

How to test this PR locally

GenAI Note

Cursor with the Claude Opus 4.5 model was used in creating this PR's contents, especially the Painless script.

elastic-vault-github-plugin-prod · 2025-12-03T19:31:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

P1llus

Do we have any sample data (anonymized) we can add to the pipeline test?

P1llus · 2025-12-03T20:05:04Z

                  field: checkpoint.subs_exp
                  ignore_missing: true
+  - script:
+      tag: script_parse_checkpoint_packets_dropped


It would be preferred if we could use a common processor to parse this out, as script regex is quite heavy, was this the only way? looking at the data I guess it does seem like that, but its quite troublesome for performance reasons

I welcome any ideas 😄. I tried split + grok but couldn't make it work.

Yeah once I noticed it was meant to produce an array then any other processor mostly goes out the window with the exception of maybe foreach, but that just introduces many other issues, so script was the way to go, I was just hoping we could have managed to skip the regex part, as in theory the data is structured in some ways.

I see. Yeah a review of https://www.elastic.co/docs/reference/scripting-languages/painless/painless-ingest-processor-context has shown to me how this is possible, so I've implemented a non-regexp version.

P1llus · 2025-12-03T20:09:18Z

      description: |
        Amount of packets dropped.
+    - name: packets_dropped
+      type: nested


Just leaving it as a note rather than something that needs to be resolved, nested types are something we do try to avoid as much as possible, but at this point I am unsure about any other better alternative

Yeah it's a tradeoff. With nested types we accurately show the underlying structure, so that they can do "find packets where source is X and destination is Y", but it does come at a cost. The alternative could be to use a group but then we'll only have a list of all sources and a list of all destinations in the event and the information about which source goes to which destination would be lost.

I don't know the details, but my assumption was that there is no performance penalty as long as nobody is searching on that field.

Yeah nested fields don't induce a performance penalty but you cannot use aggregations on them, so any dashboard visualization etc wont work, the only things you can do is querying for example on the interface name of any of the objects

P1llus · 2025-12-03T20:12:15Z

+  - script:
+      tag: script_parse_checkpoint_packets_dropped
+      description: Parse packets field containing connection tuples into structured packets_dropped array.
+      if: ctx.checkpoint?.packets instanceof String && ctx.checkpoint.packets.contains('<')


I am a bit unsure but it could be startsWith is better performance wise, though if the only times this is a string is when it is the SecureXL format it might be okay.
If we had sample data we could see if there was an already parsed field that showcases that it's SecureXL type data so we wouldn't need to use either

Good point. I don't know if their tuple format is well-documented, for example if it's guaranteed that it does not start with a whitespace. The cleanest way is probably to try a conversion to number first and only run the script if it fails – I'll do that.

It actually starts with a whitespace – < – so I went with trim().startsWith('<')

botelastic · 2026-01-03T11:48:43Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

…ropped-packets

…parsing

elasticmachine · 2026-01-05T02:35:14Z

Pinging @elastic/integration-experience (Team:Integration-Experience)

elasticmachine · 2026-01-08T02:24:13Z

💚 Build Succeeded

Buildkite Build
Commit: 4bcc4e6

History

💚 Build #36089 succeeded 619eec9
💚 Build #35056 succeeded 0f68a67

cc @ilyannn

taylor-swanson

LGTM

taylor-swanson · 2026-01-08T16:14:41Z

+  - script:
+      tag: script_parse_checkpoint_packets_dropped
+      description: Parse packets field containing connection tuples into structured packets_dropped array.
+      if: ctx.checkpoint?.packets instanceof String && ctx.checkpoint.packets.trim().startsWith('<')


As a note, if statements are also painless scripts, so could move this into the source of this processor and it should work just the same, basically one script invocation instead of two. I've never benchmarked it (and not sure if it matters, I don't think conditionals are included in benchmarks), but it's probably very minimal overhead, especially once the JIT has warmed up.

Sure but this way we have a clear visual on when the script is run, I think it's moire readable this way.

elastic-vault-github-plugin-prod · 2026-01-08T17:37:35Z

Package checkpoint - 1.45.0 containing this change is available at https://epr.elastic.co/package/checkpoint/1.45.0/

…6235) The specification at https://support.checkpoint.com/results/sk/sk144192 describes the `packets` field in the _Security Gateway - SecureXL Fields_ section as the tuple of - Source IP address - Source Port - Destination IP address - Destination Port - Protocol Number The current pipeline, however, parses the `packets` field as an integer, handling only the format described in the _Security Gateway - Firewall Fields_ section. When this field comes as defined in the SecureXL section – a string describing the dropped packets – the current pipeline fails. In practice the tuple can also contain an additional member denoting the network interface. This PR adds the parsing to the pipeline. The result is an array of json objects, each structured according to the ECS/ We handle both the case when the interface name is present (as in practice) and when it's not (as per specification). The added group is defined as `nested` to preserve the relationships within each tuple. The parsing is implemented as a Painless script; alternative approaches were tried, but were not ultimately successful. Cursor with the Claude Opus 4.5 model was used in creating this PR's contents, especially the Painless script.

[checkpoint] Add the processor for SecureXL fields

7233193

ilyannn self-assigned this Dec 3, 2025

ilyannn added enhancement New feature or request Integration:checkpoint Check Point Team:Integration-Experience Security Integrations Integration Experience [elastic/integration-experience] labels Dec 3, 2025

ilyannn changed the title ~~[checkpoint] Add the processor for SecureXL fields~~ Dec 3, 2025

ilyannn changed the title ~~[checkpoint] Add the processor for packets field in SecureXL format~~ Dec 3, 2025

ilyannn changed the title ~~[checkpoint] Process the packets field in SecureXL format~~ Dec 3, 2025

[checkpoint] Update the metadata

0f68a67

P1llus reviewed Dec 3, 2025

View reviewed changes

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Dec 3, 2025

botelastic Bot added the Stalled label Jan 3, 2026

Merge remote-tracking branch 'elastic/main' into ilyannn/checkpoint-d…

a2ad80f

…ropped-packets

botelastic Bot removed the Stalled label Jan 5, 2026

ilyannn added 3 commits January 5, 2026 03:06

Add pipeline test for checkpoint packet drop logs with packets field …

6c48005

…parsing

Use trim().startsWith instead of contains for packets field check

d827624

Refactor packets parsing to use splitOnToken instead of regex

619eec9

ilyannn removed the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jan 5, 2026

ilyannn marked this pull request as ready for review January 5, 2026 02:35

ilyannn requested a review from a team as a code owner January 5, 2026 02:35

ilyannn enabled auto-merge (squash) January 5, 2026 02:39

ilyannn added 2 commits January 8, 2026 02:52

Merge branch 'main' into ilyannn/checkpoint-dropped-packets

4564cdf

Up the manifest version

4bcc4e6

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jan 8, 2026

taylor-swanson approved these changes Jan 8, 2026

View reviewed changes

ilyannn merged commit 0231876 into elastic:main Jan 8, 2026
8 checks passed

Navigation Menu

Uh oh!

[Check Point] Process the packets field in SecureXL format - #16235

[Check Point] Process the packets field in SecureXL format#16235
ilyannn merged 8 commits into
elastic:mainfrom
ilyannn:ilyannn/checkpoint-dropped-packets

ilyannn commented Dec 3, 2025 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Dec 3, 2025

P1llus left a comment

P1llus Dec 3, 2025 •

edited

Loading

ilyannn Dec 4, 2025 •

edited

Loading

P1llus Dec 4, 2025

ilyannn Jan 5, 2026

P1llus Dec 3, 2025

ilyannn Dec 4, 2025

ilyannn Dec 4, 2025

P1llus Dec 4, 2025

P1llus Dec 3, 2025

ilyannn Dec 4, 2025

ilyannn Jan 5, 2026

botelastic Bot commented Jan 3, 2026

elasticmachine commented Jan 5, 2026

elasticmachine commented Jan 8, 2026

taylor-swanson left a comment

taylor-swanson Jan 8, 2026

ilyannn Jan 8, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jan 8, 2026

Labels

5 participants

Uh oh!

Conversation

ilyannn commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Screenshot

Checklist

Author's Checklist

How to test this PR locally

GenAI Note

elastic-vault-github-plugin-prod Bot commented Dec 3, 2025

🚀 Benchmarks report

P1llus left a comment

Choose a reason for hiding this comment

P1llus Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

ilyannn Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

botelastic Bot commented Jan 3, 2026

elasticmachine commented Jan 5, 2026

elasticmachine commented Jan 8, 2026

💚 Build Succeeded

History

taylor-swanson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jan 8, 2026

Labels

5 participants

ilyannn commented Dec 3, 2025 •

edited

Loading

P1llus Dec 3, 2025 •

edited

Loading

ilyannn Dec 4, 2025 •

edited

Loading