[checkpoint] Change fingerprint creation

[ash-darin]

Enhancement

Proposed commit message

This changes the fields used to create a unique fingerprint for the event. The currently selected fields are lacking in achieving this as rejects have been observed. Checkpoint apparently often generate events that are "updates" to previous events. These events have the timestamp of the original event but incorporate a "lastupdatetime" timestamp and an update_count. Fingerprint creation therefore creates duplicates.

• @timestamp is included, but @timestamp is renamed before use and is therefore always nil. -> removed
• segment_time is not unique, replaced with lastupdatetime
• add update_count

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.

How to test this PR locally

Check your logs/indices if any messages are rejected. Apply fix and check whether numbers go down..

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[elasticmachine]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[ilyannn]

Hey @efd6, @gogochan — this PR changes the checkpoint fingerprint fields used for document _id calculation. It swaps out @timestamp and checkpoint.segment_time for checkpoint.lastupdatetime and checkpoint.update_count, keeping checkpoint.loguid, checkpoint.time, and checkpoint.sequencenum.

@efd6 — you built the original fingerprint logic in #6483 to handle loguid collisions. Would you mind weighing in on whether dropping @timestamp and segment_time could reintroduce that issue?

@gogochan — you added sequencenum to the fingerprint in #12439. Does this change look right to you?

🤖 Generated by Claude Code/Opus 4.6

[efd6]

@ilyannn The change will not reintroduce the issue I fixed. However segment_time should not be removed from the set of fields. It should be

fields:
  - checkpoint.loguid
  - checkpoint.time
  - checkpoint.segment_time
  - checkpoint.lastupdatetime
  - checkpoint.sequencenum
  - checkpoint.update_count
target_field: "_id"
ignore_missing: true

The description text in the changelog could be improved. Suggest something like "Fix document fingerprint to prevent rejection of Check Point update events. Previously, fingerprints lacked fields that would distinguish update events from their originals."

The integration should probably add a latest transform as well, but that is future work.

[ash-darin]

HI @efd6 ,

as I have used the described fingerprint forsome time and have not observed drops, do you remember why segment_time is important? It's perfectly possible that your combination is only observed in certain circumstances, we all only see what the installations before us produce after all. You mentiond an issue, do you have a link?

P.S. I don't mean to haggle over this, I am just genuinely curious, especially if you have any documentation I could use to develop my understanding further.

[ilyannn]

@ash-darin Thank you for your work. Can you please resolve the conflicts? I'm running the checks in the meanwhile.

[ash-darin]

@ash-darin Thank you for your work. Can you please resolve the conflicts? I'm running the checks in the meanwhile.

@ilyannn I have corrected the conflicts. Didn't notice them, probably a problem with my notifications.

[ilyannn]

LGTM, everything as done as @efd6 requested

[ilyannn]

/test

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package checkpoint 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
firewall	1182.03	938.09	-243.94 (-20.64%)	💔

To see the full report comment with /test benchmark fullreport

[ilyannn]

/test

[ilyannn]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 851073e

History

• 💚 Build #43163 succeeded 161d292
• 💚 Build #43089 succeeded f300b52

[elastic-vault-github-plugin-prod]

Package checkpoint - 1.46.0 containing this change is available at https://epr.elastic.co/package/checkpoint/1.46.0/