[salesforce] Order EventLogFile queries by the cursor field (CreatedDate) by shmsr · Pull Request #19954 · elastic/integrations

shmsr · 2026-07-03T08:51:07Z

What does this PR do?

Orders the Apex, Login, and Logout EventLogFile queries by CreatedDate (the cursor field) instead of LogDate, in both the default and value queries.

Background: two unrelated timestamps

Every EventLogFile record has two datetime fields that mean different things:

LogDate — the period the log covers (e.g. the start of the hour/day).
CreatedDate — when Salesforce actually generated the file. This lags LogDate, and the lag is variable (minutes to days), so CreatedDate order does not follow LogDate order.

These data streams track the collection cursor on CreatedDate (cursor.field: CreatedDate) and resume with WHERE CreatedDate > <cursor>, but the queries sorted the results by LogDate:

-- before
... WHERE CreatedDate > [[ .cursor.event_log_file.last_event_time ]] AND (<event types>) ORDER BY LogDate ASC NULLS FIRST

The bug, with an example

The input watermarks the cursor from the last record it processes in a page. When the page is ordered by LogDate, that last record is not necessarily the one with the greatest CreatedDate.

Consider two log files where the one covering the later period happened to be generated first:

#	LogDate (period)	CreatedDate (generated)
1	`2026-06-22T00:00:00Z`	`2026-06-24T11:28:06Z`
2	`2026-06-23T00:00:00Z`	`2026-06-24T11:08:05Z`

ORDER BY LogDate ASC returns them as #1 then #2. The input processes #1 (sets watermark 11:28:06), then #2 (overwrites watermark with 11:08:05). So the stored last_event_time ends at 2026-06-24T11:08:05Z — earlier than a record it already ingested (11:28:06).

On the next poll:

WHERE CreatedDate > 2026-06-24T11:08:05Z ...

…record #1 (11:28:06) matches again and is re-collected. The cursor effectively lags behind the data and re-fetches already-ingested files each poll.

This is not a contrived case — sorting real EventLogFile results by LogDate produces multiple such CreatedDate inversions per page whenever files are generated slightly out of period order (common for hourly logs).

The fix

Order by the cursor field so the last record always carries the maximum CreatedDate:

-- after
... WHERE CreatedDate > [[ .cursor.event_log_file.last_event_time ]] AND (<event types>) ORDER BY CreatedDate ASC NULLS FIRST

With the example above, ORDER BY CreatedDate ASC returns #2 then #1, so the watermark ends at 2026-06-24T11:28:06Z (the true maximum) and the next poll (CreatedDate > 11:28:06) does not re-collect either file. This matches the already-correct SetupAuditTrail query, which orders by its cursor field.

Why is it important?

Prevents repeated re-collection of already-ingested EventLogFile data and keeps the collection cursor moving strictly forward.

Compatibility / upgrade safety

The change is limited to the ORDER BY clause. The WHERE filter and cursor.field are unchanged, so:

Existing persisted cursors remain valid CreatedDate values and are interpreted identically.
For any given cursor value, the exact same set of records is returned — only their order (and the resulting watermark) changes — so no records are skipped on upgrade. A previously lagging cursor simply re-fetches its small lag window once and then advances cleanly.

Checklist

I have reviewed tips for building integrations and this change adheres to the guidelines.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package lint in packages/salesforce.
elastic-package test system -v for the apex, login, and logout data streams (the mock server query matchers in _dev/deploy/docker/files/config.yml are updated to match the new ordering).

…ate) The Apex, Login, and Logout EventLogFile queries sorted results by `LogDate` while tracking the collection cursor on `CreatedDate`. Those two fields are not correlated for EventLogFile records: Salesforce can create a log for an earlier `LogDate` period after one for a later period, so the last record in `LogDate` order frequently does not carry the maximum `CreatedDate`. Because the input watermarks the cursor from the last processed record, the stored `event_log_file.last_event_time` could be set below the newest `CreatedDate` already ingested, so the next poll re-collected data it had already fetched. Order these queries by `CreatedDate` (the cursor field), matching the already-correct SetupAuditTrail query, so the watermark only advances. The change is limited to the ORDER BY clause; the WHERE filter and cursor field are unchanged, so existing persisted cursors remain valid and no data is skipped on upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-03T08:54:21Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Copilot

Pull request overview

Aligns Salesforce EventLogFile ordering with the CreatedDate cursor to prevent cursor regression and duplicate collection.

Changes:

Orders Apex, Login, and Logout queries by CreatedDate.
Updates system-test mock query matchers.
Bumps the package to 1.8.1 with a changelog entry.

Repository review skills can be installed with /plugin install integration-skills@elastic-integration-skills.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`packages/salesforce/manifest.yml`	Bumps package version.
`packages/salesforce/changelog.yml`	Documents the cursor fix.
`packages/salesforce/data_stream/apex/agent/stream/salesforce.yml.hbs`	Corrects Apex query ordering.
`packages/salesforce/data_stream/login/agent/stream/salesforce.yml.hbs`	Corrects Login query ordering.
`packages/salesforce/data_stream/logout/agent/stream/salesforce.yml.hbs`	Corrects Logout query ordering.
`packages/salesforce/_dev/deploy/docker/files/config.yml`	Updates mock query expectations.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

elastic-vault-github-plugin-prod · 2026-08-11T07:56:23Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

ishleenk17

1 question.
Otherwise looks good!

ishleenk17 · 2026-08-19T05:51:18Z

    query:
-      default: SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE {{#if initial_interval}}LogDate > [[ (formatTime (now.Add (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.000Z0700") ]] AND {{/if}}{{#if log_file_interval}}Interval = '{{log_file_interval}}' AND {{/if}}(EventType = 'ApexCallout' OR EventType = 'ApexExecution' OR EventType = 'ApexRestApi' OR EventType = 'ApexSoap' OR EventType = 'ApexTrigger' OR EventType = 'ExternalCustomApexCallout') ORDER BY LogDate ASC NULLS FIRST
-      value: SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE {{#if log_file_interval}}Interval = '{{log_file_interval}}' AND {{/if}}CreatedDate > [[ .cursor.event_log_file.last_event_time ]] AND (EventType = 'ApexCallout' OR EventType = 'ApexExecution' OR EventType = 'ApexRestApi' OR EventType = 'ApexSoap' OR EventType = 'ApexTrigger' OR EventType = 'ExternalCustomApexCallout') ORDER BY LogDate ASC NULLS FIRST
+      default: SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE {{#if initial_interval}}LogDate > [[ (formatTime (now.Add (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.000Z0700") ]] AND {{/if}}{{#if log_file_interval}}Interval = '{{log_file_interval}}' AND {{/if}}(EventType = 'ApexCallout' OR EventType = 'ApexExecution' OR EventType = 'ApexRestApi' OR EventType = 'ApexSoap' OR EventType = 'ApexTrigger' OR EventType = 'ExternalCustomApexCallout') ORDER BY CreatedDate ASC NULLS FIRST


@shmsr :

So the initial lookback window is bounded by LogDate, but once the cursor is established, all subsequent polls use CreatedDate.

Is that right ?

Yes!

LogDate is the start of the interval the events occurred in (the hour/day the file covers). CreatedDate is when Salesforce generated the file. So, CreatedDate is now being used to find new files, including late ones as files are not always generated in LogDate order, and a later job can regenerate a file for an earlier LogDate.

So the first poll (no cursor) depends on LogDate for initial_interval. After that we cursor on CreatedDate.

mergify · 2026-08-19T05:52:40Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

elastic-vault-github-plugin-prod · 2026-08-20T06:15:30Z

✅ All changelog entries have the correct PR link.

infra-vault-gh-plugin-prod · 2026-08-20T06:38:26Z

💚 Build Succeeded

Buildkite Build
Commit: 575a250

History

💚 Build #47605 succeeded 02fd821
💚 Build #46675 succeeded 8ad1b2e
💚 Build #46285 succeeded bc35fac
💚 Build #45456 succeeded 4da7d60

cc @shmsr

elastic-vault-github-plugin-prod · 2026-08-20T07:28:46Z

Package salesforce - 1.8.1 containing this change is available at https://epr.elastic.co/package/salesforce/1.8.1/

shmsr requested a review from a team as a code owner July 3, 2026 08:51

[salesforce] Point changelog entry at the PR

ad74671

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

shmsr self-assigned this Jul 3, 2026

shmsr added the Integration:salesforce Salesforce label Jul 3, 2026

shmsr requested a review from stefans-elastic July 3, 2026 08:52

Merge branch 'main' into salesforce-elf-cursor-order-fix

4da7d60

andrewkroh added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label Jul 3, 2026

shmsr added 3 commits July 20, 2026 23:06

Merge branch 'main' into salesforce-elf-cursor-order-fix

bc35fac

Merge branch 'main' into salesforce-elf-cursor-order-fix

8ad1b2e

Merge branch 'main' into salesforce-elf-cursor-order-fix

02fd821

shmsr requested a balanced review from Copilot and removed request for stefans-elastic August 11, 2026 07:28

Copilot started reviewing on behalf of shmsr August 11, 2026 07:28 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

ishleenk17 approved these changes Aug 19, 2026

View reviewed changes

Merge branch 'main' into salesforce-elf-cursor-order-fix

575a250

Copilot AI review requested due to automatic review settings August 20, 2026 06:10

Copilot started reviewing on behalf of shmsr August 20, 2026 06:11 View session

Copilot AI reviewed Aug 20, 2026

View reviewed changes

shmsr merged commit fdb3e51 into elastic:main Aug 20, 2026
9 checks passed

shmsr mentioned this pull request Aug 25, 2026

[salesforce] Bounded real-time batching and same-timestamp resume #20884

Closed

14 tasks

Navigation Menu

Uh oh!

[salesforce] Order EventLogFile queries by the cursor field (CreatedDate) - #19954

[salesforce] Order EventLogFile queries by the cursor field (CreatedDate)#19954
shmsr merged 7 commits into
elastic:mainfrom
shmsr:salesforce-elf-cursor-order-fix

shmsr commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026

Copilot AI left a comment

elastic-vault-github-plugin-prod Bot commented Aug 11, 2026

ishleenk17 left a comment

ishleenk17 Aug 19, 2026

shmsr Aug 19, 2026 •

edited

Loading

mergify Bot commented Aug 19, 2026

Copilot AI left a comment

elastic-vault-github-plugin-prod Bot commented Aug 20, 2026

infra-vault-gh-plugin-prod Bot commented Aug 20, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 20, 2026

Labels

4 participants

Uh oh!

Conversation

shmsr commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Background: two unrelated timestamps

The bug, with an example

The fix

Why is it important?

Compatibility / upgrade safety

Checklist

How to test this PR locally

github-actions Bot commented Jul 3, 2026

✅ Elastic Docs Style Checker (Vale)

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

elastic-vault-github-plugin-prod Bot commented Aug 11, 2026

🚀 Benchmarks report

ishleenk17 left a comment

Choose a reason for hiding this comment

ishleenk17 Aug 19, 2026

Choose a reason for hiding this comment

shmsr Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mergify Bot commented Aug 19, 2026

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

elastic-vault-github-plugin-prod Bot commented Aug 20, 2026

infra-vault-gh-plugin-prod Bot commented Aug 20, 2026

💚 Build Succeeded

History

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 20, 2026

Labels

4 participants

shmsr commented Jul 3, 2026 •

edited

Loading

shmsr Aug 19, 2026 •

edited

Loading