[Trendmicro Vision One] - Fixed cursor & pagination logic for "detections" data stream by ShourieG · Pull Request #12916 · elastic/integrations

ShourieG · 2025-02-27T08:41:16Z

Type of change

Bug
Breaking Change

Proposed commit message

Fixed cursor & pagination logic for "detections" data stream to mirror a similar logic in implemented for a google_workspace fix.

Please refer to this issue to see the details.

NOTE

This will be a breaking change for the detections data stream as cursor values are completely changed, so existing state will get affected.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

…nario in pull #34274

ShourieG · 2025-02-27T08:54:17Z

TODO - test this with a live account

elastic-vault-github-plugin-prod · 2025-02-27T09:11:31Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-02-27T11:39:58Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

This seems reasonable, but needs some commentary clean ups.

efd6 · 2025-02-27T21:45:59Z

+      # If last pagination cycle was finished successfully
+      # we move the startDate pointer forward
+      # else we reprocess the last interval from the beginning.
+      # If none of the values are in the cursor it means is a fresh start


Suggested change

# If none of the values are in the cursor it means is a fresh start

# If none of the values are in the cursor it means it is a fresh start

efd6 · 2025-02-27T21:46:57Z

  - set:
      target: url.params.startDateTime
-      value: '[[formatDate ((parseDate .cursor.last_update_at).Add (parseDuration "-{{additional_look_back}}"))]]'
+      # If last pagination cycle was finished successfully


Suggested change

# If last pagination cycle was finished successfully

# If the last pagination cycle was completed successfully

efd6 · 2025-02-27T21:47:19Z

-      value: '[[formatDate ((parseDate .cursor.last_update_at).Add (parseDuration "-{{additional_look_back}}"))]]'
+      # If last pagination cycle was finished successfully
+      # we move the startDate pointer forward
+      # else we reprocess the last interval from the beginning.


Suggested change

# else we reprocess the last interval from the beginning.

# otherwise we reprocess the last interval from the beginning.

efd6 · 2025-02-27T21:49:11Z

+      # and we pick the default value.
+      value: >-
+        [[- if eq .cursor.pagination_finished "true" -]]
+        # look-back time should only be applied for fresh pagination intervals


If you want to include comments in the Go template, you will need to use the template comment syntax.

Suggested change

# look-back time should only be applied for fresh pagination intervals

[[- /* look-back time should only be applied for fresh pagination intervals */ -]]

What is the precise definition of "fresh" here? I can see two possible interpretations; one matches the behaviour here (not started but after one that has completed), the other doesn't (never started). The immediately obvious interpretation is the second one, so I think this needs to be clearer.

By fresh pagination cycle I meant a completely new pagination cycle that occurs on every new interval not between pages during a particular interval, I'll make it a bit clearer

ShourieG · 2025-02-28T06:00:56Z

@efd6, addressed all the comments

efd6

Pending whether this TODO was TODONE and the change below.

efd6 · 2025-03-03T03:55:23Z

 # newer versions go on top
+- version: "1.25.2"
+  changes:
+    - description: Added pagination and cursor fix to mirror the logic for a similar google_workspace scenario in pull - https://github.com/elastic/beats/pull/34274.


Suggested change

- description: Added pagination and cursor fix to mirror the logic for a similar google_workspace scenario in pull - https://github.com/elastic/beats/pull/34274.

- description: Fixed pagination and cursor handling.

(users who want to know about pull requests will know how to find them, presenting them to other people is not really helpful, so instead of putting it here, put it in the commit message or just rely on the discussion in this PR — sorry, I missed this on the first round)

ShourieG · 2025-03-14T07:07:17Z

cc: @narph @jamiehynds, planning on merging this soon but just as a fyi, this will be a breaking change for the detection data stream as the existing state will be lost due to a complete overhaul of the cursor logic. Should we give any other warnings for our customers before merging ?

narph · 2025-03-14T16:10:59Z

cc: @narph @jamiehynds, planning on merging this soon but just as a fyi, this will be a breaking change for the detection data stream as the existing state will be lost due to a complete overhaul of the cursor logic. Should we give any other warnings for our customers before merging ?

@ShourieG lets not merge this till we discuss the impact on our users/customers

jamiehynds · 2025-03-14T16:27:52Z

@ShourieG what's the actual user impact? They upgrade the integration, we lose the state and they end up re-ingesting detections? Any ideas as to timeframe - will the last day, week, month worth of detections be re-ingested after upgrading the integration? If we don't implement this, we run the risk of incorrectly dropping data?

ShourieG · 2025-03-15T07:21:18Z

@jamiehynds, So if we implement this, the detections data stream will lose all existing state, the reingestion process will depend on the customers configuration as to how far they want to look back. If the customer does not change any configs and let things stay the way they are, the ingestion would start with the default startDateTime value which is calculated on the basis of (now - initial_interval) where initial_interval is user configured.
The timeframe would completely depend on the value of initial_interval.

If we don't implement this, we would end up in a similar state with what happened with google_workspace a couple of years back when the integration was dropping quite a bit of data due to how the api returned the responses. Basically we would get users complaining of data inconsistency which can vary from a small amount to quite large amounts on a case by case basis.

jamiehynds · 2025-03-19T09:19:50Z

@ShourieG while not ideal, I think implementing this change now is the less of two evils. Most users will likely prefer to deal with some reingestion than run the risk of dropping data due to current cursor logic. Plus, our default is 24h initial interval, and it's 'just' detections, so don't expect it do be a vast amount of data. We just need to make it clear in the changelog description that there's a breaking change included in this update and be clear on the impact (reingestion)

elasticmachine · 2025-03-20T09:56:55Z

💚 Build Succeeded

Buildkite Build
Commit: 3823c89

History

💚 Build #23352 succeeded eb22bb2
💚 Build #23275 succeeded c2db322
💚 Build #22894 succeeded 68dacfb
💚 Build #22835 succeeded d54f8d6
💚 Build #22804 succeeded 1096647

cc @ShourieG

elastic-sonarqube · 2025-03-20T09:57:03Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-03-20T10:45:53Z

Package trend_micro_vision_one - 2.0.0 containing this change is available at https://epr.elastic.co/package/trend_micro_vision_one/2.0.0/

…logic for "detections" data stream (#12916) * added pagination and cursor fix to mirror the logic for a similar scenario in pull #34274 * updated changelog & manifest * updated changelog * updated look back calculation logic * updated comments * addressed Dan's suggestions * updated changelog * updated changelog with appropriate breaking-change type * Update changelog with breaking change details

ShourieG added 2 commits February 27, 2025 13:53

added pagination and cursor fix to mirror the logic for a similar sce…

bf8e1c7

…nario in pull #34274

updated changelog & manifest

79b39d4

ShourieG self-assigned this Feb 27, 2025

ShourieG added Integration:trend_micro_vision_one TrendAI Vision One Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] bugfix Pull request that fixes a bug issue labels Feb 27, 2025

ShourieG added 3 commits February 27, 2025 14:14

updated changelog

5190a0e

updated look back calculation logic

6ab3593

updated comments

1096647

ShourieG requested a review from efd6 February 27, 2025 08:52

ShourieG removed the request for review from efd6 February 27, 2025 08:55

ShourieG marked this pull request as ready for review February 27, 2025 11:39

ShourieG requested a review from a team as a code owner February 27, 2025 11:39

ShourieG requested a review from efd6 February 27, 2025 11:40

efd6 reviewed Feb 27, 2025

View reviewed changes

addressed Dan's suggestions

d54f8d6

ShourieG requested a review from efd6 March 3, 2025 03:45

efd6 approved these changes Mar 3, 2025

View reviewed changes

ShourieG added 3 commits March 3, 2025 13:25

updated changelog

68dacfb

merged with upstream & resolved conflicts

c2db322

updated changelog with appropriate breaking-change type

eb22bb2

ShourieG added the breaking change label Mar 13, 2025

jamiehynds mentioned this pull request Mar 18, 2025

[Discuss] Support providing hints for breaking changes elastic/package-spec#817

Open

Update changelog with breaking change details

3823c89

ShourieG merged commit 7424baf into elastic:main Mar 20, 2025

ShourieG deleted the vision_one/pagination_fix branch March 20, 2025 10:29

Navigation Menu

Uh oh!

[Trendmicro Vision One] - Fixed cursor & pagination logic for "detections" data stream - #12916

[Trendmicro Vision One] - Fixed cursor & pagination logic for "detections" data stream#12916
ShourieG merged 10 commits into
elastic:mainfrom
ShourieG:vision_one/pagination_fix

ShourieG commented Feb 27, 2025 •

edited

Loading

ShourieG commented Feb 27, 2025 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Feb 27, 2025 •

edited

Loading

elasticmachine commented Feb 27, 2025

efd6 left a comment

efd6 Feb 27, 2025

efd6 Feb 27, 2025

efd6 Feb 27, 2025

efd6 Feb 27, 2025

ShourieG Feb 28, 2025

ShourieG commented Feb 28, 2025

efd6 left a comment •

edited

Loading

efd6 Mar 3, 2025

ShourieG commented Mar 14, 2025

narph commented Mar 14, 2025

jamiehynds commented Mar 14, 2025

ShourieG commented Mar 15, 2025 •

edited

Loading

jamiehynds commented Mar 19, 2025

elasticmachine commented Mar 20, 2025

elastic-sonarqube Bot commented Mar 20, 2025

elastic-vault-github-plugin-prod Bot commented Mar 20, 2025

Labels

5 participants

	# If none of the values are in the cursor it means is a fresh start
	# If none of the values are in the cursor it means it is a fresh start

	# If last pagination cycle was finished successfully
	# If the last pagination cycle was completed successfully

	# else we reprocess the last interval from the beginning.
	# otherwise we reprocess the last interval from the beginning.

	# look-back time should only be applied for fresh pagination intervals
	[[- /* look-back time should only be applied for fresh pagination intervals */ -]]

	- description: Added pagination and cursor fix to mirror the logic for a similar google_workspace scenario in pull - https://github.com/elastic/beats/pull/34274.
	- description: Fixed pagination and cursor handling.

Uh oh!

Conversation

ShourieG commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Proposed commit message

NOTE

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

ShourieG commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elastic-vault-github-plugin-prod Bot commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

elasticmachine commented Feb 27, 2025

efd6 left a comment

Choose a reason for hiding this comment

efd6 Feb 27, 2025

Choose a reason for hiding this comment

efd6 Feb 27, 2025

Choose a reason for hiding this comment

efd6 Feb 27, 2025

Choose a reason for hiding this comment

efd6 Feb 27, 2025

Choose a reason for hiding this comment

ShourieG Feb 28, 2025

Choose a reason for hiding this comment

ShourieG commented Feb 28, 2025

efd6 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

efd6 Mar 3, 2025

Choose a reason for hiding this comment

ShourieG commented Mar 14, 2025

narph commented Mar 14, 2025

jamiehynds commented Mar 14, 2025

ShourieG commented Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jamiehynds commented Mar 19, 2025

elasticmachine commented Mar 20, 2025

💚 Build Succeeded

History

elastic-sonarqube Bot commented Mar 20, 2025

Quality Gate passed

elastic-vault-github-plugin-prod Bot commented Mar 20, 2025

Labels

5 participants

ShourieG commented Feb 27, 2025 •

edited

Loading

ShourieG commented Feb 27, 2025 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Feb 27, 2025 •

edited

Loading

efd6 left a comment •

edited

Loading

ShourieG commented Mar 15, 2025 •

edited

Loading