[sentinel_one] Add Support for Optional Threat Timeline Collection

[mohitjha-elastic]

Proposed commit message

sentinel_one: Adds optional threat timeline collection to the threat data stream

This change adds optional threat timeline collection to the SentinelOne threat data stream.
When enabled, the integration retrieves paginated timeline entries for each threat and emits
each entry as a separate event under the `sentinel_one.threat.timeline.*` fields.
Additionally, STAR rule timeline entries are mapped to the ECS `rule.id`, `rule.name`, and
`rule.description` fields to provide enriched detection metadata.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install the elastic package locally.
• Start the elastic stack using the elastic package.
• Move to integrations/packages/sentinel_one directory.
• Run the following command to run tests.

elastic-package test -v

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

Elastic Docs Style Checker (Vale)

Summary: 1 suggestion found

💡 Suggestions (1): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/sentinel_one/data_stream/threat/manifest.yml	38	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[vera-review-bot]

Severity: 🟡 Medium confidence: medium path: packages/sentinel_one/data_stream/threat/agent/stream/httpjson_as_cel.yml.hbs:96

A list-fetch error after the first successful page is silently swallowed: the phase-2 empty-worklist branch overwrites the phase-1 error event because the !has(state.worklist) guard never fires once worklist has been set.

Details

When the threat-list request fails (non-200), phase 1 returns a single-object error {"events": {"error": {...}}, "want_more": false} and does not set worklist. Phase 2 is meant to bail out via !has(state.worklist) ? state. However, draining a page sets worklist to {"data": tail(...)}, which becomes {"data": []} for the last item and is never removed. So after the first successful fetch in a session, has(state.worklist) is permanently true. On the next interval, if the list fetch errors, state.worklist is still the stale {"data": []}, so phase 2 skips the guard, falls into the empty-worklist branch, and state.with({"events": [], ...}) overwrites the error object produced by phase 1. The result: the ERROR log / degraded-status signal is lost and the failure is silently skipped (the cursor is untouched so data is retried next interval, but the error is never surfaced). The // Exit early due to GET failure guard is effectively dead code after the first evaluation.

Recommendation:

Preserve a phase-1 error before entering the timeline stage. Because events is cleared between evaluations and phase-1 success never sets events, has(state.events) reliably means "the list fetch errored" — short-circuit on it:

).as(state,
  state.with(
    has(state.events) ? // list fetch failed this evaluation; preserve the error and skip the timeline stage.
      state
    : !has(state.worklist) ?
      state
    : (size(state.?worklist.?data.orValue([])) > 0) ?
      state.worklist.data[0].as(threat,
        // ... unchanged timeline logic ...
      )
    :
      (state.?next_page.token.orValue(null) != null) ?
        {"events": [{"retry": true}], "want_more": true}
      :
        {"events": [], "want_more": false, "list_updated_at_gte": ""}
  )
)

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🔵 Low confidence: medium path: packages/sentinel_one/data_stream/threat/agent/stream/httpjson_as_cel.yml.hbs:206

The empty-worklist branch returns want_more: true with an empty events array; the input only re-evaluates on a non-empty events, so an empty threat-list page carrying a non-null nextCursor stalls pagination until the next interval instead of following the cursor immediately.

Details

In the phase-2 empty-worklist branch, when state.next_page.token is present the program returns {"events": [], "want_more": true}. The CEL input triggers an immediate re-evaluation on want_more: true only when at least one event was published; with an empty events array the pending next_page.token is not followed within the current collection cycle. Collection does resume on the next scheduled interval (state persists next_page.token and phase 1 re-fetches with the cursor), so this is not a permanent stall, but pages after an empty-but-more list response are delayed by one interval. This path is reached only if the list API returns an empty data array together with a non-null pagination.nextCursor.

Recommendation:

Emit a placeholder event on the continuation path so the re-evaluation actually fires, and drop it before indexing:

(state.?next_page.token.orValue(null) != null) ?
  {"events": [{"retry": true}], "want_more": true}
:
  {"events": [], "want_more": false, "list_updated_at_gte": ""}

and discard the placeholder in the agent processors (or the ingest pipeline):

processors:
  - drop_event.when.equals.retry: true

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

Suggested change

	(size(state.?worklist.?data.orValue([])) > 0) ?
	state.?worklist.data[0].hasValue() ?

[efd6]

Suggested change

	"list_updated_at_gte": (state.?cursor.?last_update_at.orValue("") != "") ?
	string(state.cursor.last_update_at)
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339),
	"list_updated_at_gte": state.?cursor.last_update_at.orValue((now - duration(state.initial_interval)).format(time_layout.RFC3339)),

This will require rejigging the code below. In general, avoid using "", null, 0 etc as a sentinel for absence; we have absence to signal absence.

[kcreddy]

Also, it looks like its supposed to be last_updated_at_gte instead of list_updated_at_gte ? nvm

[efd6]

Suggested change

	"list_updated_at_gte": has_more_list ? state.list_updated_at_gte : "",
	?"list_updated_at_gte": has_more_list ? optional.of(state.list_updated_at_gte) : optional.none(),

[efd6]

Suggested change

	"events": (size(body.?data.orValue([])) > 0) ?
	"events": body.?data[0].hasValue() ?

[efd6]

Suggested change

	: (size(state.?worklist.?data.orValue([])) > 0) ?
	: state.?worklist.data[0].hasValue() ?

[efd6]

Suggested change

	"id": string(resp.Status),
	"id": resp.Status,

[efd6]

Suggested change

	string(resp.Status) + " (" + string(resp.StatusCode) + ")"
	resp.Status + " (" + string(resp.StatusCode) + ")"

[efd6]

Avoid "" for absence (also below).

[kcreddy]

Does missing permissions also cause 404? In CEL code, 404 emits a threat document, but other non-200 returns error. So just want to clarify the wording here.

[mohitjha-elastic]

Updated the wording. Thanks!

[kcreddy]

Looks like we are overriding previous value after adding timeline. This could break?

[efd6]

All of this should be persisted into the cursor. See #20022.

[vera-review-bot]

Severity: 🔵 Low confidence: medium path: packages/sentinel_one/data_stream/threat/_dev/test/policy/test-all.yml:8

The ssl test var was reformatted from a multi-line YAML block scalar into a single escaped one-line string, an unrelated change that makes the test config hard to read; restore the block-scalar form.

Details

This PR adds optional threat timeline collection, but it also rewrites the ssl var in this policy test from a readable literal block scalar (ssl: | followed by indented YAML) into one long double-quoted string with embedded \n escapes. The two forms parse to the same value, so this is not a functional change, but the escaped one-liner is effectively unreadable and unrelated to the timeline feature. Unrelated, readability-reducing churn in a test fixture is worth reverting to keep the diff focused and the file maintainable.

Recommendation:

Restore the literal block-scalar form so the SSL config stays readable (cert/key bodies elided here):

  ssl: |
    enabled: true
    supported_protocols:
      - TLSv1.2
    cipher_suites:
      - ECDHE-ECDSA-AES-128-CBC-SHA
      - ECDHE-ECDSA-AES-256-GCM-SHA384
    curve_types:
      - P-256
    certificate_authorities:
      - |
        <localhost test CA PEM>
    certificate: |
      <localhost test certificate PEM>
    key: |
      <localhost test key PEM>

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

Severity: 🟠 High confidence: high path: packages/sentinel_one/data_stream/threat/agent/stream/httpjson_as_cel.yml.hbs:195

On a transient (non-404) timeline error the threat document is dropped, not just the enrichment: the branch emits only an error, advances the worklist with tail(), and bumps last_update_at, so the underlying threat is never emitted or re-collected. Emit the threat document on this branch too, like the 404 branch does.

Details

Timeline enrichment is an optional secondary call, but its failure handling is not best-effort for non-404 responses. When the timeline GET returns any status other than 200 or 404 (e.g. a transient 500/502/503 or a 429 rate-limit), this branch returns events containing only an error object and no threat message, while still advancing the worklist ("worklist": {"data": tail(state.cursor.worklist.data)}) and moving the list watermark forward (?"last_update_at": threat.?threatInfo.updatedAt.or(...)). The error-only event has error.message set with no message/event.original, so the pipeline's terminate processor discards it. Net result: the current threat is permanently lost, and because last_update_at was advanced past it, the next interval's list query (updatedAt__gte) will not return it again. The sibling 404 branch (lines 164-176) handles this correctly by emitting [{"message": threat.encode_json()}] before advancing, so a threat is never dropped merely because the timeline endpoint was unavailable. The two branches should be consistent: a failed best-effort enrichment must not discard the primary threat record.

Recommendation:

On the non-404 error branch, still emit the threat document (as the 404 branch does) so enrichment failure never drops the primary record. Optionally keep the error in a separate event for visibility:

:
  {
    "events": [
      {"message": threat.encode_json()},
      {
        "error": {
          "code": string(resp.StatusCode),
          "id": resp.Status,
          "message": "GET " + state.url.trim_right("/") + "/" + string(threat.id) + "/timeline: " + (
            (size(resp.Body) != 0) ?
              string(resp.Body)
            :
              resp.Status + " (" + string(resp.StatusCode) + ")"
          ),
        },
      },
    ],
    "cursor": state.cursor.with(
      {
        "worklist": {"data": tail(state.cursor.worklist.data)},
        "next_chain": {},
        "has_more_timeline": false,
        ?"last_update_at": threat.?threatInfo.updatedAt.or(state.?cursor.?last_update_at),
      }
    ),
    "want_more": size(state.cursor.worklist.data) > 1 || state.?cursor.?next_page.token.orValue(null) != null,
  }

Alternatively, if transient errors should be retried instead of skipped, do not tail() the worklist and do not advance last_update_at on this branch so the same threat is re-attempted on the next execution.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Review summary

Issues found across the latest commits 3a7160f — 1 high

• 🟠 On a transient (non-404) timeline error the threat document is dropped, not just the enrichment: the branch emits only an error, advances the worklist with tail(), and bumps last_update_at, so the underlying threat is never emitted or re-collected. Emit the threat document on this branch too, like the 404 branch does. (link) (Unresolved)

Issues found across earlier commits 5a9cf69…4a27717 (24 commits) — 1 low

• 🔵 The ssl test var was reformatted from a multi-line YAML block scalar into a single escaped one-line string, an unrelated change that makes the test config hard to read (link) (Unresolved)

Issues found across earlier commits b0c46c5 — 1 medium, 1 low

• 🟡 A list-fetch error after the first successful page is silently swallowed: the phase-2 empty-worklist branch overwrites the phase-1 error event because the !has(state.worklist) guard never fires once worklist has been set. (link) (Unresolved)
• 🔵 The empty-worklist branch returns want_more: true with an empty events array (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 3a7160f

History

• 💚 Build #45699 succeeded 4a27717
• 💚 Build #45613 succeeded acd947c
• 💚 Build #45525 succeeded b0c46c5

cc @mohitjha-elastic

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

Package sentinel_one - 2.11.0 containing this change is available at https://epr.elastic.co/package/sentinel_one/2.11.0/