m365_defender,microsoft_defender_endpoint: switch vulnerability data stream to delta API

[kcreddy]

Proposed commit message

m365_defender,microsoft_defender_endpoint: switch vulnerability data stream to delta API

Replace the SoftwareVulnerabilitiesExport (gzip file download) API with
the SoftwareVulnerabilityChangesByMachine (paginated JSON delta) API for
the vulnerability data stream. The gzip approach caused high memory usage
because entire compressed files were decompressed in memory.

The delta API returns only new, fixed, and updated vulnerabilities since
a given timestamp, using OData pagination. This eliminates gzip handling
from the CEL program entirely.

Key changes across both packages:

- Rewrite cel.yml.hbs to use paginated delta API with @odata.nextLink
  pagination and sinceTime cursor management.
- Add initial_interval variable (default 336h / 14 days, the API max)
  to control the first historical lookback window.
- Remove sas_valid_hours and max_retries variables.
- Add EventTimestamp (date) and Status (keyword) fields; remove
  SecurityUpdateAvailable (boolean).
- Add Painless script processor to normalize camelCase JSON field names
  to PascalCase, handling API response casing inconsistencies between
  the full-export and delta endpoints.
- Update ingest pipeline to process new fields.
- Widen transform source query from now-4h to now-90d and bump
  retention_policy.max_age to 2160h to accommodate delta semantics.
- Add vulnerability.status to transform destination field mappings.
- Bump fleet_transform_version to 0.4.0 to trigger reinstall.
- Update mock server configs, system test configs, pipeline test
  fixtures (with both PascalCase and camelCase events), policy test
  expected files, script tests, and Rally benchmark definitions.
- Change default polling interval from 4h to 6h.
- Bump m365_defender to 5.14.0, microsoft_defender_endpoint to 4.7.0.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[github-actions]

Vale Linting Results

Summary: 2 warnings found

⚠️ Warnings (2)

File	Line	Rule	Message
packages/m365_defender/_dev/build/docs/README.md	24	Elastic.DirectionalLanguage	Don't use directional language. Use 'in the following section' instead of 'listed below'.
packages/m365_defender/docs/README.md	24	Elastic.DirectionalLanguage	Don't use directional language. Use 'in the following section' instead of 'listed below'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

TL;DR

m365_defender is failing due to generated docs drift (README.md is outdated) and needs a docs rebuild committed to the PR. The microsoft_defender_endpoint failure cannot be root-caused from the attached log file because it only contains teardown/artifact-upload output after the failure.

Remediation

• Regenerate and commit the m365 docs output (from repo root): elastic-package build -v -C packages/m365_defender (or run the package build flow used locally) so packages/m365_defender/docs/README.md matches generated output.
• Re-run the Check integrations microsoft_defender_endpoint step and capture full step output (or provide the failing test section/JUnit failure text); current log starts at container teardown, so the triggering error is missing.

Investigation details

Root Cause

1. Configuration/docs drift (m365_defender)

   • The failing step reports a README mismatch during lint/check:
     • README.md is outdated. Rebuild the package with 'elastic-package build'
     • checking readme files are up-to-date failed: files do not match
   • This is a generated-doc consistency failure in the package check stage.

2. Inconclusive (microsoft_defender_endpoint)

   • The provided log file contains only teardown and artifact upload lines, then exit status 1, with no preceding test/assertion/error lines to attribute a concrete root cause.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/43553
• Job/step: Check integrations m365_defender
  • Log excerpt (/tmp/gh-aw/buildkite-logs/integrations-check-integrations-m365_defender.txt):

README.md is outdated. Rebuild the package with 'elastic-package build'
Error: checking package failed: checking readme files are up-to-date failed: files do not match

• Job/step: Check integrations microsoft_defender_endpoint
  • Log excerpt (/tmp/gh-aw/buildkite-logs/integrations-check-integrations-microsoft_defender_endpoint.txt):

--- [microsoft_defender_endpoint] failed
🚨 Error: The command exited with status 1
user command error: exit status 1
~~~ Uploading artifacts
...
~~~ Running repository pre-exit hook
--- Take down the Elastic stack
Done

Verification

• Not run in this workflow (read-only detective run).

Follow-up

• If you attach the first failing assertion/error chunk for microsoft_defender_endpoint (or the relevant JUnit failure body), I can map it directly to the exact source file/line and suggest a concrete patch.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• m365_defender,microsoft_defender_endpoint: switch vulnerability data stream to delta API #19034 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #19034 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[efd6]

The failures here appear to be due to the mock events being outside the 90 day source query causing the transforms to fail.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8e14f8e

History

• 💔 Build #43553 failed 22e5a0d
• 💚 Build #43551 succeeded 0e0d808
• 💔 Build #43550 failed a4b08bf
• 💔 Build #43549 failed 48ee87f
• 💚 Build #43003 succeeded d6b9680

cc @kcreddy

[elastic-vault-github-plugin-prod]

Package m365_defender - 5.14.0 containing this change is available at https://epr.elastic.co/package/m365_defender/5.14.0/

[elastic-vault-github-plugin-prod]

Package microsoft_defender_endpoint - 4.7.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/4.7.0/