Skip to content

Use ECS definitions for email.attachments and threat.enrichments - #19396

Merged
efd6 merged 9 commits into
elastic:mainfrom
chrisberkhout:ssi-nested-ecs-fields
Jun 10, 2026
Merged

Use ECS definitions for email.attachments and threat.enrichments#19396
efd6 merged 9 commits into
elastic:mainfrom
chrisberkhout:ssi-nested-ecs-fields

Conversation

@chrisberkhout

@chrisberkhout chrisberkhout commented Jun 5, 2026 •
edited by efd6
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Use ECS definitions for email.attachments and threat.enrichments

The external ECS definition is explicitly used for these fields, because
the dynamic template shipped with Elasticsearch doesn't map any nested
fields[1] and these are the ones we use.

This is for all relevant SSI integrations:

- abnormal_security
- amazon_security_lake
- box_events
- cloudflare_logpush
- darktrace
- dataminr_pulse
- eset_protect
- google_workspace
- ironscales
- microsoft_exchange_online_message_trace
- mimecast
- o365
- proofpoint_365totalprotection
- proofpoint_essentials
- proofpoint_on_demand
- proofpoint_tap
- sublime_security
- ti_greynoise
- ti_rapid7_threat_command
- zscaler_zia

For the threat intelligence integrations, the change is made in both the
source data streams and the transform destination index field mappings.

There are subfields of these nested fields that are mapped differently
(in ECS). Those are untouched when possible, and the external ECS
definitions for for them are added wherever necessary to work around an
issue in `elastic-package`'s system test validation[2].

The aws_bedrock integration uses ECS git@v8.11.0. That version doesn't
include the field gen_ai.response.finish_reasons, which the integration
manually maps to keyword but later versions of ECS map to nested. That
manual mapping is untouched here, but should be revisited with a general
ECS update.

[1]: https://github.com/elastic/elasticsearch/issues/150902
[2]: https://github.com/elastic/elastic-package/issues/3639

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues
@chrisberkhout chrisberkhout self-assigned this Jun 5, 2026
@chrisberkhout
chrisberkhout requested review from a team as code owners June 5, 2026 11:17
@chrisberkhout chrisberkhout added Integration:All Applies to all integrations [Integration not found in source] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jun 5, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jun 5, 2026

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@chrisberkhout
Use ECS definitions for email.attachments and threat.enrichments.
da79f39 
The external ECS definition is explicitly used for these fields, because
the dynamic template shipped with Elasticsearch doesn't map any nested
fields and these are the ones we use.

This is for all relevant SSI integrations:

- abnormal_security
- amazon_security_lake
- box_events
- cloudflare_logpush
- darktrace
- dataminr_pulse
- eset_protect
- google_workspace
- ironscales
- microsoft_exchange_online_message_trace
- mimecast
- o365
- proofpoint_365totalprotection
- proofpoint_essentials
- proofpoint_on_demand
- proofpoint_tap
- sublime_security
- ti_google_threat_intelligence
- ti_greynoise
- ti_rapid7_threat_command
- zscaler_zia

For the threat intelligence integrations, the change is made in both the
source data streams and the transform destination index field mappings.

There are subfields of these nested fields that are mapped differently
(in ECS), and those are untouched.

The aws_bedrock integration uses ECS git@v8.11.0. That version doesn't
include the field gen_ai.response.finish_reasons, which the integration
manually maps to keyword but later versions of ECS map to nested. That
manual mapping is untouched here, but should be revisited with a general
ECS update.
@chrisberkhout
Version bumps, changelog entries.
a4e6511
@chrisberkhout
chrisberkhout force-pushed the ssi-nested-ecs-fields branch from cb11670 to a4e6511 Compare June 5, 2026 11:34
@elastic elastic deleted a comment from github-actions Bot Jun 5, 2026
@elastic elastic deleted a comment from elasticmachine Jun 5, 2026
@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
@andrewkroh andrewkroh added Integration:eset_protect ESET PROTECT Integration:amazon_security_lake Amazon Security Lake Integration:o365 Microsoft Office 365 Integration:microsoft_exchange_online_message_trac Microsoft Exchange Online Message Trace Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Integration:ironscales IRONSCALES Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:dataminr_pulse Dataminr Pulse Integration:mimecast Mimecast Integration:box_events Box Events Integration:ti_greynoise GreyNoise (Community supported) Integration:proofpoint_essentials Proofpoint Essentials Integration:google_workspace Google Workspace documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:cloudflare_logpush Cloudflare Logpush Integration:proofpoint_on_demand Proofpoint On Demand Integration:proofpoint_tap Proofpoint TAP Integration:abnormal_security Abnormal AI labels Jun 5, 2026
efd6 and others added 2 commits June 9, 2026 14:59
@efd6
Merge branch 'master' into ssi-nested-ecs-fields
b3a57b6
@kcreddy
Merge remote-tracking branch 'upstream/main' into ssi-nested-ecs-fields
7ffa0b9 
# Conflicts:
#	packages/eset_protect/changelog.yml
#	packages/eset_protect/manifest.yml
@efd6
efd6 force-pushed the ssi-nested-ecs-fields branch from 13f2eb0 to 48a12c7 Compare June 9, 2026 23:09
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

TL;DR

All 4 failed jobs are the same failure: the package README files are stale versus current generated output, so check package aborts with checking readme files are up-to-date failed: files do not match.

Remediation

  • Regenerate and commit README output for the affected packages: packages/cloudflare_logpush, packages/ironscales, packages/o365, and packages/proofpoint_on_demand (run elastic-package build for each package).
  • Re-run the same package checks used in CI (.buildkite/scripts/test_one_package.sh ...) to confirm the readme drift is resolved.
Investigation details

Root Cause

This is a documentation generation drift failure (not a runtime test failure): Buildkite’s package check regenerated README content and found diffs, then failed.

The diffs are all in ECS field-table content around email.attachments nested fields (for example, new rows like email.attachments.file.hash.md5, email.attachments.file.name, email.attachments.file.size, etc.).

Relevant ECS references in package build config:

  • packages/cloudflare_logpush/_dev/build/build.yml:3 (git@v9.3.0)
  • packages/ironscales/_dev/build/build.yml:3 (git@v9.3.0)
  • packages/o365/_dev/build/build.yml:3 (git@v8.11.0)
  • packages/proofpoint_on_demand/_dev/build/build.yml:3 (git@v8.11.0)

Evidence

  • Build: https://buildkite.com/elastic/integrations/builds/44335
  • Jobs:
    • Check integrations cloudflare_logpush
    • Check integrations ironscales
    • Check integrations o365
    • Check integrations proofpoint_on_demand
  • Key log excerpts:
    • /tmp/gh-aw/buildkite-logs/integrations-check-integrations-cloudflare_logpush.txt:95 → README.md is outdated. Rebuild the package with 'elastic-package build'
    • /tmp/gh-aw/buildkite-logs/integrations-check-integrations-cloudflare_logpush.txt:119 → checking readme files are up-to-date failed: files do not match
    • same failure signature appears in the other 3 job logs at the corresponding lines.

Verification

  • Not run in this workflow environment; conclusion is based on direct Buildkite log diffs and failure messages.

Follow-up

If you already regenerated READMEs and still see this, make sure all generated README changes are committed (including any package-level generated docs touched by elastic-package build).

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
@efd6
efd6 force-pushed the ssi-nested-ecs-fields branch from 48a12c7 to 9cf5ce8 Compare June 9, 2026 23:22
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.
@elasticmachine

elasticmachine commented Jun 10, 2026

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @chrisberkhout
@efd6
efd6 merged commit c4e7fa3 into elastic:main Jun 10, 2026
10 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package abnormal_security - 1.15.1 containing this change is available at https://epr.elastic.co/package/abnormal_security/1.15.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package amazon_security_lake - 2.8.3 containing this change is available at https://epr.elastic.co/package/amazon_security_lake/2.8.3/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package box_events - 3.3.1 containing this change is available at https://epr.elastic.co/package/box_events/3.3.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package cloudflare_logpush - 1.46.1 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.46.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package darktrace - 2.2.1 containing this change is available at https://epr.elastic.co/package/darktrace/2.2.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package dataminr_pulse - 0.2.2 containing this change is available at https://epr.elastic.co/package/dataminr_pulse/0.2.2/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package eset_protect - 2.5.1 containing this change is available at https://epr.elastic.co/package/eset_protect/2.5.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package google_workspace - 3.4.1 containing this change is available at https://epr.elastic.co/package/google_workspace/3.4.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package ironscales - 0.2.1 containing this change is available at https://epr.elastic.co/package/ironscales/0.2.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package microsoft_exchange_online_message_trace - 2.2.1 containing this change is available at https://epr.elastic.co/package/microsoft_exchange_online_message_trace/2.2.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package mimecast - 3.4.3 containing this change is available at https://epr.elastic.co/package/mimecast/3.4.3/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package o365 - 3.10.2 containing this change is available at https://epr.elastic.co/package/o365/3.10.2/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package proofpoint_365totalprotection - 0.1.1 containing this change is available at https://epr.elastic.co/package/proofpoint_365totalprotection/0.1.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package proofpoint_essentials - 1.1.1 containing this change is available at https://epr.elastic.co/package/proofpoint_essentials/1.1.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package proofpoint_on_demand - 1.10.1 containing this change is available at https://epr.elastic.co/package/proofpoint_on_demand/1.10.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package proofpoint_tap - 1.31.1 containing this change is available at https://epr.elastic.co/package/proofpoint_tap/1.31.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package sublime_security - 1.12.1 containing this change is available at https://epr.elastic.co/package/sublime_security/1.12.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package ti_greynoise - 0.9.1 containing this change is available at https://epr.elastic.co/package/ti_greynoise/0.9.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package ti_rapid7_threat_command - 2.10.1 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.10.1/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package zscaler_zia - 3.22.1 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.22.1/
1 similar comment
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Package zscaler_zia - 3.22.1 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.22.1/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:abnormal_security Abnormal AI Integration:amazon_security_lake Amazon Security Lake Integration:box_events Box Events Integration:cloudflare_logpush Cloudflare Logpush Integration:darktrace Darktrace Integration:dataminr_pulse Dataminr Pulse Integration:eset_protect ESET PROTECT Integration:google_workspace Google Workspace Integration:ironscales IRONSCALES Integration:microsoft_exchange_online_message_trac Microsoft Exchange Online Message Trace Integration:mimecast Mimecast Integration:o365 Microsoft Office 365 Integration:proofpoint_essentials Proofpoint Essentials Integration:proofpoint_on_demand Proofpoint On Demand Integration:proofpoint_tap Proofpoint TAP Integration:proofpoint_365totalprotection Proofpoint 365 Total Protection (Partner supported) Integration:sublime_security Sublime Security Integration:ti_greynoise GreyNoise (Community supported) Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:zscaler_zia Zscaler Internet Access Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants

@chrisberkhout @navnit-elastic @efd6 @elasticmachine @andrewkroh @kcreddy