Skip to content

{o365.audit},{m365_defender.alert}: Enhance field mappings - #12888

Merged
kcreddy merged 7 commits into
elastic:mainfrom
kcreddy:o365-m365_defender-new-fields
Mar 3, 2025
Merged

{o365.audit},{m365_defender.alert}: Enhance field mappings#12888
kcreddy merged 7 commits into
elastic:mainfrom
kcreddy:o365-m365_defender-new-fields

Conversation

@kcreddy

@kcreddy kcreddy commented Feb 25, 2025

Copy link
Copy Markdown
Contributor

Proposed commit message

Extract `email.*` fields from `m365_defender.alert` data-stream.
Also derive `source.ip` from `evidence.senderIp`

Extract `email.*`, `related.*`, `user.*` fields from `Data` and `Data.Entities` fields inside 
`o365.audit` data-stream.
Other enhancements to `o365.audit`:
- Make `AttachmentData` and `AuthDetails` as `nested` to make independent queries.
- Add several top-level `o365.audit` fields based on sample events.
- Add drop-null processor.
- Handle error.message at end of the pipeline.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Pipeline tests passes containing new fields:
1. M365 Defender

--- Test results for package: m365_defender - START ---
╭───────────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                                                      │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ alert       │ pipeline  │ (ingest pipeline warnings test-alert.log)                      │ PASS   │  348.92925ms │
│ m365_defender │ alert       │ pipeline  │ test-alert.log                                                 │ PASS   │  99.644209ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-alert.log)                      │ PASS   │ 376.436666ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-app-and-identity.log)           │ PASS   │ 342.705083ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-device.log)                     │ PASS   │ 364.856291ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-email.log)                      │ PASS   │ 327.063792ms │
│ m365_defender │ event       │ pipeline  │ test-alert.log                                                 │ PASS   │ 139.360208ms │
│ m365_defender │ event       │ pipeline  │ test-app-and-identity.log                                      │ PASS   │    132.109ms │
│ m365_defender │ event       │ pipeline  │ test-device.log                                                │ PASS   │  1.29156225s │
│ m365_defender │ event       │ pipeline  │ test-email.log                                                 │ PASS   │   95.93325ms │
│ m365_defender │ incident    │ pipeline  │ (ingest pipeline warnings test-incident.log)                   │ PASS   │ 357.501791ms │
│ m365_defender │ incident    │ pipeline  │ test-incident.log                                              │ PASS   │ 274.046417ms │
│ m365_defender │ log         │ pipeline  │ (ingest pipeline warnings test-m365-defender-empty-ndjson.log) │ PASS   │  322.74475ms │
│ m365_defender │ log         │ pipeline  │ (ingest pipeline warnings test-m365-defender-ndjson.log)       │ PASS   │  371.32925ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-empty-ndjson.log                            │ PASS   │  40.019459ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-ndjson.log                                  │ PASS   │ 159.439917ms │
╰───────────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

2. O365
New files added containing new fields:

  • test-wl-airinvestigation.json
  • test-wl-securitycompliancecenter.json
  • test-wl-threatintelligence.json
--- Test results for package: o365 - START ---
╭─────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────���───┤
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-azuread-events.json)                   │ PASS   │ 358.723375ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-azuread-sts-logon-events.json)         │ PASS   │ 384.870416ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-bad-ips.json)                          │ PASS   │ 385.783834ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-data-duplicated-querytime-events.json) │ PASS   │ 324.737417ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-data-insights-api-events.json)         │ PASS   │ 405.747208ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-dlp-exchange-events.json)              │ PASS   │ 387.674084ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-dlp-sharepoint-events.json)            │ PASS   │ 381.929792ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-exchange-admin-events.json)            │ PASS   │ 381.202584ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-exchange-item-events.json)             │ PASS   │ 398.749625ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-ip-formats-events.json)                │ PASS   │ 357.165834ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-modified-properites.json)              │ PASS   │ 356.582209ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-ms-teams-events.json)                  │ PASS   │ 356.578584ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-parameter-string.json)                 │ PASS   │    456.804ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-platform-attribute-events.json)        │ PASS   │   361.8045ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sec-comp-alerts-events.json)           │ PASS   │  370.36875ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sharepoint-events.json)                │ PASS   │ 367.199917ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sharepointfileop-events.json)          │ PASS   │ 372.218375ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sp-sharing-op-events.json)             │ PASS   │ 383.053708ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-stringly-json-events.json)             │ PASS   │ 413.074083ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-airinvestigation.json)              │ PASS   │ 351.694333ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-securitycompliancecenter.json)      │ PASS   │ 368.423084ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-threatintelligence.json)            │ PASS   │ 356.965541ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-yammer-events.json)                    │ PASS   │ 342.123791ms │
│ o365    │ audit       │ pipeline  │ test-azuread-events.json                                              │ PASS   │ 3.071293167s │
│ o365    │ audit       │ pipeline  │ test-azuread-sts-logon-events.json                                    │ PASS   │  1.98057625s │
│ o365    │ audit       │ pipeline  │ test-bad-ips.json                                                     │ PASS   │  77.807208ms │
│ o365    │ audit       │ pipeline  │ test-data-duplicated-querytime-events.json                            │ PASS   │  51.384292ms │
│ o365    │ audit       │ pipeline  │ test-data-insights-api-events.json                                    │ PASS   │  99.803625ms │
│ o365    │ audit       │ pipeline  │ test-dlp-exchange-events.json                                         │ PASS   │ 124.631375ms │
│ o365    │ audit       │ pipeline  │ test-dlp-sharepoint-events.json                                       │ PASS   │ 130.957167ms │
│ o365    │ audit       │ pipeline  │ test-exchange-admin-events.json                                       │ PASS   │    1.117815s │
│ o365    │ audit       │ pipeline  │ test-exchange-item-events.json                                        │ PASS   │  159.23625ms │
│ o365    │ audit       │ pipeline  │ test-ip-formats-events.json                                           │ PASS   │  118.68825ms │
│ o365    │ audit       │ pipeline  │ test-modified-properites.json                                         │ PASS   │  59.363083ms │
│ o365    │ audit       │ pipeline  │ test-ms-teams-events.json                                             │ PASS   │  77.816958ms │
│ o365    │ audit       │ pipeline  │ test-parameter-string.json                                            │ PASS   │    71.5595ms │
│ o365    │ audit       │ pipeline  │ test-platform-attribute-events.json                                   │ PASS   │   45.77825ms │
│ o365    │ audit       │ pipeline  │ test-sec-comp-alerts-events.json                                      │ PASS   │  75.670084ms │
│ o365    │ audit       │ pipeline  │ test-sharepoint-events.json                                           │ PASS   │ 133.682083ms │
│ o365    │ audit       │ pipeline  │ test-sharepointfileop-events.json                                     │ PASS   │ 342.594334ms │
│ o365    │ audit       │ pipeline  │ test-sp-sharing-op-events.json                                        │ PASS   │ 232.921417ms │
│ o365    │ audit       │ pipeline  │ test-stringly-json-events.json                                        │ PASS   │  96.670833ms │
│ o365    │ audit       │ pipeline  │ test-wl-airinvestigation.json                                         │ PASS   │  90.202166ms │
│ o365    │ audit       │ pipeline  │ test-wl-securitycompliancecenter.json                                 │ PASS   │ 124.492833ms │
│ o365    │ audit       │ pipeline  │ test-wl-threatintelligence.json                                       │ PASS   │  94.889416ms │
│ o365    │ audit       │ pipeline  │ test-yammer-events.json                                               │ PASS   │  68.542625ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: o365 - END   ---
Done

Related issues

@kcreddy kcreddy self-assigned this Feb 25, 2025
@kcreddy kcreddy added Integration:o365 Microsoft Office 365 Integration:m365_defender Microsoft Defender XDR enhancement New feature or request labels Feb 25, 2025
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Feb 25, 2025

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@kcreddy
kcreddy marked this pull request as ready for review February 28, 2025 11:39
@kcreddy
kcreddy requested a review from a team as a code owner February 28, 2025 11:39
@kcreddy
kcreddy marked this pull request as draft February 28, 2025 11:40
@kcreddy
kcreddy marked this pull request as ready for review February 28, 2025 11:45
Comment thread packages/o365/changelog.yml Outdated
changes:
- description: Extrac ECS fields from Data and AttachmentData.
type: enhancement
link: https://github.com/elastic/integrations/pull/1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
link: https://github.com/elastic/integrations/pull/1
link: https://github.com/elastic/integrations/pull/12888

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This keeps happening, I will follow your epb alias from now.

Comment thread packages/o365/changelog.yml Outdated
# newer versions go on top
- version: "2.10.0"
changes:
- description: Extrac ECS fields from Data and AttachmentData.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- description: Extrac ECS fields from Data and AttachmentData.
- description: Extract ECS fields from Data and AttachmentData.
Comment on lines +244 to +245
- name: Description
type: keyword

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

match_only_text?

Sadly this is not possible in the m365 case where the text is much longer.

field: o365audit.Parameters._raw
if: ctx.o365audit?.NetworkMessageId == null || ctx.o365audit.NetworkMessageId == ''
patterns:
- '^(-)?Identity\s(\")?%{DATA:o365audit.NetworkMessageId}(\")?$'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- '^(-)?Identity\s(\")?%{DATA:o365audit.NetworkMessageId}(\")?$'
- '^-?Identity\s"?%{DATA:o365audit.NetworkMessageId}"?$'
@kcreddy
kcreddy requested a review from efd6 March 3, 2025 05:42

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after the doc vet passes.

Comment thread packages/o365/data_stream/audit/fields/fields.yml
@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @kcreddy

@kcreddy
kcreddy merged commit acecb4d into elastic:main Mar 3, 2025
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package m365_defender - 2.23.0 containing this change is available at https://epr.elastic.co/package/m365_defender/2.23.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package o365 - 2.10.0 containing this change is available at https://epr.elastic.co/package/o365/2.10.0/

flexitrev pushed a commit that referenced this pull request Mar 20, 2025
Extract `email.*` fields from `m365_defender.alert` data-stream.
Also derive `source.ip` from `evidence.senderIp`

Extract `email.*`, `related.*`, `user.*` fields from `Data` and `Data.Entities` fields inside 
`o365.audit` data-stream.
Other enhancements to `o365.audit`:
- Make `AttachmentData` and `AuthDetails` as `nested` to make independent queries.
- Add several top-level `o365.audit` fields based on sample events.
- Add drop-null processor.
- Handle error.message at end of the pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:m365_defender Microsoft Defender XDR Integration:o365 Microsoft Office 365

3 participants