Skip to content

[M365_Defender] Enhancement - Add support for IdentityInfo Table - #12214

Merged
ShourieG merged 0 commit into
elastic:mainfrom
jvalente-salemstate:m365_defender-add-identityinfo
Jan 15, 2025
Merged

[M365_Defender] Enhancement - Add support for IdentityInfo Table#12214
ShourieG merged 0 commit into
elastic:mainfrom
jvalente-salemstate:m365_defender-add-identityinfo

Conversation

@jvalente-salemstate

@jvalente-salemstate jvalente-salemstate commented Dec 31, 2024

Copy link
Copy Markdown
Contributor

Proposed commit message

Microsoft now forwards AdvancedHunting-IdentityInfo via the XDR streaming API. This adds support for this table and its fields.

Fields were checked against MS docs, which is not current. Additional fields from sample events were included.

 cat m365_sample.txt| jq 'keys'
[
  "Tenant",
  "_TimeReceivedBySvc",
  "category",
  "operationName",
  "properties",
  "tenantId",
  "time"
]

---

cat m365_sample.txt| jq '.properties|keys'
[
  "AccountDisplayName",
  "AccountDomain",
  "AccountName",
  "AccountObjectId",
  "AccountUpn",
  "Address",
  "BlastRadius",
  "ChangeSource",
  "City",
  "CloudSid",
  "CompanyName",
  "Country",
  "CreatedDateTime",
  "CriticalityLevel",
  "DeletedDateTime",
  "Department",
  "DistinguishedName",
  "EmailAddress",
  "EmployeeId",
  "GivenName",
  "IsAccountEnabled",
  "JobTitle",
  "Manager",
  "OnPremSid",
  "OtherMailAddresses",
  "Phone",
  "ReportId",
  "RiskLevel",
  "RiskLevelDetails",
  "SipProxyAddress",
  "SourceProvider",
  "State",
  "Surname",
  "Tags",
  "Timestamp",
  "Type"
]

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues
@jvalente-salemstate
jvalente-salemstate requested a review from a team as a code owner December 31, 2024 17:03
@andrewkroh andrewkroh added enhancement New feature or request Integration:m365_defender Microsoft Defender XDR Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 31, 2024
@elasticmachine

elasticmachine commented Dec 31, 2024

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
ShourieG
ShourieG reviewed Jan 2, 2025
View reviewed changes
Comment thread packages/m365_defender/data_stream/event/fields/fields.yml Outdated
@ShourieG

ShourieG commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

/test
2 similar comments
@ShourieG

ShourieG commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

/test
@kcreddy

kcreddy commented Jan 6, 2025

Copy link
Copy Markdown
Contributor

/test
@kcreddy

kcreddy commented Jan 13, 2025

Copy link
Copy Markdown
Contributor

/test
@ShourieG

ShourieG commented Jan 13, 2025

Copy link
Copy Markdown
Contributor

@jvalente-salemstate, can you run elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build once. This should standardise the doc changes across the integration which is causing a CI failure right now.
@jvalente-salemstate

jvalente-salemstate commented Jan 13, 2025

Copy link
Copy Markdown
Contributor Author

elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build

We should be good.

elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build
Build the package
README.md file rendered: ~/git/integrations/packages/m365_defender/docs/README.md
2025/01/13 14:17:14  INFO License text found in "~/git/integrations/LICENSE.txt" will be included in package
2025/01/13 14:17:15  INFO Skipped errors: found 3 validation errors:
   1. references found in dashboard kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json: m365_defender-fcf25960-44af-11ed-8375-0168a9970c06 (search), m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
   2. references found in dashboard kibana/dashboard/m365_defender-c0b796d0-720a-11ed-8657-c59f6ece834c.json: m365_defender-64a31410-722c-11ed-8657-c59f6ece834c (search) (SVR00004)
   3. references found in dashboard kibana/dashboard/m365_defender-d80d7840-4366-11ed-b1f2-e917f608bd03.json: m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
Package built: ~/git/integrations/build/packages/m365_defender-2.19.0.zip
Done
Format the package
Done
Lint the package
2025/01/13 14:17:15  INFO Skipped errors: found 3 validation errors:
   1. references found in dashboard kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json: m365_defender-fcf25960-44af-11ed-8375-0168a9970c06 (search), m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
   2. references found in dashboard kibana/dashboard/m365_defender-c0b796d0-720a-11ed-8657-c59f6ece834c.json: m365_defender-64a31410-722c-11ed-8657-c59f6ece834c (search) (SVR00004)
   3. references found in dashboard kibana/dashboard/m365_defender-d80d7840-4366-11ed-b1f2-e917f608bd03.json: m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
Done
Lint the package
2025/01/13 14:17:16  INFO Skipped errors: found 3 validation errors:
   1. references found in dashboard kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json: m365_defender-fcf25960-44af-11ed-8375-0168a9970c06 (search), m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
   2. references found in dashboard kibana/dashboard/m365_defender-c0b796d0-720a-11ed-8657-c59f6ece834c.json: m365_defender-64a31410-722c-11ed-8657-c59f6ece834c (search) (SVR00004)
   3. references found in dashboard kibana/dashboard/m365_defender-d80d7840-4366-11ed-b1f2-e917f608bd03.json: m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
Done
Build the package
README.md file rendered: ~/git/integrations/packages/m365_defender/docs/README.md
2025/01/13 14:17:16  INFO License text found in "~/git/integrations/LICENSE.txt" will be included in package
2025/01/13 14:17:16  INFO Skipped errors: found 3 validation errors:
   1. references found in dashboard kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json: m365_defender-fcf25960-44af-11ed-8375-0168a9970c06 (search), m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
   2. references found in dashboard kibana/dashboard/m365_defender-c0b796d0-720a-11ed-8657-c59f6ece834c.json: m365_defender-64a31410-722c-11ed-8657-c59f6ece834c (search) (SVR00004)
   3. references found in dashboard kibana/dashboard/m365_defender-d80d7840-4366-11ed-b1f2-e917f608bd03.json: m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
Package built: ~/git/integrations/build/packages/m365_defender-2.19.0.zip
Done
Build the package
README.md file rendered: ~/git/integrations/packages/m365_defender/docs/README.md
2025/01/13 14:17:16  INFO License text found in "~/git/integrations/LICENSE.txt" will be included in package
2025/01/13 14:17:17  INFO Skipped errors: found 3 validation errors:
   1. references found in dashboard kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json: m365_defender-fcf25960-44af-11ed-8375-0168a9970c06 (search), m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
   2. references found in dashboard kibana/dashboard/m365_defender-c0b796d0-720a-11ed-8657-c59f6ece834c.json: m365_defender-64a31410-722c-11ed-8657-c59f6ece834c (search) (SVR00004)
   3. references found in dashboard kibana/dashboard/m365_defender-d80d7840-4366-11ed-b1f2-e917f608bd03.json: m365_defender-989afc60-44a5-11ed-8375-0168a9970c06 (search) (SVR00004)
Package built: ~/git/integrations/build/packages/m365_defender-2.19.0.zip
Done

I was missing a few descriptions in fields.yml and had a duplicate.
@ShourieG

ShourieG commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

/test
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@ShourieG

ShourieG commented Jan 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

@jvalente-salemstate, it looks good as of now, but can we add some more tests so we can have the test coverage up for SonarQube?
@jvalente-salemstate

jvalente-salemstate commented Jan 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@jvalente-salemstate, it looks good as of now, but can we add some more tests so we can have the test coverage up for SonarQube?

I could try, but I can't access SonarQube to see which tests or the results. Is it the null fields in the two samples I added?

I can (even if that isn't the issue) add another sample with a few more populated since I found a couple that did have more info.
@ShourieG

ShourieG commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

/test
@elastic-sonarqube

elastic-sonarqube Bot commented Jan 15, 2025

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
69.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@elasticmachine

elasticmachine commented Jan 15, 2025

Copy link
Copy Markdown

💚 Build Succeeded

History

  • 💚 Build #20377 succeeded a57836302d59400481a7d4c5150fa88dfd557169
  • 💔 Build #20331 failed a9fe85b9fe7755c595b0d886b70606a96c9d5715
  • 💔 Build #20054 failed bf267a202f9e865b6c9be39511cee1d028954852
  • 💔 Build #19960 failed 66cdfc56d06b797fb32ab6e96f8ffcfde1861935
  • 💔 Build #19959 failed 66cdfc56d06b797fb32ab6e96f8ffcfde1861935
ShourieG
ShourieG approved these changes Jan 15, 2025
View reviewed changes

@ShourieG ShourieG left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,test coverage issues seems to be an existing problem from before
@ShourieG
ShourieG merged commit f7dace9 into elastic:main Jan 15, 2025
@kcreddy

kcreddy commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

I could try, but I can't access SonarQube to see which tests or the results. Is it the null fields in the two samples I added?

I can (even if that isn't the issue) add another sample with a few more populated since I found a couple that did have more info.

It seems to be failing mainly on fields.yml among the newly committed code.
Screenshot 2025-01-15 at 2 30 45 PM

As per elastic/elastic-package#1915 (comment), the fields files are checked when CI system tests are run for the package. But we don't have system tests for this event data-stream as it is based on azure-eventhub input. I don't think we can do much other than adding system tests for event data-stream.
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

Package m365_defender - 2.19.0 containing this change is available at https://epr.elastic.co/package/m365_defender/2.19.0/
@jvalente-salemstate
jvalente-salemstate deleted the m365_defender-add-identityinfo branch January 15, 2025 14:14
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@jvalente-salemstate
[M365_Defender] Enhancement - Add support for IdentityInfo Table (ela…
8067da1 
…stic#12214)
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@jvalente-salemstate
[M365_Defender] Enhancement - Add support for IdentityInfo Table (ela…
9aff7b7 
…stic#12214)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:m365_defender Microsoft Defender XDR Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

5 participants

@jvalente-salemstate @elasticmachine @ShourieG @kcreddy @andrewkroh