entityanalytics_ad: add support for empty Active Directory groups by efd6 · Pull Request #18235 · elastic/integrations

efd6 · 2026-04-07T04:30:06Z

Proposed commit message

entityanalytics_ad: add group data stream and pipeline

The beats entity-analytics input already collects all Active Directory
groups for membership enrichment; their names and SIDs appear within
user and device entities via the user.group fields. Groups with no
direct members are invisible to this process because they have no
member links to follow.

The include_empty_groups option (elastic/beats#49093) makes the input
emit standalone documents for these otherwise-invisible groups so they
appear in the asset inventory. This change adds the integration-side
support: a group ingest pipeline, a routing rule to direct group
documents to a dedicated data stream, and ECS and vendor field
definitions.

The entity pipeline dispatch is updated to recognise group documents
(ctx.group.id != null) and the fallback entity pipeline condition
now excludes them. Pipeline tags are corrected to distinguish user,
device, and group dispatch.

Also adds system testing, including document routing tests (second commit).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Active Directory Entity Analytics]: Add Groups and Group Policies Datasets #16519

Screenshots

github-actions · 2026-04-07T04:31:52Z

Vale Linting Results

Summary: 4 warnings found

⚠️ Warnings (4)

File	Line	Rule	Message
packages/entityanalytics_ad/docs/README.md	383	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/entityanalytics_ad/docs/README.md	385	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.
packages/entityanalytics_ad/docs/README.md	409	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/entityanalytics_ad/docs/README.md	410	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

github-actions · 2026-05-06T02:21:16Z

TL;DR

The Buildkite failure is from the Check go sources step detecting a dirty tree after mage check; specifically packages/entityanalytics_ad/_dev/deploy/docker/ldap-mock-service/main.go is rewritten during CI but that update was not committed.

Remediation

From the PR branch, run mage check at repo root and commit any resulting changes (at minimum packages/entityanalytics_ad/_dev/deploy/docker/ldap-mock-service/main.go).
Re-run the same CI check (.buildkite/scripts/check_sources.sh) or push the commit and confirm the Check go sources step is clean.

Investigation details

Root Cause

check_sources.sh runs mage -v check and then enforces a clean working tree via check_git_diff (.buildkite/scripts/check_sources.sh#L11-L14). In this build, that post-check diff check fails because the repository becomes dirty with:

packages/entityanalytics_ad/_dev/deploy/docker/ldap-mock-service/main.go: needs update

That indicates source normalization/generation expected by mage check was not fully committed in the PR.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/42376
Job/step: :white_check_mark: Check go sources (.buildkite/scripts/check_sources.sh)
Key log excerpt:

--- Check if any files modified
git update-index
packages/entityanalytics_ad/_dev/deploy/docker/ldap-mock-service/main.go: needs update
🚨 Error: The command exited with status 1

Verification

Not run locally in this workflow (read-only detective pass); conclusion is based on Buildkite logs and repository check script behavior.

Follow-up

If mage check produces unexpected edits repeatedly, capture git diff -- packages/entityanalytics_ad/_dev/deploy/docker/ldap-mock-service/main.go from the PR branch and include it in the next CI run for pinpointing the exact formatter/generator rule.

Note

🔒 Integrity filter blocked 3 items

The following items were blocked because they don't meet the GitHub integrity level.

entityanalytics_ad: add support for empty Active Directory groups #18235 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#18235 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#18235 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

elastic-vault-github-plugin-prod · 2026-05-06T06:47:04Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-05-06T06:57:54Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

chrisberkhout

The commit description and the new group pipeline description both mention empty groups specifically.

I see the new option in the input config to include empty groups. Do we get information about non-empty groups? If so, is that existing data that's getting new support (pipeline and data stream) or is that data new?

Can you please rewrite the commit description and pipeline description to clarify the difference between existing and new support for groups, and empty vs non-empty groups.

efd6 · 2026-05-20T21:58:24Z

Can you please rewrite the commit description and pipeline description to clarify the difference between existing and new support for groups, and empty vs non-empty groups.

Done. PTAL

chrisberkhout · 2026-05-21T13:25:58Z

Sorry for the closer/reopen. I hit the wrong button.

chrisberkhout

Can you please rewrite the commit description and pipeline description to clarify the difference between existing and new support for groups, and empty vs non-empty groups.

Done. PTAL

Thanks! That's much clearer.

Another possibility would be to have the empty group docs in the user data stream, just without the user fields, so they line up with existing group data. But keeping them separate probably is better, and it's well explained in the README, etc.

It's separate from this change, but the LLM caught a point where the proposed commit message differs from the implementation. The proposed commit message says:

their names and SIDs appear within user and device entities via the user.group and device.group fields

However, for devices, they're put into user.group, and that's what we see in the existing pipeline test output.

Wire the include_empty_groups option from the beats input into the integration package. This adds a group ingest pipeline, a routing rule to direct group documents to a dedicated data stream, ECS and vendor field definitions for groups, and a new group data stream with its own manifest and field files. The entity pipeline dispatch is updated to recognise group documents (ctx.group.id != null) and the fallback entity pipeline condition now excludes them. Pipeline tags are corrected to distinguish user, device, and group dispatch.

Add a gldap-based LDAP mock server that serves synthetic user, device, and group entries for local and CI testing without a real Active Directory server. Add a script test (routed_data_streams.txt) that verifies documents are routed from the entity data stream to the user, device, and group destination data streams. This covers a gap in the standard system test runner which cannot assert on routed data streams. Also add a pipeline test for the group ingest pipeline and a system test config wired to the LDAP mock.

elasticmachine · 2026-05-21T22:58:21Z

💚 Build Succeeded

Buildkite Build
Commit: d2027b1

History

💚 Build #43182 succeeded feef661
💚 Build #42387 succeeded 322c534
💔 Build #42384 failed 326ee8f
💔 Build #42376 failed d163448
💔 Build #40902 failed 0ef99ef
💔 Build #40899 failed 470489a

cc @efd6

elastic-vault-github-plugin-prod · 2026-05-22T08:43:19Z

Package entityanalytics_ad - 0.19.0 containing this change is available at https://epr.elastic.co/package/entityanalytics_ad/0.19.0/

The beats entity-analytics input already collects all Active Directory groups for membership enrichment; their names and SIDs appear within user and device entities via the user.group fields. Groups with no direct members are invisible to this process because they have no member links to follow. The include_empty_groups option (elastic/beats#49093) makes the input emit standalone documents for these otherwise-invisible groups so they appear in the asset inventory. This change adds the integration-side support: a group ingest pipeline, a routing rule to direct group documents to a dedicated data stream, and ECS and vendor field definitions. The entity pipeline dispatch is updated to recognise group documents (ctx.group.id != null) and the fallback entity pipeline condition now excludes them. Pipeline tags are corrected to distinguish user, device, and group dispatch.

efd6 self-assigned this Apr 7, 2026

efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:entityanalytics_ad Active Directory Entity Analytics labels Apr 7, 2026

efd6 force-pushed the 16519-entityanalitics_ad branch from 0719417 to 470489a Compare April 7, 2026 04:30

efd6 force-pushed the 16519-entityanalitics_ad branch from 470489a to 0ef99ef Compare April 7, 2026 04:39

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Apr 7, 2026

efd6 force-pushed the 16519-entityanalitics_ad branch from 0ef99ef to d163448 Compare May 6, 2026 02:08

This comment has been minimized.

Sign in to view

efd6 force-pushed the 16519-entityanalitics_ad branch 2 times, most recently from 326ee8f to 322c534 Compare May 6, 2026 06:23

efd6 marked this pull request as ready for review May 6, 2026 06:57

efd6 requested a review from a team as a code owner May 6, 2026 06:57

efd6 mentioned this pull request May 19, 2026

Allow data stream overrides in system tests for specific integration packages elastic/elastic-package#1917

Open

chrisberkhout requested changes May 20, 2026

View reviewed changes

chrisberkhout reviewed May 20, 2026

View reviewed changes

Comment thread packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/env.txt

efd6 force-pushed the 16519-entityanalitics_ad branch from 322c534 to feef661 Compare May 20, 2026 21:58

efd6 requested a review from a team as a code owner May 20, 2026 21:58

efd6 requested a review from chrisberkhout May 20, 2026 21:58

chrisberkhout closed this May 21, 2026

chrisberkhout reopened this May 21, 2026

chrisberkhout requested changes May 21, 2026

View reviewed changes

efd6 added 2 commits May 22, 2026 07:58

efd6 force-pushed the 16519-entityanalitics_ad branch from feef661 to d2027b1 Compare May 21, 2026 22:33

efd6 requested a review from chrisberkhout May 21, 2026 22:33

chrisberkhout approved these changes May 22, 2026

View reviewed changes

efd6 merged commit f7d9727 into elastic:main May 22, 2026
11 checks passed

This was referenced May 22, 2026

[bug-hunter] entityanalytics_ad group events are mislabeled as activedirectory_user #19170

Closed

[breaking-change] Undocumented breaking changes detected (2026-05-22) #19172

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

entityanalytics_ad: add support for empty Active Directory groups#18235

entityanalytics_ad: add support for empty Active Directory groups#18235
efd6 merged 2 commits into
elastic:mainfrom
efd6:16519-entityanalitics_ad

efd6 commented Apr 7, 2026 •

edited

Loading

github-actions Bot commented Apr 7, 2026 •

edited

Loading

This comment has been minimized.

github-actions Bot commented May 6, 2026

Root Cause

Evidence

Verification

Follow-up

elastic-vault-github-plugin-prod Bot commented May 6, 2026

infra-vault-gh-plugin-prod Bot commented May 6, 2026

chrisberkhout left a comment

Uh oh!

efd6 commented May 20, 2026 •

edited

Loading

chrisberkhout commented May 21, 2026

chrisberkhout left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented May 21, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 22, 2026

Labels

4 participants

Uh oh!

Conversation

efd6 commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

github-actions Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vale Linting Results

This comment has been minimized.

github-actions Bot commented May 6, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

elastic-vault-github-plugin-prod Bot commented May 6, 2026

🚀 Benchmarks report

infra-vault-gh-plugin-prod Bot commented May 6, 2026

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

chrisberkhout commented May 21, 2026

chrisberkhout left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented May 21, 2026

💚 Build Succeeded

History

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 22, 2026

Labels

4 participants

efd6 commented Apr 7, 2026 •

edited

Loading

github-actions Bot commented Apr 7, 2026 •

edited

Loading

efd6 commented May 20, 2026 •

edited

Loading

chrisberkhout left a comment •

edited

Loading