Entra ID Entity Analytics (minimal-state) aborts every sync: build membership graph: add group : key required

Summary

The minimal-state Entra ID provider (entcollect/entraid, used by the
entityanalytics_entra_id integration) lists groups with a $select that omits
id. Microsoft Graph honours $select strictly, so every group returns with an
empty id. The membership graph is stored in a bbolt scratch DB keyed by the
group id; an empty key is rejected with key required, and buildGraph aborts
the entire sync — users and devices included — before anything is published.

Result: events_pipeline_published_total stays at 0. No user or device
entities are ever ingested.

This is a regression: it only affects the minimal-state path, which the
integration now enables by default (including for existing policies on upgrade).

Affected versions

• Beats: 9.5.0 and 9.5.1 (both confirmed from diagnostics).
• Beats vendors github.com/elastic/entcollect v0.0.0-20260720203654-e61fb8788d9a
  (commit e61fb8788d9a) — see go.mod:236.
• Integration entityanalytics_entra_id: 1.13.0 and 1.14.0 (both confirmed).
  The default flip landed in integrations PR
  #20212 (package 1.13.0).

Symptoms

• Integration/component status is HEALTHY (the input process runs and is
  connected to Elasticsearch).

• Graph auth succeeds; users are fetched (~164) and, with dataset: all,
  devices too (~90).

• No fetched groups log line appears; the error fires ~100–130 ms after
  fetched users.

• Every full and incremental sync fails with:

  Error running full sync / Error running incremental sync
  error.message: entraid: build membership graph: add group : key required
  

• events_pipeline_published_total = 0.

• Switching dataset: all → users does not help — buildGraph runs regardless
  of dataset.

Root cause

All links pinned to entcollect@e61fb8788d9a.

1. The group projection omits id.
   provider/entraid/conf.go#L54:

   defaultSelectGroups = []string{"displayName", "members"}

   Groups are fetched from the non-delta /groups list endpoint, with the
   select passed through verbatim
   (api.go#L172-L177):

   startURL := g.baseURL + "/groups"
   if len(selectFields) > 0 {
       startURL += "?$select=" + strings.Join(selectFields, ",")
   }

   → GET /groups?$select=displayName,members. Per Microsoft Graph, $select
   returns only the requested properties, so Group.ID is empty. (members is a
   navigation property — ignored on a list query without $expand; members are
   fetched separately via getGroupMembers.)

2. The empty id aborts the whole sync.
   buildGraph stores each group keyed by g.ID
   (provider.go#L386-L412):

   mg, err := newScratchBackend(p.cfg.ScratchDir)
   ...
   for _, g := range groups {
       if err := mg.addGroup(g); err != nil {
           mg.close()
           return nil, fmt.Errorf("add group %s: %w", g.ID, err) // g.ID=="" → "add group : ..."
       }

   addGroup writes to bbolt using the group id as the key
   (scratch_backend.go#L35-L37):

   func (mg *scratchBackend) addGroup(g Group) error {
       return mg.db.PutJSON(bucketGroups, g.ID, g)
   }

   bbolt rejects an empty key with ErrKeyRequired ("key required"). The error is
   wrapped in both FullSync and IncrementalSync as
   entraid: build membership graph: %w
   (provider.go#L102,
   #L271),
   producing exactly the observed message. Because buildGraph returns on the
   first failure, nothing is published.

3. Why users and devices are fine but groups fail.
   Users and devices use the delta endpoints (/users/delta,
   /devices/delta), whose UnmarshalJSON extracts id from the raw payload
   regardless of $select — delta always returns id for change tracking
   (api.go#L89-L104).
   Only groups use the plain non-delta list, which strictly honours $select.

This affects agentless too

The membership graph is always built with the bbolt scratch backend —
buildGraph calls newScratchBackend(p.cfg.ScratchDir) unconditionally, with no
agent/agentless branch. mapBackend (in membership.go) is explicitly a
test-only double and is never used in production. The manifest's "ES-backed state
/ reduced local storage" refers to the persistent cursor/entity store
(entcollect.Store), not the per-sync membership scratch graph. So the
key required failure occurs in every deployment mode.

Edge case: a tenant with zero groups skips the loop and syncs fine; the failure
requires at least one group.

Required fix

elastic/entcollect (provider/entraid)

1. Correctness — request id and drop the useless members navigation
   property in conf.go:

   defaultSelectGroups = []string{"id", "displayName"}

2. Resilience — in buildGraph, skip and log groups with an empty id
   instead of aborting the whole sync, so a single malformed entry can never
   again zero out the entire pull:

   for _, g := range groups {
       if g.ID == "" {
           log.Warn("skipping group with empty id", "displayName", g.DisplayName)
           continue
       }
       ...
   }

3. Close the test gap — the in-memory mapBackend.addGroup
   (membership.go) accepts an empty key while the production scratchBackend
   rejects it, which is why CI missed this. Make the map double reject empty keys
   so tests mirror bbolt semantics, and add a buildGraph unit test with an
   id-less group asserting it is skipped rather than fatal.

4. Optional hardening — also add id to defaultSelectUsers and
   defaultSelectDevices (currently masked only because the delta endpoints
   always return id).

elastic/beats

• Bump the entcollect dependency in go.mod (currently
  v0.0.0-20260720203654-e61fb8788d9a) to the commit containing the fix.
• Add a changelog fragment describing the bugfix.

Workaround (no new build)

Set the group $select via the integration's Custom Options:

select:
  groups:
    - id
    - displayName

This restores the group id, so addGroup succeeds and syncs publish again.
(use_minimal_state: false uses the legacy delta-groups path that always
returns id, but it is hidden in Fleet and not recommended.)

References

• Regression trigger: elastic/integrations#20212 (package 1.13.0)
• Latent defect introduced: elastic/entcollect#9 (entraid provider added)