cisco_duo: add user data stream and latest transform - #20768
Conversation
efd6
added
enhancement
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
activity |
4149.38 | 3215.43 | -933.95 (-22.51%) | 💔 |
auth |
2652.52 | 2141.33 | -511.19 (-19.27%) | 💔 |
summary |
50000 | 28571.43 | -21428.57 (-42.86%) | 💔 |
telephony |
58823.53 | 41666.67 | -17156.86 (-29.17%) | 💔 |
To see the full report comment with /test benchmark fullreport
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| delay: 120s | ||
| retention_policy: | ||
| time: | ||
| field: "@timestamp" |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/cisco_duo/elasticsearch/transform/latest_user/transform.yml:31
The retention policy expires destination documents on @timestamp, but the pipeline sets @timestamp to the user's account-creation date, so every user created more than 30 days ago is deleted from the latest index. Retain on event.ingested instead, or drop the retention policy.
Details
The new ingest pipeline's first date processor (packages/cisco_duo/data_stream/user/elasticsearch/ingest_pipeline/default.yml, tag date_created) parses json.created with no target_field, so it overwrites @timestamp with the Duo account-creation time. The committed fixture confirms this: in test-user.log-expected.json @timestamp equals cisco_duo.user.created (2021-07-15T12:27:34.000Z) for both documents.
The retention policy here deletes destination documents whose @timestamp is older than 30 days. Because @timestamp is the account-creation date rather than an observation time, it never advances as the user is re-collected, so any account older than 30 days is written by the transform and then immediately purged. For a typical tenant the logs-cisco_duo_latest.user alias would be empty or contain only recently created accounts, which contradicts the transform's stated purpose ("Retains only the most recent record per user for up-to-date correlation").
Retaining on event.ingested expresses the intended semantics — drop users that have not been seen from the API for 30 days — and matches the field already used for sync and latest.sort. It requires the event.ingested declaration described in finding 2. Alternatively, remove retention_policy entirely, or stop overwriting @timestamp in the pipeline so it reflects collection time.
Recommendation:
Expire on the field that tracks when the record was last seen:
retention_policy:
time:
field: "event.ingested"
max_age: 30dThis depends on event.ingested being declared in the transform's field definitions (see the related finding on fields/ecs.yml), otherwise the destination index maps it as a keyword and the retention policy cannot use it.
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - external: ecs | ||
| name: user.name |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/cisco_duo/elasticsearch/transform/latest_user/fields/ecs.yml:24
event.ingested is used as the destination index sort field and is written into every destination document, but it is not declared in the transform's field definitions, so the generated index template has no mapping for it. Add an external: ecs entry for event.ingested.
Details
elasticsearch/transform/latest_user/manifest.yml sets index.sort.field: ["event.ingested"] on the destination index template, and transform.yml uses event.ingested for both latest.sort and sync.time.field. None of the five files under elasticsearch/transform/latest_user/fields/ declares event.ingested, so the destination index template carries no mapping for it.
Two consequences: Elasticsearch resolves index.sort.* against the mappings present at index-creation time and rejects creation with unknown index sort field:[event.ingested] when the field is unmapped — dynamic templates do not satisfy this because they only apply as documents are indexed. And even setting the sort aside, a latest transform copies the whole source document, so event.ingested lands in the destination index and, with date_detection: false plus the strings_as_keyword dynamic template, is mapped as keyword — conflicting with its date mapping in logs-cisco_duo.user-*.
Every comparable transform in the repo that references event.ingested declares it explicitly (for example packages/ti_misp/elasticsearch/transform/latest_ioc/fields/ecs.yml and packages/ti_opencti/elasticsearch/transform/latest_ioc/fields/ecs.yml).
Recommendation:
Declare event.ingested alongside the other external ECS fields so the destination index template maps it as a date before the index is created.
| - external: ecs | |
| name: user.name | |
| - external: ecs | |
| name: user.name | |
| - external: ecs | |
| name: event.ingested |
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ignore_missing: true | ||
| if: ctx.event?.original != null | ||
| description: 'The `message` field is no longer required if the document has an `event.original` field.' | ||
| - json: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/cisco_duo/data_stream/user/elasticsearch/ingest_pipeline/default.yml:43
A document produced by the CEL program's error branch has neither message nor event.original, so this json processor fails and the collector error is re-reported as a generic pipeline error. Add a terminate processor ahead of it to short-circuit collector-error documents.
Details
The CEL program in agent/stream/cel.yml.hbs returns the single-object error shape {"events": {"error": {...}}} on any non-200 response. That produces a document carrying error.message (and error.code/error.id) with no message field, so rename_event_original is a no-op and this json processor runs against a missing event.original without ignore_missing. It throws, the pipeline-level on_failure fires, and the document is indexed with event.kind: pipeline_error plus a second appended error.message — masking the original HTTP status and message the collector reported.
This data stream is new and the package enables agentless deployment (policy_templates[0].deployment_modes.agentless.enabled: true), so the CEL-opening error guard applies here. The terminate processor is available on the stack range allowed by conditions.kibana.version (^8.18.0 || ^9.0.0).
Recommendation:
Stop processing collector-error documents before any parsing runs (place it with the other opening processors, ahead of rename_event_original):
- terminate:
tag: terminate_collector_error
if: ctx.error?.message != null && ctx.message == null
description: >-
Stop processing documents that report a CEL collector error
and carry no payload to parse.🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| bool: | ||
| must_not: | ||
| - term: | ||
| event.kind: pipeline_error |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/cisco_duo/elasticsearch/transform/latest_user/transform.yml:8
The transform source query only excludes event.kind: pipeline_error, which misses documents whose failure came from a per-processor on_failure, and it scans cold/frozen tiers. Add error.message and _tier exclusions.
Details
The date_created processor in the new pipeline has its own on_failure that appends to error.message without setting event.kind: pipeline_error, so a user record with an unparseable created value keeps event.kind: state and passes this filter into the latest index. Excluding documents that have error.message catches both that case and the pipeline-level failures already covered.
Separately, with frequency: 1m and continuous sync this transform repeatedly queries logs-cisco_duo.user-* with no tier restriction, so once older backing indices roll into the cold or frozen tier the checkpoint searches hit them.
Recommendation:
Filter out error documents and expensive tiers in the source query:
source:
index:
- "logs-cisco_duo.user-*"
query:
bool:
must_not:
- term:
event.kind: pipeline_error
- exists:
field: error.message
- terms:
_tier:
- data_frozen
- data_cold🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - input: cel | ||
| enabled: false | ||
| vars: | ||
| - name: limit |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/cisco_duo/data_stream/user/manifest.yml:7
The new user data stream declares no interval variable of its own, so it inherits the shared policy-template default of 1m and re-fetches the entire Duo user directory every 60 seconds. Add a stream-level interval var with an inventory-appropriate default such as 24h.
Details
The cel input in the root manifest defines interval with default: 1m and the description "Not recommended requesting logs more than once per minute" (packages/cisco_duo/manifest.yml:130-137). That default is sized for the incremental event-log streams (activity, auth, telephony, trust_monitor), which page a bounded time window each run.
This stream's template reuses the same value (interval: {{interval}} in data_stream/user/agent/stream/cel.yml.hbs:2), but the program is a full-inventory pull, not an incremental one: whenever want_more is false it resets offset to "0" and pages through every user again (cel.yml.hbs:16-24). Because the stream manifest declares no interval var, an operator cannot slow it down without also slowing every other Cisco Duo stream.
Concretely, with the shipped defaults limit: 100 and rate_limit: "0.5" (0.5 req/s = 30 requests/min), a tenant with more than ~3000 users cannot finish a single pass inside a 1-minute interval, so the input polls the Duo Admin API back-to-back continuously and writes a full copy of the user directory into logs-cisco_duo.user-* on every pass. The latest transform deduplicates the destination index but does nothing to reduce the source-index volume or the API load.
Comparable user-inventory data streams declare their own interval: packages/island_browser/data_stream/user/manifest.yml:11-18 uses default: 24h while that package's event-log streams use 30m/1h.
Recommendation:
Declare an interval var on the user stream. A stream-level var shadows the policy-template input var of the same name, so this lets the inventory poll run on its own schedule while the event-log streams keep their 1m default:
vars:
- name: interval
type: text
title: Interval
description: "Duration between requests to the Cisco Duo Admin API for the full user inventory. NOTE: Supported units for this parameter are h/m/s."
multi: false
required: true
show_user: true
default: 24h
- name: limit
type: integer
title: Limit
description: Maximum number of records to fetch on each request. Max is 300.
show_user: false
required: true
default: 100🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: event.original | ||
| target_field: json | ||
| tag: json_event_original | ||
| - date: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/cisco_duo/data_stream/user/elasticsearch/ingest_pipeline/default.yml:53
The date_created processor has no target_field, so it overwrites @timestamp with the user's account-creation date and inventory documents land years in the past; drop it and keep the guarded date_created_field processor that already writes cisco_duo.user.created.
Details
Two problems with this processor.
-
No
target_fieldmeans thedateprocessor writes@timestamp. The generated fixtures show the effect: sample_event.json has@timestamp: 2021-05-01T00:00:00.000Zagainstevent.ingested: 2026-08-18T04:07:18Z. For anevent.kind: stateinventory stream the collection time is the meaningful timestamp — a directory of accounts created over many years is scattered across the time axis, so a default Discover or dashboard time filter shows little or nothing, and the freshness of the inventory is invisible. Comparable state/inventory data streams only date-convert into their own namespaced field and leave@timestampat the agent collection time (packages/lastpass/data_stream/user/elasticsearch/ingest_pipeline/default.yml:30-32, packages/kolide/data_stream/people/elasticsearch/ingest_pipeline/default.yml:144-150). -
Unlike its sibling on the very next processor (line 64, which carries
if: ctx.json?.created != null), this one is unguarded. A user record withoutcreatedtherefore fails it and appends toerror.message. The latest transform's source query excludes every document witherror.messagepresent (packages/cisco_duo/elasticsearch/transform/latest_user/transform.yml:9-11), so such a user would silently never reachlogs-cisco_duo_latest.user.
Removing this processor fixes both: cisco_duo.user.created is still populated by the guarded processor below it.
Recommendation:
Delete the unguarded processor so @timestamp stays at the collection time, leaving only the guarded date conversions:
- json:
field: event.original
target_field: json
tag: json_event_original
- date:
field: json.created
target_field: cisco_duo.user.created
tag: date_created_field
formats:
- UNIX
if: ctx.json?.created != null
- date:
field: json.last_login
target_field: cisco_duo.user.last_login
tag: date_last_login
formats:
- UNIX
if: ctx.json?.last_login != nullNote that test-user.log-expected.json and sample_event.json will need regenerating, since both currently encode the account-creation date as @timestamp.
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - set: | ||
| field: event.kind | ||
| tag: set_event_kind | ||
| value: state | ||
| - set: | ||
| field: event.category | ||
| tag: set_event_category | ||
| value: ["iam"] | ||
| - set: | ||
| field: event.type | ||
| tag: set_event_type | ||
| value: ["user", "info"] | ||
| - terminate: | ||
| tag: terminate_collector_error | ||
| if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null | ||
| description: >- | ||
| Stop processing documents that report a CEL collector error | ||
| and carry no payload to parse. |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/cisco_duo/data_stream/user/elasticsearch/ingest_pipeline/default.yml:36
terminate_collector_error runs after event.kind, event.category and event.type are set, so CEL collector-error documents get indexed as state/iam/user,info events; move the terminate ahead of those three set processors.
Details
A CEL collector error is published as a document carrying only error.* and no message, which is exactly what the terminate at line 31 is there to stop. But the three set processors at lines 19-30 have already run by then, so the error document is indexed with event.kind: state, event.category: ["iam"] and event.type: ["user", "info"] — indistinguishable from a real Duo user record in IAM dashboards, counts and detection rules that filter on those fields. Reordering costs nothing and does not change the pipeline test output, since the test documents all carry message and never trigger the terminate.
Recommendation:
Move the terminate block above the three event.* set processors so collector-error documents stop before any event categorisation is applied.
| - set: | |
| field: event.kind | |
| tag: set_event_kind | |
| value: state | |
| - set: | |
| field: event.category | |
| tag: set_event_category | |
| value: ["iam"] | |
| - set: | |
| field: event.type | |
| tag: set_event_type | |
| value: ["user", "info"] | |
| - terminate: | |
| tag: terminate_collector_error | |
| if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null | |
| description: >- | |
| Stop processing documents that report a CEL collector error | |
| and carry no payload to parse. | |
| - terminate: | |
| tag: terminate_collector_error | |
| if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null | |
| description: >- | |
| Stop processing documents that report a CEL collector error | |
| and carry no payload to parse. | |
| - set: | |
| field: event.kind | |
| tag: set_event_kind | |
| value: state | |
| - set: | |
| field: event.category | |
| tag: set_event_category | |
| value: ["iam"] | |
| - set: | |
| field: event.type | |
| tag: set_event_type | |
| value: ["user", "info"] |
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,72 @@ | |||
| { | |||
| "@timestamp": "2021-05-01T00:00:00.000Z", | |||
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/cisco_duo/data_stream/user/sample_event.json:2
The sample event was generated before the date_created fix, so its @timestamp is the account-creation date (2021-05-01) while event.ingested is 2026-08-18; regenerate sample_event.json from a fresh system test so @timestamp reflects collection time.
Details
The pipeline no longer writes @timestamp at all: date_created_field now targets cisco_duo.user.created (default.yml:53-59), and nothing else sets @timestamp, so the agent's collection time is what lands in the document. The pipeline test's expected file confirms this, as it contains no @timestamp key.
This sample event, however, has @timestamp exactly equal to cisco_duo.user.created (2021-05-01T00:00:00.000Z, from created: 1619827200) while event.ingested is 2026-08-18T04:07:18Z. A five-year gap between collection time and ingest time is not producible by the current pipeline, so this file was generated against the earlier revision that had the unguarded date processor (see prior review finding on default.yml) and was not regenerated after that processor was fixed.
This matters beyond tidiness: packages/cisco_duo/docs/README.md embeds this JSON verbatim as the documented example for the user dataset, so the published docs show a timestamp the integration cannot produce. It also makes the sample useless for sanity-checking the transform, whose latest.sort is @timestamp.
Recommendation:
Re-run the system test for this data stream and commit the regenerated sample_event.json, then rebuild the docs so packages/cisco_duo/docs/README.md picks up the new example. After regeneration @timestamp should sit alongside event.ingested rather than matching cisco_duo.user.created:
{
"@timestamp": "2026-08-18T04:07:15.123Z",
"cisco_duo": {
"user": {
"created": "2021-05-01T00:00:00.000Z",
"last_login": "2021-07-22T11:26:40.000Z"
}
},
"event": {
"ingested": "2026-08-18T04:07:18Z"
}
}🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| type: constant_keyword | ||
| description: Data stream dataset. | ||
| - name: data_stream.namespace | ||
| type: constant_keyword |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/cisco_duo/elasticsearch/transform/latest_user/fields/base-fields.yml:8
The transform destination maps data_stream.namespace as a valueless constant_keyword, so once the first document pins it to one namespace, documents collected under any other Fleet namespace are rejected; declare it as keyword instead.
Details
A constant_keyword field with no value in the mapping configures itself from the first indexed document and then rejects any document carrying a different value. That is safe in the source data stream, where each backing index belongs to exactly one namespace, but the transform's source pattern is logs-cisco_duo.user-* (transform.yml:3) and every namespace converges into the single destination index logs-cisco_duo_latest.dest_user-1.
data_stream.namespace is present in the source _source (visible in this PR's sample_event.json, where it is 58062), so the latest transform copies it through. As soon as the integration is configured on a second agent policy with a different namespace, those documents fail to index and the transform accumulates index failures rather than reflecting the full user inventory.
data_stream.type and data_stream.dataset are not affected: their values are constant across all source documents (logs and cisco_duo.user), so constant_keyword is correct for those two. Only namespace varies. Several packages that hit this already override the type explicitly, e.g. packages/aws/elasticsearch/transform/latest_cdr_misconfigurations/fields/base-fields.yml.
Recommendation:
Map the namespace as a plain keyword in the transform's destination field definitions so documents from any namespace are accepted:
- name: data_stream.namespace
type: keyword
description: Data stream namespace.🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| {{fields "trust_monitor"}} | ||
|
|
||
| ### User |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/cisco_duo/_dev/build/docs/README.md:120
The new user dataset section is added to the docs, but the "Data streams" bullet list further up the same file still lists only the eight pre-existing datasets; add a Retrieve Users entry there too.
Details
_dev/build/docs/README.md has a "Data streams" section (lines 26-37) enumerating every event type the integration collects, each linking to the corresponding Cisco Duo Admin API doc anchor. This PR appends a ### User section at the bottom of the file but leaves that list untouched, so a reader scanning the overview will not learn that user inventory collection exists. Every other data stream in the package is represented there, so the omission reads as an oversight rather than a deliberate choice.
The generated packages/cisco_duo/docs/README.md inherits the same gap.
Recommendation:
Add the user endpoint to the "Data streams" list alongside the existing entries, then rebuild the docs:
- [**Telephony Logs (legacy)**](https://duo.com/docs/adminapi#telephony-logs-(legacy-v1))
- [**Trust Monitor**](https://duo.com/docs/adminapi#trust-monitor)
- [**Users**](https://duo.com/docs/adminapi#retrieve-users)🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - [**Telephony Logs**](https://duo.com/docs/adminapi#telephony-logs) | ||
| - [**Telephony Logs (legacy)**](https://duo.com/docs/adminapi#telephony-logs-(legacy-v1)) | ||
| - [**Trust Monitor**](https://duo.com/docs/adminapi#trust-monitor) | ||
| - [**Users**](https://duo.com/docs/adminapi#retrieve-users) |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/cisco_duo/_dev/build/docs/README.md:38
The new user data stream calls GET /admin/v1/users, which Duo gates behind the "Grant resource - Read" Admin API permission, but the Requirements section still lists only "Grant read information" and "Grant read log" - add the resource-read permission to that list.
Details
The Requirements section of this template (line 21, unchanged by this PR) tells administrators: "For this integration you will require Grant read information and Grant read log permissions." Duo's Admin API reference states for Retrieve Users: 'Requires "Grant resource - Read" API permission.' That permission covers "resource objects such as end users, policies, and devices" and is a separate checkbox on the Admin API application from the log/information grants the package uses today - it is not implied by the administrator's Owner role.
An Admin API application provisioned exactly as the README describes therefore returns HTTP 403 with body {"stat": "FAIL", "code": 40301, "message": "Access forbidden"} for this endpoint. The CEL program's non-200 branch converts that into a collector-error document rather than user records, and the ingest pipeline's terminate_collector_error processor stops it, so the new data stream silently produces nothing for anyone who followed the documented setup.
Fix this in the _dev/build/docs/README.md template; docs/README.md is generated from it and must be regenerated.
Recommendation:
Extend the permissions bullet in the Requirements section so the resource-read grant is listed alongside the existing ones:
- For this integration you will require **Grant read information** and **Grant read log** permissions. The **Users** data stream additionally requires the **Grant resource - Read** permission, which is what authorizes `GET /admin/v1/users`.🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| list.removeIf(v -> v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)); | ||
| } | ||
| handleMap(ctx); | ||
| - remove: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/cisco_duo/data_stream/user/elasticsearch/ingest_pipeline/default.yml:207
The pipeline renames a fixed allowlist of user attributes and then removes the whole json object, so documented Duo user fields such as lockout_reason, last_directory_sync, external_id and u2ftokens are silently discarded - add renames (and matching field definitions) for the attributes worth keeping.
Details
The Custom fields block renames sixteen specific keys out of json, and this processor then deletes json outright. Anything the Duo Admin API returns that is not in that allowlist never reaches the document.
Duo documents these additional top-level user attributes that the allowlist omits: lockout_reason, last_directory_sync, external_id, directory_key, u2ftokens, desktop_authenticators, enable_auto_prompt, password_last_updated, date_of_birth, entra_federated_user_id, and alias1 through alias4.
lockout_reason is the most consequential loss: this PR's own field definition for cisco_duo.user.status documents the value "locked out", and Duo states the reason for that lockout is carried in lockout_reason. The data stream therefore surfaces the locked-out state while dropping the only field that explains it. last_directory_sync, external_id and directory_key are the fields needed to correlate Duo users with the upstream directory, and u2ftokens / desktop_authenticators are authenticator inventory - the core value of a user inventory data stream for a security integration.
This is a deliberate-looking allowlist, so if some of these are intentionally out of scope that is fine, but the security-relevant ones above are worth capturing while the mapping is being written rather than in a follow-up that needs another field-mapping change.
Recommendation:
Add renames for the attributes worth retaining before the remove: json step, for example:
- rename:
field: json.lockout_reason
target_field: cisco_duo.user.lockout_reason
ignore_missing: true
tag: rename_lockout_reason
- rename:
field: json.last_directory_sync
target_field: cisco_duo.user.last_directory_sync
ignore_missing: true
tag: rename_last_directory_sync
- rename:
field: json.external_id
target_field: cisco_duo.user.external_id
ignore_missing: true
tag: rename_external_id
- rename:
field: json.u2ftokens
target_field: cisco_duo.user.u2ftokens
ignore_missing: true
tag: rename_u2ftokensNote that last_directory_sync is a Unix timestamp, so it needs a date processor like created/last_login rather than a plain rename if it is mapped as a date. Add the corresponding entries to both copies of the field definitions - data_stream/user/fields/fields.yml and elasticsearch/transform/latest_user/fields/fields.yml - since they are kept byte-identical:
- name: lockout_reason
type: keyword
description: |
The reason the user is locked out, when status is `locked out`.
- name: external_id
type: keyword
description: |
The user's ID in an external directory.
- name: u2ftokens
type: flattened
description: |
List of U2F tokens associated with the user.🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| ### User | ||
|
|
||
| This is the `user` dataset. |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/cisco_duo/_dev/build/docs/README.md:123
This PR installs a latest transform but the README never mentions it, so users have no way to discover the logs-cisco_duo_latest.user alias or the transform privileges it needs - add the standard transform note the other transform-shipping packages use.
Details
The PR adds elasticsearch/transform/latest_user, which creates the destination index logs-cisco_duo_latest.dest_user-1 and the alias logs-cisco_duo_latest.user, and starts a managed transform (start: true, unattended: true). None of this is documented: the README describes the user dataset only, so a reader cannot tell that a transform is installed, that querying the deduplicated latest state means targeting the alias rather than logs-cisco_duo.user-*, or that installing the package now requires transform privileges.
This is the convention in the other packages that ship latest transforms - island_browser, axonius, extrahop, ironscales, ibm_qradar, jupiter_one, cyera, xm_cyber and the ti_* packages all carry the same note plus a transform-health troubleshooting step. Adding it here keeps cisco_duo consistent with them and gives users a place to look when the transform goes unhealthy.
Recommendation:
Add the standard note near the top of the template (the other packages place it just after the Data streams list), and regenerate docs/README.md:
This integration installs [Elastic latest transforms](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview). For more details, check the [Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup) setup and requirements.
The latest state of each Duo user is available through the `logs-cisco_duo_latest.user` alias.🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Adds a user data stream that periodically polls GET /admin/v1/users to collect the full Duo user inventory. Because the endpoint is snapshot-only (it has no time-range filtering), each poll retrieves all users from offset 0. A latest Elasticsearch transform (logs-cisco_duo_latest.user) keeps one current record per user ID, sorted by event.ingested, so consumers always see up-to-date user state rather than the historical append of every poll. Pipeline test inputs in _dev/test/pipeline/test-user.log and the mock server responses in _dev/deploy/docker/files/config.yml are synthetic data hand-crafted to match the response structure described in the Duo Admin API documentation: https://duo.com/docs/adminapi#users
| @@ -0,0 +1,2 @@ | |||
| {"user_id":"DUJXXXXXXXXXXXXXXXXX","username":"narroway","realname":"Narrow Way","email":"narroway@example.com","status":"active","created":1626352054,"last_login":1626953200,"notes":"","is_enrolled":true,"firstname":"Narrow","lastname":"Way","groups":[{"group_id":"DGXXXXXXXXXXXXXXXXXX","name":"Employees","status":"Active","desc":""}],"phones":[{"phone_id":"DPXXXXXXXXXXXXXXXXXX","number":"+15555551234","type":"mobile","platform":"Apple iOS","activated":true}],"tokens":[],"webauthncredentials":[],"custom_attributes":{},"aliases":{}} | |||
| {"user_id":"DUJYYYYYYYYYYYYYYYYY","username":"jsmith","realname":"Jane Smith","email":"jsmith@example.com","status":"disabled","created":1619827200,"is_enrolled":false,"groups":[],"phones":[],"tokens":[],"webauthncredentials":[],"custom_attributes":{},"aliases":{}} | |||
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/cisco_duo/data_stream/user/_dev/test/pipeline/test-user.log:2
The four attribute handlers added in the latest revision (lockout_reason, external_id, u2ftokens, last_directory_sync) are not exercised by either fixture document, so a wrong target_field or date format in them would pass CI unnoticed - add a third fixture user that carries those keys.
Details
Both documents in this fixture come from the same Duo sample user shape: user_id, username, realname, email, status, created, last_login, notes, is_enrolled, firstname, lastname, groups, phones, tokens, webauthncredentials, custom_attributes, aliases. None of them contains lockout_reason, external_id, u2ftokens or last_directory_sync, so rename_lockout_reason, rename_external_id, rename_u2ftokens and date_last_directory_sync in default.yml never execute during elastic-package test pipeline, and the matching field definitions in fields/fields.yml are never type-checked against real values. last_directory_sync is the riskiest of the four: it is a UNIX-epoch integer converted by a date processor into a date-mapped field, and a format or target mistake there only surfaces in production. notes is likewise only present as an empty string, which the drop_null_values script removes, so the match_only_text mapping is never populated either. New data streams should have fixture coverage for every processor branch they ship.
Recommendation:
Append a third user to test-user.log that carries the untested attributes, then regenerate the expected file with elastic-package test pipeline --generate:
{"user_id":"DUJZZZZZZZZZZZZZZZZZ","username":"lockedout","realname":"Locked Out","email":"lockedout@example.com","status":"locked out","created":1619827200,"last_login":1626953200,"last_directory_sync":1657227493,"lockout_reason":"pending deletion","external_id":"1a2345b6-7cd8-9e0f-g1hi-23j45kl6m789","notes":"Imported from directory sync","is_enrolled":true,"firstname":"Locked","lastname":"Out","groups":[],"phones":[],"tokens":[],"u2ftokens":[{"date_added":1444678994,"registration_id":"D2SXXXXXXXXXXXXXXXXX"}],"webauthncredentials":[],"custom_attributes":{},"aliases":{}}🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| "phones": [ | ||
| { | ||
| "activated": true, | ||
| "number": "+15555551234", |
There was a problem hiding this comment.
Do you think we should mask/redact phone numbers behind an option?
There was a problem hiding this comment.
We don't redact it elsewhere.
| "iam" | ||
| ], | ||
| "id": "DUJYYYYYYYYYYYYYYYYY", | ||
| "kind": "state", |
There was a problem hiding this comment.
Do you think we can set this to asset irrespective of ecs.version?
| "preserve_original_event" | ||
| ], | ||
| "user": { | ||
| "email": "jsmith@example.com", |
There was a problem hiding this comment.
Do you think user.email is a good enough identifier to be added to related.user?
https://www.elastic.co/docs/reference/ecs/ecs-related#field-related-user
| @@ -0,0 +1,20 @@ | |||
| - name: data_stream.type | |||
There was a problem hiding this comment.
Could use external: ecs on these fields.
|
✅ All changelog entries have the correct PR link. |
|
🟢 Reviewed the latest commits 50b57fc — nothing new beyond already posted comments. Review summaryIssues found across earlier commits 6e11835 — 1 medium
Issues found across earlier commits 25f7dd3 — 2 medium, 1 low
Issues found across earlier commits c1fc6d9 — 2 medium, 1 low
Issues found across earlier commits 3f1b955 — 1 high, 1 medium, 1 low
Issues found across earlier commits 8a8ec00 — 2 high, 2 medium
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
cc @efd6 |
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package cisco_duo - 2.12.0 containing this change is available at https://epr.elastic.co/package/cisco_duo/2.12.0/ |
2 participants
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots