[workday][activity] Add Activity datastream - #19319
Conversation
Elastic Docs Style Checker (Vale)Summary: 1 warning, 1 suggestion found
|
| File | Line | Rule | Message |
|---|---|---|---|
| packages/workday/_dev/build/docs/README.md | 5 | Elastic.EndPuntuaction | Don't end headings with punctuation. |
💡 Suggestions (1): Optional style improvements. Apply when helpful.
| File | Line | Rule | Message |
|---|---|---|---|
| packages/workday/changelog.yml | 1 | Elastic.Versions | Use 'later versions' instead of 'newer versions' when referring to versions. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
🚀 Benchmarks reportTo see the full report comment with |
![vera-review-bot[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| ) | ||
| : (resp.StatusCode == 401) ? | ||
| { | ||
| "events": [{"message": "retry"}], |
There was a problem hiding this comment.
🟠 HIGH data_stream/activity/.../cel.yml.hbs:153
401 retry emits non-JSON event that produces pipeline_error documents
On HTTP 401 the program returns "events": [{"message": "retry"}] with want_more: true. The ingest pipeline renames message to event.original then runs the json processor on it. Because retry is not valid JSON, the json processor fails and the pipeline-level on_failure fires, indexing a pipeline_error document on every token-refresh cycle. This pollutes the index and produces noisy error tags during normal operation.
Recommendation:
Use the single-object error shape (which preserves cursor and triggers retry without emitting documents) instead of a synthetic event:
: (resp.StatusCode == 401) ?
{
"events": {
"error": {
"code": string(resp.StatusCode),
"id": string(resp.Status),
"message": "GET " + base_url + "/activityLogging: token expired, retrying",
},
},
"cursor": {
?"last_timestamp": state.?cursor.last_timestamp,
?"max_ingested_time": state.?cursor.max_ingested_time,
"access_token": "",
},
"want_more": true,
"Offset": state.Offset,
}
🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| initial_interval: {{initial_interval}} | ||
| batch_size : {{batch_size}} | ||
| Offset: 0 | ||
| redact: |
There was a problem hiding this comment.
🟠 HIGH data_stream/activity/.../cel.yml.hbs:25
redact.fields is empty so credentials appear in state and request traces
redact.fields: ~ leaves the redaction list null. The CEL state holds client_id, client_secret, refresh_token, and cursor.access_token — all sensitive. With enable_request_tracer enabled (a documented user-facing option) these values can be written to agent state logs and HTTP request traces.
Recommendation:
Enumerate the sensitive paths so the agent redacts them before logging:
redact:
fields:
- client_id
- client_secret
- refresh_token
- cursor.access_token🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,284 @@ | |||
| --- | |||
| description: Pipeline for processing activity logs. | |||
| processors: | |||
There was a problem hiding this comment.
🟡 MEDIUM data_stream/activity/.../default.yml:3
Missing Agentless opener removes cloud/host leakage
The package declares agentless.enabled: true in manifest.yml, but the pipeline does not strip Agentless-injected cloud.* / host.* fields before processing. Without the opener those fields land in Workday documents because the Agentless runner populates them.
Recommendation:
Add the standard Agentless cleanup processor between set_ecs_version and the existing terminate:
processors:
- set:
field: ecs.version
tag: set_ecs_version
value: 9.3.0
- remove:
field:
- cloud.account.id
- cloud.availability_zone
- cloud.image.id
- cloud.instance.id
- cloud.instance.name
- cloud.machine.type
- cloud.provider
- cloud.region
- cloud.service.name
- host
tag: remove_agentless_fields
ignore_missing: true
if: ctx.input?.type == 'cel'
- terminate:
tag: data_collection_error
if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| tag: rename_deviceType | ||
| target_field: workday.activity.device_type | ||
| ignore_missing: true | ||
| - set: |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/activity/.../default.yml:88
device.type not normalized to ECS canonical lowercase values
Workday returns deviceType as Desktop (confirmed by sample event and pipeline tests). The pipeline copies it verbatim into device.type. ECS device.type canonical values are lowercase (desktop, mobile, tablet, other); mixed-case values break cross-source dashboard aggregations.
Recommendation:
Lowercase the value after the copy_from step:
- set:
field: device.type
tag: set_device_type_from_activity_device_type
copy_from: workday.activity.device_type
ignore_empty_value: true
- lowercase:
field: device.type
tag: lowercase_device_type
ignore_missing: true🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| - append: | ||
| field: error.message | ||
| value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' | ||
| - remove: |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/activity/.../default.yml:219
Custom workday.activity fields are removed but still declared in fields.yml
remove_custom_duplicate_fields unconditionally drops workday.activity.ip_address, workday.activity.request_time, workday.activity.user_agent, and workday.activity.device_type, yet all four are declared in fields/fields.yml. The declarations will never receive data. Either delete those declarations, or gate the removal on a preserve_duplicate_custom_fields variable so users can opt in.
Recommendation:
Gate the removal behind a variable so the declared fields can be populated when requested:
- remove:
field:
- workday.activity.ip_address
- workday.activity.request_time
- workday.activity.user_agent
- workday.activity.device_type
tag: remove_custom_duplicate_fields
ignore_missing: true
if: ctx._conf?.preserve_duplicate_custom_fields != trueIf the project does not want to expose that variable, delete the four entries from fields/fields.yml instead.
🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| data_stream: | ||
| vars: | ||
| preserve_original_event: true | ||
| preserve_duplicate_custom_fields: true |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/activity/.../test-common-config.yml:34
Test sets preserve_duplicate_custom_fields but no manifest declares it
The system-test config sets preserve_duplicate_custom_fields: true, but no variable with that name is declared in any manifest, and the ingest pipeline has no if referencing it. The setting is silently dropped — the test does not exercise what it claims to.
Recommendation:
Either remove the unused setting, or implement the variable in the data-stream manifest and the pipeline (see finding #5) and keep this as the preserve=true case:
data_stream:
vars:
preserve_original_event: true
batch_size: 2🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| field: error.message | ||
| value: |- | ||
| Processor '{{{_ingest.on_failure_processor_type}}}' | ||
| {{{#_ingest.on_failure_processor_tag}}}with tag '{{{_ingest.on_failure_processor_tag}}}}' |
There was a problem hiding this comment.
🔵 LOW data_stream/activity/.../default.yml:274
Pipeline on_failure mustache has stray extra closing brace
{{{_ingest.on_failure_processor_tag}}}} closes the mustache reference with three braces and then includes a literal }. Rendered messages contain with tag 'set_event_kind}' — confusing in alerts and downstream log search.
Recommendation:
Drop the extra brace:
on_failure:
- append:
field: error.message
value: |-
Processor '{{{_ingest.on_failure_processor_type}}}'
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{_ingest.on_failure_processor_tag}}}'
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{_ingest.on_failure_message}}}'🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| token_url: https://{{hostname}}/ccx/oauth2/{{tenant}}/token | ||
| initial_interval: {{initial_interval}} | ||
| batch_size : {{batch_size}} | ||
| Offset: 0 |
There was a problem hiding this comment.
🔵 LOW data_stream/activity/.../cel.yml.hbs:24
state.Offset uses inconsistent capitalization
All other state keys are lowercase (client_id, refresh_token, initial_interval, batch_size) but Offset is capitalized. Mixing conventions hurts readability and risks state.offset vs state.Offset typos in conditional access.
Recommendation:
Rename to lowercase consistently everywhere Offset appears:
state:
client_id: {{client_id}}
client_secret: {{client_secret}}
refresh_token: {{refresh_token}}
token_url: https://{{hostname}}/ccx/oauth2/{{tenant}}/token
initial_interval: {{initial_interval}}
batch_size: {{batch_size}}
offset: 0🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
| ##### Enable User Activity Logging | ||
|
|
||
| 1. Sign in to your Workday tenant as a Security Administrator. | ||
| 2 .In the Workday search bar, search for and open the Edit Tenant Setup - System task. |
There was a problem hiding this comment.
🔵 LOW _dev/build/docs/README.md:40
Numbered list typo "2 ." breaks ordered-list rendering
Step 2 reads 2 .In the Workday search bar, search for... — the period sits after a space, so Markdown does not treat the line as the second list item.
Recommendation:
Fix the period placement:
2. In the Workday search bar, search for and open the Edit Tenant Setup - System task.🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
|
|
||
| Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. | ||
|
|
||
| ### configure |
There was a problem hiding this comment.
🔵 LOW _dev/build/docs/README.md:100
Lowercase "configure" heading breaks Title Case pattern
All sibling headings use Title Case (### Compatibility, ### Validation, ### Agent-based installation); ### configure looks unedited next to them.
Recommendation:
Use Title Case:
### Configure the integration🤖 AI-Generated Review | Vera Review Bot
⚠️ Automated review — verify suggestions before applying.
andrewkroh
added
New Integration
| "activity": { | ||
| "activity_action": "READ", | ||
| "session_id": "c7c6ff", | ||
| "system_account": "wd-implementer", |
There was a problem hiding this comment.
@muskan-agarwal26 would this be mapped to user.name?
There was a problem hiding this comment.
systemAccount is the Workday user identifier (e.g. wd-implementer for the ISU), so user.name is the right ECS target. Thanks!
💚 Build Succeeded
History
|
efd6
left a comment
efd6
left a comment
There was a problem hiding this comment.
LGTM, but please wait for @jamiehynds
|
@jamiehynds , could you please have a look again, I have resolved your comment. |
LGTM |
|
Package workday - 0.1.0 containing this change is available at https://epr.elastic.co/package/workday/0.1.0/ |
andrewkroh
added
dashboard
7 participants
Proposed commit message
The release includes activity data stream and associated dashboard.
Workday fields are mapped to their corresponding ECS fields where possible.
Test samples were derived from live data samples, which were subsequently
sanitized.
Checklist
changelog.ymlfile.How to test this PR locally
To test the workday package:
Related issues
Screenshots