Skip to content

[Workday][Sign-on] Add Sign-on data stream - #20039

Merged
chemamartinez merged 7 commits into
elastic:mainfrom
muskan-agarwal26:workday-sign_on-0.1.1
Jul 17, 2026
Merged

[Workday][Sign-on] Add Sign-on data stream#20039
chemamartinez merged 7 commits into
elastic:mainfrom
muskan-agarwal26:workday-sign_on-0.1.1

Conversation

@muskan-agarwal26

@muskan-agarwal26 muskan-agarwal26 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

This PR adds the `sign_on` data stream to the Workday integration, along
with its associated dashboard and visualizations.

Workday fields are mapped to their corresponding ECS fields where
possible.

Test samples were derived from live data samples, which were
subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

To test the Workday package:

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/workday directory.
  • Run the following command to run tests.

elastic-package test

Note: pipeline/system tests were not run in this session. `elastic-package build` and `elastic-package check` pass on the package (the only warnings are the intentionally-excluded SVR00004 dashboard-reference checks). Run `elastic-package test` locally to execute the full pipeline/system test suite.

Related issues

Screenshots

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 2 suggestions found

💡 Suggestions (2): Optional style improvements. Apply when helpful.
File Line Rule Message
packages/workday/_dev/build/docs/README.md 12 Elastic.Ellipses In general, don't use an ellipsis.
packages/workday/data_stream/sign_on/fields/fields.yml 12 Elastic.WordChoice Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Co-authored-by: Cursor <cursoragent@cursor.com>
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@andrewkroh andrewkroh added Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:workday Workday labels Jul 8, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@muskan-agarwal26
muskan-agarwal26 marked this pull request as ready for review July 8, 2026 17:33
@muskan-agarwal26
muskan-agarwal26 requested a review from a team as a code owner July 8, 2026 17:33
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Check integrations workday failed while running .buildkite/scripts/test_one_package.sh packages/workday .... The actionable issue in this PR is that Activity credentials were moved from package input vars to data-stream vars, but the Activity system-test config still supplies them as package-level vars, so the generated policy for the system test is missing required stream variables.

Remediation

  • Update packages/workday/data_stream/activity/_dev/test/system/test-common-config.yml so hostname, tenant, client_id, client_secret, and refresh_token are under data_stream.vars; keep shared input vars such as ssl under top-level vars.
  • Re-run the Workday package checks, especially elastic-package test system -v -p packages/workday and then the full package test command used by CI.
Investigation details

Root Cause

The PR moves Activity-specific required variables into the Activity data stream manifest:

  • packages/workday/data_stream/activity/manifest.yml:9-43 now declares hostname, tenant, client_id, client_secret, and refresh_token as required stream vars.
  • packages/workday/manifest.yml:44-52 no longer defines those variables at the package input level; only shared input vars such as proxy_url/ssl remain there.
  • packages/workday/data_stream/activity/_dev/test/system/test-common-config.yml:3-8 still sets those credentials under top-level vars, while only preserve_original_event and batch_size are under data_stream.vars at lines 31-34.

That layout matched the old manifest, but after this PR the required Activity variables must be supplied through data_stream.vars for system tests.

Evidence

  • Build: https://buildkite.com/elastic/integrations/builds/45748
  • Job/step: Check integrations workday
  • Command: .buildkite/scripts/test_one_package.sh packages/workday origin/main bd625d54dc03a3c02bac41dc1ae4d4170e3589dc
  • Captured log excerpt: --- [workday] failed followed by Error: The command exited with status 1. The prefetched Buildkite log only contains the job tail and artifact upload summary; the detailed test assertion output was not included in the local log file.

Verification

  • Reviewed the PR diff and file contents for the changed Workday package.
  • Local elastic-package verification was not run because elastic-package is not installed in this environment, and Docker-in-Docker is unavailable for system tests.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jul 8, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/lifecycle.yml Outdated
Comment thread packages/workday/_dev/build/build.yml
Comment thread packages/workday/data_stream/sign_on/fields/fields.yml Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs Outdated
Comment thread packages/workday/data_stream/sign_on/agent/stream/cel.yml.hbs
Comment thread packages/workday/manifest.yml
@muskan-agarwal26
muskan-agarwal26 requested a review from efd6 July 9, 2026 10:22
Comment thread packages/workday/_dev/build/build.yml
}
},
"delete": {
"min_age": "30d",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/workday/data_stream/sign_on/elasticsearch/ilm/default_policy.json:16

The new sign_on data stream ships a custom ILM policy that deletes data after 30 days, silently discarding authentication/audit telemetry; drop the delete phase (or the custom policy) and let the stack-managed logs lifecycle govern retention.

Details

The sign_on data stream introduces a package-shipped ILM policy with a delete phase at min_age: 30d. This hard-deletes sign-on records after 30 days, which is short for security/authentication audit data that teams typically retain for investigations and compliance. The pre-existing activity data stream ships no custom ILM policy and relies on the stack-managed logs lifecycle, so this also makes retention inconsistent within the same package and overrides the retention strategy the user would otherwise control.

Recommendation:

Prefer not shipping a custom lifecycle and letting the stack-managed logs policy handle retention (remove the ilm_policy line in manifest.yml and the ilm/default_policy.json file). If a custom rollover is genuinely required, at minimum drop the auto-delete so audit data is not silently removed:

{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_age": "2d",
                        "max_size": "50gb"
                    },
                    "set_priority": {
                        "priority": 100
                    }
                }
            }
        }
    }
}

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

title: Sign-on logs
description: Collect Sign-on logs from Workday.
template_path: cel.yml.hbs
vars:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: medium path: packages/workday/data_stream/sign_on/manifest.yml:9

The sign_on cel.yml.hbs references proxy_url and ssl but the sign_on manifest defines neither var, so those template branches are dead and users cannot configure a proxy or custom CA for the HTTPS Workday endpoint; add the vars (as the activity stream already does).

Details

The shared sign_on template (agent/stream/cel.yml.hbs) contains {{#if proxy_url}}resource.proxy_url: {{proxy_url}}{{/if}} and {{#if ssl}}resource.ssl: {{ssl}}{{/if}}, but the sign_on manifest's stream vars list does not declare proxy_url or ssl. As a result those branches never render and operators have no way to route the Workday Custom Report (RaaS) HTTPS request through a proxy or trust a custom/internal CA — capabilities the activity policy template does expose. This is a configuration-completeness gap for enterprise Workday deployments.

Recommendation:

Declare the proxy_url and ssl vars in the sign_on stream, mirroring the activity input:

      - name: proxy_url
        type: text
        title: Proxy URL
        description: proxy configuration in the form of https://<user>:<password>@<server name/ip>:<port>.
        multi: false
        required: false
        show_user: false
      - name: ssl
        type: yaml
        title: SSL Configuration
        description: SSL configuration options. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#ssl-common-config) for details.
        multi: false
        required: false
        show_user: false

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

"authentication",
"session"
],
"end": "2026-06-19T06:25:22.000Z",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@uri-weisman do you think it makes sense that this session_end timestamp is mapped to host.entity.lifecycle.last_activity?

Would you map it to the session_start field instead?

https://www.elastic.co/docs/reference/ecs/ecs-entity#field-entity-lifecycle-last-activity

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, but we just need to verify that for activity logs, besides sign-on we populate this field.

Comment thread packages/workday/data_stream/sign_on/elasticsearch/ingest_pipeline/default.yml Outdated

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most recent Vera review is correct.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@vera-review-bot

Copy link
Copy Markdown

No issues across the latest commits f523150.

Review summary

Issues found across earlier commits e1abf069478bbe (34 commits) — 1 low
  • 🔵 The top-level on_failure error template has a stray extra brace (}}}} instead of }}}) after the processor tag, so rendered pipeline-error messages end the tag value with a spurious }. Remove the extra brace to match the sibling activity pipeline. (link) (Unresolved)
Issues found across earlier commits 7d20ff6 — 1 medium
  • 🟡 The Sign-on dashboard's authentication control and 3 visualizations key only on workday.sign_on.Authentication_Type, but the README-recommended report column maps to workday.sign_on.Authentication_Type_for_Signon (link) (Resolved)
Issues found across earlier commits 94d8c84 — 1 high, 1 medium, 1 low
  • 🟠 This PR bumps the ECS pin in build.yml to v9.4.0 but the existing activity pipeline still emits ecs.version 9.3.0 (link) (Resolved)
  • 🟡 The new sign_on data stream ships a custom ILM policy that deletes data after 30 days, silently discarding authentication/audit telemetry (link) (Unresolved)
  • 🔵 The sign_on cel.yml.hbs references proxy_url and ssl but the sign_on manifest defines neither var, so those template branches are dead and users cannot configure a proxy or custom CA for the HTTPS Workday endpoint (link) (Unresolved)
Issues found across earlier commits bd625d5 — 2 high, 5 medium, 2 low
  • 🟠 Sign-on CEL emits API errors wrapped inside message, so ctx.error is never set and the pipeline's error handling (terminate/pipeline_error) never fires — API errors are indexed as normal sign-on events. Emit the single-object {"events": {"error": {...}}} shape like the activity stream. (link) (Resolved)
  • 🟠 Sign-on ingests the full custom report every interval with no _id fingerprint, so each poll re-indexes the same rows as new documents (unbounded duplication). Add a fingerprint processor that derives a stable _id, as the activity pipeline does. (link) (Resolved)
  • 🟡 When the report is empty the CEL emits [{"message": "{}"}], which the pipeline turns into a spurious user-signon event (empty workday.sign_on, timestamp defaulted to ingest time) on every empty poll. Emit an empty events array instead. (link) (Resolved)
  • 🟡 The sign_on data stream defines both a custom ILM policy (ilm_policy + elasticsearch/ilm/default_policy.json) and a Data Stream Lifecycle (lifecycle.yml with data_retention). Only one retention mechanism should be used (link) (Resolved)
  • 🟡 build.yml bumps the ECS pin to v9.4.0, but the existing activity pipeline still sets ecs.version: 9.3.0, leaving the package build ECS version and the activity pipeline inconsistent. Bump the activity pipeline to 9.4.0 to match. (link) (Resolved)
  • 🟡 The sign_on pipeline sets source.ip but performs no geo/ASN enrichment, unlike the sibling activity pipeline. Add geoip (city + ASN) enrichment on source.ip so the Top Source IPs and location analytics work. (link) (Resolved)
  • 🟡 Authentication events never set event.outcome, even though the report exposes Failed_Signon and Authentication_Failure_Message. Set event.outcome to success/failure so auth analytics and detections can distinguish failed sign-ons. (link) (Resolved)
  • 🔵 fields.yml declares signon_ip_address_string typo variant signin_ip_address_string that the pipeline never produces. Remove the unused field definition. (link) (Resolved)
  • 🔵 The sign_on dashboard's overview markdown links to a version-pinned integration page (workday-0.1.0), which is already stale (package is 0.1.1) and will drift on every release. Link to the unversioned integration page. (link) (Resolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

@mergify

mergify Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@chemamartinez chemamartinez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chemamartinez
chemamartinez merged commit 617ad3d into elastic:main Jul 17, 2026
12 checks passed
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package workday - 0.3.0 containing this change is available at https://epr.elastic.co/package/workday/0.3.0/

christophercutajar pushed a commit to christophercutajar/integrations that referenced this pull request Jul 21, 2026
This PR adds the `sign_on` data stream to the Workday integration, along
with its associated dashboard and visualizations.

Workday fields are mapped to their corresponding ECS fields where
possible.

Test samples were derived from live data samples, which were
subsequently sanitized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:workday Workday Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants