Skip to content

[elastic_security] Migrate Alert Data Stream Authentication From required_vars to var_groups - #20366

Merged
mohitjha-elastic merged 6 commits into
elastic:mainfrom
mohitjha-elastic:elastic_security-0.6.0
Aug 5, 2026
Merged

[elastic_security] Migrate Alert Data Stream Authentication From required_vars to var_groups#20366
mohitjha-elastic merged 6 commits into
elastic:mainfrom
mohitjha-elastic:elastic_security-0.6.0

Conversation

@mohitjha-elastic

@mohitjha-elastic mohitjha-elastic commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

elastic_security: migrate alert authentication to var_groups and require Kibana 9.4.0.

Migrate the alert data stream authentication configuration from `required_vars` to 
`var_groups` for a cleaner user experience, and bump the minimum required 
Kibana and Elastic Agent version to `^9.4.0` to support this change.
Breaking change: the authentication settings moved from the data stream to the 
integration policy level. Add Fleet input validation for duration fields and URL scheme restrictions.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install the elastic package locally.
  • Start the elastic stack using the elastic package.
  • Move to integrations/packages/elastic_security directory.
  • Run the following command to run tests.

elastic-package test -v

Related Issues

Screenshots

These are the screenshots for the var_groups to hide the different auth type.

image image image
@mohitjha-elastic mohitjha-elastic self-assigned this Jul 27, 2026
@mohitjha-elastic
mohitjha-elastic requested review from a team as code owners July 27, 2026 12:06
@mohitjha-elastic mohitjha-elastic added breaking change Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Integration:elastic_security Elastic Security labels Jul 27, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@mohitjha-elastic mohitjha-elastic added the enhancement New feature or request label Jul 27, 2026
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 27, 2026
@ShourieG

Copy link
Copy Markdown
Contributor

@vera-review-bot review

"Content-Type": ["application/json"],
"Authorization": [
state.auth_type == 'api_auth' ?
has(state.api_key) && state.api_key != "" ?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔴 Critical confidence: medium path: packages/elastic_security/data_stream/alert/agent/stream/cel.yml.hbs:55

The new presence-based auth detection cannot distinguish auth types because the state block still emits all four credential keys unconditionally; wrap each key in a {{#if}} guard so unselected credentials are absent from state.

Details

The auth_type selector was removed from state, and the Authorization header now infers the auth method from the credential values themselves. But the state: block (lines 18-21) still emits api_key, user, password and bearer_token on every render, regardless of which var_group option the user picked.

Two consequences:

  1. has(state.api_key) / has(state.user) / has(state.password) are always true — the key is always present in the map — so those guards do no work and the whole decision rests on the != "" comparisons.

  2. When a var is not set (basic_auth or bearer_auth selected, so api_key is unset), handlebars renders the line as api_key: with no value, which YAML-loads as null, not as an empty string. state.api_key != "" is therefore true for a null value, the first branch is taken, and "Apikey " + state.api_key fails with a no-such-overload evaluation error. Basic Auth and Bearer Auth users get no events at all.

The rest of this template already guards optional vars this way (proxy_url, ssl, http_client_timeout on lines 7-15) precisely because unset vars render as empty/null. The auth vars did not need guards before, because the old code selected on state.auth_type and never read the unused keys; now that the code reads them, they do.

CI does not catch this: the system test only exercises the API-key path.

Recommendation:

Emit each credential only when it is set, so has() is a genuine presence check:

state:
{{#if api_key}}
  api_key: {{api_key}}
{{/if}}
{{#if username}}
  user: {{username}}
{{/if}}
{{#if password}}
  password: {{password}}
{{/if}}
{{#if bearer_token}}
  bearer_token: {{bearer_token}}
{{/if}}
  index: .alerts-security.alerts-{{space_id}}
  batch_size: {{batch_size}}
  initial_interval: {{initial_interval}}

Then the header selection can rely on presence alone:

"Authorization": [
  has(state.api_key) ?
    "Apikey " + state.api_key
  :
    has(state.user) && has(state.password) ?
      "Basic " + base64(state.user + ":" + state.password)
    :
      "Bearer " + state.bearer_token
],

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a correctness issue, but I do think the suggested approach is cleaner.

Processor '{{{ _ingest.on_failure_processor_type }}}'
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'
{{{/_ingest.on_failure_processor_tag}}}failed in pipeline '{{{ _ingest.pipeline }}}' with message '{{{ _ingest.on_failure_message }}}'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: high path: packages/elastic_security/data_stream/alert/elasticsearch/ingest_pipeline/default.yml:150

The pipeline-level on_failure message interpolates _ingest.pipeline, which does not exist in an on_failure context and always renders empty; use _ingest.on_failure_pipeline instead.

Details

Elasticsearch exposes only four ingest metadata fields inside an on_failure block: on_failure_message, on_failure_processor_type, on_failure_processor_tag and on_failure_pipeline. _ingest.pipeline is not one of them, so the new text always renders as failed in pipeline '' with message '...' — the addition adds noise instead of the intended context.

The json processor's own on_failure in this same file (line 22) already uses the correct {{{_ingest.on_failure_pipeline}}}, so the two error messages are also inconsistent with each other.

Recommendation:

Use the on_failure-scoped metadata field:

on_failure:
  - append:
      field: error.message
      tag: append_pipeline_error_message
      value: >-
        Processor '{{{ _ingest.on_failure_processor_type }}}'
        {{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
        {{{/_ingest.on_failure_processor_tag}}}failed in pipeline '{{{ _ingest.on_failure_pipeline }}}' with message '{{{ _ingest.on_failure_message }}}'

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct for single pipeline ingestions. The syntactic consistency may be worth addressing though.

Details
PUT _ingest/pipeline/test-on-failure-fields
{
  "processors": [
    {
      "convert": {
        "field": "not_a_number",
        "type": "integer",
        "tag": "convert_test"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "debug.ingest_pipeline",
        "value": "{{{_ingest.pipeline}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_pipeline",
        "value": "{{{_ingest.on_failure_pipeline}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_processor_type",
        "value": "{{{_ingest.on_failure_processor_type}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_processor_tag",
        "value": "{{{_ingest.on_failure_processor_tag}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_message",
        "value": "{{{_ingest.on_failure_message}}}"
      }
    }
  ]
}

POST _ingest/pipeline/test-on-failure-fields/_simulate
{
  "docs": [
    {
      "_source": {
        "not_a_number": "hello"
      }
    }
  ]
}

DELETE _ingest/pipeline/test-on-failure-fields
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "not_a_number": "hello",
          "debug": {
            "ingest_pipeline": "test-on-failure-fields",
            "on_failure_message": "For input string: \"hello\"",
            "on_failure_processor_tag": "convert_test",
            "on_failure_pipeline": "test-on-failure-fields",
            "on_failure_processor_type": "convert"
          }
        },
        "_ingest": {
          "timestamp": "2026-07-28T01:04:51.588036835Z"
        }
      }
    }
  ]
}
- version: "0.6.0"
changes:
- description: |
Migrate alert data stream authentication from required_vars to var_groups for improved UX and cleaner configuration management.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/elastic_security/changelog.yml:5

Moving the credential vars from stream level to input level drops their stored values on upgrade, but neither the changelog entry nor the README tells users they must re-enter credentials; add that upgrade note.

Details

api_key, username, password and bearer_token previously lived in data_stream/alert/manifest.yml and were stored on the policy under the stream's vars. In 0.6.0 they are declared under policy_templates[].inputs[].vars instead. Fleet matches stored var values by name at the same level when it upgrades a package policy, so a var that disappears from the stream level and reappears at the input level does not carry its value forward — existing 0.5.0 policies come out of the upgrade with empty credentials and stop collecting until an operator re-enters them.

The entry is correctly typed breaking-change, but its text ("improved UX and cleaner configuration management") gives the operator no indication that action is required. The README setup section is likewise unchanged.

Recommendation:

Spell out the required action in the changelog entry, and mirror it in _dev/build/docs/README.md:

- version: "0.6.0"
  changes:
    - description: |
        Migrate alert data stream authentication from required_vars to var_groups for improved UX and cleaner configuration management.
        Bump Kibana and Elastic Agent requirement to ^9.4.0 to support this feature.
        Breaking change: the authentication settings moved from the data stream to the integration policy level.
        After upgrading, edit each existing Elastic Security integration policy and re-enter the credentials
        (API key, username/password, or bearer token) for the selected authentication type.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/20366

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

on_failure:
- append:
field: error.message
tag: append_pipeline_error_message

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: medium path: packages/elastic_security/data_stream/alert/elasticsearch/ingest_pipeline/default.yml:146

The tag backfill in this on_failure block is incomplete — the third processor (append to tags) still has no tag; add one so the package is consistent with the format_version 3.6.1 bump.

Details

This PR adds tag to three processors and bumps format_version to 3.6.1, the spec version at which processor tags are checked. The final processor of the same pipeline-level on_failure block — the append to tags at line 155 — was missed, so the block is now half-tagged and the untagged failure is the one that will be hardest to trace, since it is what runs when the pipeline itself blows up.

Recommendation:

Tag the remaining on_failure processor:

  - append:
      field: tags
      tag: append_pipeline_error_preserve_original_event_tag
      value: preserve_original_event
      allow_duplicates: false

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

title: Collect Elastic Security events via API
description: Collect events from Elastic instance via API.
vars:
- name: username

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: low path: packages/elastic_security/manifest.yml:47

The four relocated credential vars lost their show_user: true attribute in the move to the policy template; restore it so they stay in the primary configuration form rather than behind Advanced options.

Details

At the data stream level each of username, password, api_key and bearer_token carried show_user: true. The copies added under policy_templates[].inputs[].vars carry only type, title, description, multi and secret. Dropping required: false is correct here (the spec forbids required: true on vars owned by a var_group, and requiredness is inferred from var_groups[].required), but show_user is an independent display hint and is not implied by var_group membership.

Recommendation:

Keep the display hint on the relocated vars:

        vars:
          - name: username
            type: text
            title: Username
            description: The username of Elasticsearch Instance to be used with Basic Auth headers.
            multi: false
            show_user: true
          - name: password
            type: password
            title: Password
            description: The password of Elasticsearch Instance to be used with Basic Auth headers.
            multi: false
            show_user: true
            secret: true

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

"Content-Type": ["application/json"],
"Authorization": [
state.auth_type == 'api_auth' ?
has(state.api_key) && state.api_key != "" ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a correctness issue, but I do think the suggested approach is cleaner.

- type: cel
title: Collect Elastic Security events via API
description: Collect events from Elastic instance via API.
vars:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to move this to the package level?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var_groups and their referenced variables should be in the same manifest file.
Here is the discussion related to moving var_groups to package level manifest.

Processor '{{{ _ingest.on_failure_processor_type }}}'
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'
{{{/_ingest.on_failure_processor_tag}}}failed in pipeline '{{{ _ingest.pipeline }}}' with message '{{{ _ingest.on_failure_message }}}'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct for single pipeline ingestions. The syntactic consistency may be worth addressing though.

Details
PUT _ingest/pipeline/test-on-failure-fields
{
  "processors": [
    {
      "convert": {
        "field": "not_a_number",
        "type": "integer",
        "tag": "convert_test"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "debug.ingest_pipeline",
        "value": "{{{_ingest.pipeline}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_pipeline",
        "value": "{{{_ingest.on_failure_pipeline}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_processor_type",
        "value": "{{{_ingest.on_failure_processor_type}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_processor_tag",
        "value": "{{{_ingest.on_failure_processor_tag}}}"
      }
    },
    {
      "set": {
        "field": "debug.on_failure_message",
        "value": "{{{_ingest.on_failure_message}}}"
      }
    }
  ]
}

POST _ingest/pipeline/test-on-failure-fields/_simulate
{
  "docs": [
    {
      "_source": {
        "not_a_number": "hello"
      }
    }
  ]
}

DELETE _ingest/pipeline/test-on-failure-fields
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "not_a_number": "hello",
          "debug": {
            "ingest_pipeline": "test-on-failure-fields",
            "on_failure_message": "For input string: \"hello\"",
            "on_failure_processor_tag": "convert_test",
            "on_failure_pipeline": "test-on-failure-fields",
            "on_failure_processor_type": "convert"
          }
        },
        "_ingest": {
          "timestamp": "2026-07-28T01:04:51.588036835Z"
        }
      }
    }
  ]
}
@mohitjha-elastic
mohitjha-elastic requested a review from efd6 July 31, 2026 11:11
multi: false
secret: true
show_user: true
- name: api_key

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: high path: packages/elastic_security/manifest.yml:60

The auth vars moved to the input level, but data_stream/alert/_dev/test/system/test-default-config.yml still passes api_key under data_stream.vars, so it no longer reaches the policy; move it to the top-level vars block.

Details

This PR removes api_key (and username/password/bearer_token) from data_stream/alert/manifest.yml and redefines them here, under policy_templates[0].inputs[0].vars. The system test config at packages/elastic_security/data_stream/alert/_dev/test/system/test-default-config.yml was not updated: it still sets api_key: xxxx inside data_stream.vars. elastic-package applies data_stream.vars to the stream-level variables and top-level vars to the input-level variables, so api_key is now silently dropped. The rendered cel.yml.hbs then emits no api_key, no user/password and no bearer_token in the state block, the Authorization expression falls through to "Bearer " + state.bearer_token, and the CEL program fails to evaluate. No documents are ingested and the assert.hit_count: 2 check fails. Sibling packages that keep auth at the input level (okta, beyondinsight_password_safe) put those values in the top-level vars block of the test config.

Recommendation:

Move the auth variable to the input-level vars block of the system test config:

input: cel
service: elastic_security-cel
vars:
  api_key: xxxx
data_stream:
  vars:
    url: http://{{Hostname}}:{{Port}}
    interval: 5m
    initial_interval: 24h
    batch_size: 1
    preserve_original_event: true

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

"Apikey " + state.api_key
:
state.auth_type == 'basic_auth' ?
has(state.user) && has(state.password) ?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/elastic_security/data_stream/alert/agent/stream/cel.yml.hbs:66

The final Authorization branch dereferences state.bearer_token without a has() guard, so an upgraded policy with no credentials fails with an opaque CEL evaluation error; guard it like the other two branches.

Details

The Authorization expression now guards the first two branches with has(state.api_key) and has(state.user) && has(state.password), but the trailing else branch accesses state.bearer_token unconditionally. Because every state key is now emitted behind a {{#if}} block, the key is absent whenever bearer_token is unset. This is exactly the state of an existing policy after the breaking change this PR introduces: the four auth vars are redefined at the input level, so an upgraded policy carries no value for any of them until the user re-enters credentials. In that window the CEL program aborts with a no such key: bearer_token evaluation error rather than producing an authentication failure the user can act on.

Recommendation:

Guard the bearer branch the same way the other two are guarded, so a missing credential yields an unauthenticated request (and a clear 401 in the error path) instead of a CEL evaluation failure:

"Authorization": [
  has(state.api_key) ?
    "Apikey " + state.api_key
  :
    has(state.user) && has(state.password) ?
      "Basic " + base64(state.user+":"+state.password)
    :
      has(state.bearer_token) ?
        "Bearer " + state.bearer_token
      :
        ""
],

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Migrate alert data stream authentication from required_vars to var_groups for improved UX and cleaner configuration management.
Bump Kibana and Elastic Agent requirement to ^9.4.0 to support this feature.
Add Fleet input validation for duration fields and URL scheme restrictions.
Breaking change: the authentication settings moved from the data stream to the integration policy level.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/elastic_security/changelog.yml:8

The changelog documents a breaking change that requires users to re-enter credentials after upgrading, but the README has no upgrade note; add one to _dev/build/docs/README.md.

Details

This entry states that the authentication settings move from the data stream to the integration policy level and that every existing policy must be edited to re-enter credentials. packages/elastic_security/_dev/build/docs/README.md (and its generated copy docs/README.md) still only describes a first-time setup flow; it carries no upgrade section. Users who upgrade in place from 0.5.0 read the README, see nothing about re-entering credentials, and their alert collection stops with an authentication/CEL error until they discover the changelog entry.

Recommendation:

Add an upgrade note to packages/elastic_security/_dev/build/docs/README.md (then regenerate docs/README.md with elastic-package build):

### Upgrading to 0.6.0 or later

Starting with version 0.6.0 the authentication settings are configured at the integration policy
level instead of the `alert` data stream level. Credentials configured in earlier versions are not
carried over. After upgrading, edit each existing **Elastic Security** integration policy, choose the
authentication type, and re-enter the corresponding credential (API key, username and password, or
bearer token). Data collection stays paused until this is done.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

conditions:
kibana:
version: "^9.1.1"
version: "^9.4.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/elastic_security/manifest.yml:14

The bump to Kibana ^9.4.0 plus a new agent ^9.4.0 condition blocks every 9.1-9.3 user from upgrading, but none of the three features this PR adds needs 9.4; lower the floor to ^9.2.2 and drop the agent condition.

Details

The changelog justifies the bump as needed 'to support this feature', but the three features introduced here already ship in this repo at much lower floors:

  • input-level var_groups: packages/okta/manifest.yml declares them under conditions.kibana.version: "^8.19.8 || ^9.2.2".
  • type: duration vars: packages/aws_securityhub uses them at ^9.3.0; package-spec only requires format_version >= 3.5.0.
  • url_allowed_schemes: packages/ti_anyrun/manifest.yml uses it at ^8.19.2 || ^9.0.5 with format_version: 3.4.0.

The previous floor was ^9.1.1, so as written 0.6.0 becomes uninstallable for every existing user on 9.1.x, 9.2.x and 9.3.x — they are stranded on 0.5.1 and cannot pick up the auth migration at all.

The new conditions.agent.version: "^9.4.0" has the same problem and no stated basis: the CEL program change in this PR only swaps state.auth_type == '...' comparisons for has(state.api_key) / has(state.user), and has() is core CEL that has been available since the input was introduced. No new CEL extension, config option, or input capability is used, so nothing pins the agent to 9.4.0. (Compare packages/aws/manifest.yml, which does set agent: ^9.4.0 but documents the reason inline — auth.aws support in CEL/HTTPJSON.)

Recommendation:

Lower the Kibana floor to the version that actually supports input-level var_groups, and remove the agent condition unless a specific 9.4.0 agent feature is required (if one is, add an inline comment naming it, as the aws package does):

conditions:
  kibana:
    version: "^9.2.2"
  elastic:
    subscription: basic

And update the changelog wording so it no longer claims 9.4.0 is required by the feature:

- version: "0.6.0"
  changes:
    - description: |
        Migrate alert data stream authentication from required_vars to var_groups for improved UX and cleaner configuration management.
        Bump Kibana requirement to ^9.2.2, the minimum version that supports input-level var_groups.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/20366

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

default: 10000
- name: http_client_timeout
type: text
type: duration

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: medium path: packages/elastic_security/data_stream/alert/manifest.yml:49

Switching http_client_timeout from text to duration narrows the accepted units from Go's full set to ms/s/m/h, so an existing policy holding a ns or us value stops validating; call that out in the breaking-change entry.

Details

The duration var type in package-spec constrains values to the pattern ^(\d+[smh]|\d+ms)+$ — only ms, s, m and h are accepted. The old type: text accepted anything, and the previous description explicitly advertised ns and us as supported units (the diff removes them from the description, which acknowledges the narrowing).

resource.timeout in the CEL input is a Go duration, so 500us or 250000000ns were legitimate values under 0.5.1 and are rejected under 0.6.0. Any existing policy carrying such a value fails validation on upgrade rather than migrating cleanly.

The changelog entry mentions 'Add Fleet input validation for duration fields' but does not say that previously-valid values become invalid, so a user hitting this on upgrade has no pointer to the cause. Impact is limited because the var is show_user: false with a 60s default, hence low severity.

Recommendation:

Keep the type change and extend the breaking-change note so the failure mode is discoverable:

- version: "0.6.0"
  changes:
    - description: |
        Migrate alert data stream authentication from required_vars to var_groups for improved UX and cleaner configuration management.
        Add Fleet input validation for duration fields and URL scheme restrictions.
        Breaking change: Initial Interval, Interval and HTTP Client Timeout are now duration fields and only accept ms/s/m/h units.
        Policies that stored a value using ns or us units must be updated to an equivalent ms/s/m/h value before upgrading.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/20366

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

service: elastic_security-cel
wait_for_data_timeout: 2m
vars:
api_key: xxxx

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/elastic_security/data_stream/alert/_dev/test/system/test-default-config.yml:5

This PR replaced the explicit auth_type switch with presence-based has() checks in the CEL program, but the only system test config still exercises api_auth, so the basic_auth and bearer_auth branches are never executed; add a system test config per var_group option and matching mock rules.

Details

The var_group added in packages/elastic_security/manifest.yml offers three options (api_auth, basic_auth, bearer_auth), and cel.yml.hbs now selects the Authorization header by testing which credential keys are present in state (has(state.api_key), then has(state.user) && has(state.password), then the bearer fallback). test-default-config.yml is the only system test config and it sets api_key, which is also the first var_group option and therefore the default selection, so every run takes the first branch. All three mock rules in packages/elastic_security/_dev/deploy/docker/files/config.yml also require request_headers.Authorization: "Apikey xxxx", so the basic_auth and bearer_auth header construction is not covered end to end by any test in this package. Okta, the other package in this repo using var_groups, ships one system test config per auth option for exactly this reason.

Recommendation:

Add one system test config per var_group option and a mock rule matching each Authorization header. For example:

# data_stream/alert/_dev/test/system/test-basic-auth-config.yml
input: cel
service: elastic_security-cel
wait_for_data_timeout: 2m
vars:
  username: elastic
  password: xxxx
data_stream:
  vars:
    url: http://{{Hostname}}:{{Port}}
    interval: 10s
    initial_interval: 24h
    batch_size: 1
    preserve_original_event: true
# copy the numeric_keyword_fields list and assert block from test-default-config.yml

and a matching rule in _dev/deploy/docker/files/config.yml so the mock only answers when the Basic header is built correctly:

  - path: /.alerts-security.alerts-default/_search
    methods: ['GET']
    request_headers:
      Authorization:
        - "Basic <base64 of the username:password pair used above>"
    responses:
      - status_code: 200
        body: '{"hits":{"hits":[]}}'

Repeat with a test-bearer-auth-config.yml setting bearer_token and a rule matching "Bearer xxxx".


🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@vera-review-bot

Copy link
Copy Markdown

Review summary

Issues found across the latest commits 98c11ef — 1 medium
  • 🟡 This PR replaced the explicit auth_type switch with presence-based has() checks in the CEL program, but the only system test config still exercises api_auth, so the basic_auth and bearer_auth branches are never executed (link) (Unresolved)
Issues found across earlier commits 5e50cc05215ccf (52 commits) — 1 medium, 1 low
  • 🟡 The bump to Kibana ^9.4.0 plus a new agent ^9.4.0 condition blocks every 9.1-9.3 user from upgrading, but none of the three features this PR adds needs 9.4 (link) (Unresolved)
  • 🔵 Switching http_client_timeout from text to duration narrows the accepted units from Go's full set to ms/s/m/h, so an existing policy holding a ns or us value stops validating (link) (Unresolved)
Issues found across earlier commits b092f82, c4dd03e — 1 high, 2 medium
  • 🟠 The auth vars moved to the input level, but data_stream/alert/_dev/test/system/test-default-config.yml still passes api_key under data_stream.vars, so it no longer reaches the policy (link) (Unresolved)
  • 🟡 The final Authorization branch dereferences state.bearer_token without a has() guard, so an upgraded policy with no credentials fails with an opaque CEL evaluation error (link) (Unresolved)
  • 🟡 The changelog documents a breaking change that requires users to re-enter credentials after upgrading, but the README has no upgrade note (link) (Unresolved)
Issues found across earlier commits 7b66c5c — 1 critical, 2 medium, 2 low
  • 🔴 The new presence-based auth detection cannot distinguish auth types because the state block still emits all four credential keys unconditionally (link) (Unresolved)
  • 🟡 The pipeline-level on_failure message interpolates _ingest.pipeline, which does not exist in an on_failure context and always renders empty (link) (Unresolved)
  • 🟡 Moving the credential vars from stream level to input level drops their stored values on upgrade, but neither the changelog entry nor the README tells users they must re-enter credentials (link) (Unresolved)
  • 🔵 The tag backfill in this on_failure block is incomplete — the third processor (append to tags) still has no tag (link) (Unresolved)
  • 🔵 The four relocated credential vars lost their show_user: true attribute in the move to the policy template (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @mohitjha-elastic

@mergify

mergify Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request
@mohitjha-elastic
mohitjha-elastic merged commit ddd4d20 into elastic:main Aug 5, 2026
11 checks passed
@mohitjha-elastic
mohitjha-elastic deleted the elastic_security-0.6.0 branch August 5, 2026 08:49
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package elastic_security - 0.6.0 containing this change is available at https://epr.elastic.co/package/elastic_security/0.6.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:elastic_security Elastic Security Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

4 participants