[wiz] Add Defend V2 Data Stream - #20071
Conversation
mohitjha-elastic
added
documentation
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![vera-review-bot[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| field: cloud.instance.id | ||
| tag: append_cloud_instance_id_from_triggering_events | ||
| value: '{{{_ingest._value.raw_audit_log_record.common.vm_external_id}}}' | ||
| if: ctx._ingest?._value?.raw_audit_log_record?.common?.vm_external_id != null |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/wiz/data_stream/defend_v2/elasticsearch/ingest_pipeline/default.yml:397
host.name, cloud.instance.id, file.path and event.action are never populated from triggering-event audit records because of a broken inner if guard; remove the redundant inner if so these mappings match the working sibling processors.
Details
The four foreach processors that map fields out of wiz.defend_v2.triggering_events.nodes[].raw_audit_log_record — cloud.instance.id (tag foreach_triggering_events_into_cloud_instance_id), host.name (foreach_triggering_events_into_host_name), file.path (foreach_triggering_events_into_file_path) and event.action (foreach_triggering_events_into_event_action) — each carry an inner if: ctx._ingest?._value?.raw_audit_log_record?.... In the committed expected output (test-defend-v2.log-expected.json) none of these fields appear on events 2 and 3, even though those events' audit records contain common.vm_external_id, common.host_info.hostname and alert.event.kind. That the source data is present and accessible at that point is proven by the very next processor: the script_set_host_os_type_from_triggering_events painless script reads the same node.raw_audit_log_record.common.host_info.os_type and correctly sets host.os.type: linux on both events. The sibling foreach processors that lack an inner if (process.command_line, process.executable, process.name, source.ip) populate correctly, and event 1 (empty rawAuditLogRecord / null runtimeProgram.name) shows those fields simply omitted — confirming the drop-null script already handles missing values without needing the guard. The result is that these ECS host/action/file attribution fields ship silently unmapped for a security detection data stream, and related.hosts (which is appended from host.name) is consequently never populated either.
Recommendation:
Remove the inner if from the four triggering-event foreach processors so they behave like the working process.* / source.ip foreaches; empty values are already stripped by the trailing script_to_drop_null_values processor. For example, for cloud.instance.id:
- foreach:
field: wiz.defend_v2.triggering_events.nodes
tag: foreach_triggering_events_into_cloud_instance_id
if: ctx.wiz?.defend_v2?.triggering_events?.nodes instanceof List
processor:
append:
field: cloud.instance.id
tag: append_cloud_instance_id_from_triggering_events
value: '{{{_ingest._value.raw_audit_log_record.common.vm_external_id}}}'
allow_duplicates: false
on_failure:
- append:
field: error.message
tag: append_cloud_instance_id_failure_into_error_message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'Apply the same change to the host.name, file.path and event.action foreach processors, then regenerate the pipeline test expected file and confirm host.name, cloud.instance.id and event.action now appear on the sensor events.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
@mohitjha-elastic this looks legitimate. I don't see any of these fields in the generated samples. Please check.
| services: | ||
| wiz-audit: | ||
| image: docker.elastic.co/observability/stream:v0.22.0 | ||
| image: docker.elastic.co/observability/stream:v0.20.0 |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/wiz/_dev/deploy/docker/docker-compose.yml:4
docker-compose downgrades the elastic/stream mock image from v0.22.0 to v0.20.0 for every service; this looks like an unintended regression — keep v0.22.0.
Details
This PR changes docker.elastic.co/observability/stream from v0.22.0 to v0.20.0 on all eight services (wiz-audit, wiz-cloud_configuration_finding, wiz-cloud_configuration_finding_full_posture, wiz-issue, wiz-vulnerability, wiz-defend-no-auth, wiz-defend-basic, wiz-defend-token) as well as the newly added wiz-defend_v2 service. The change is unrelated to adding the defend_v2 data stream and downgrades the mock HTTP server used by system tests for the whole package, which appears accidental (e.g. copied from an older base). This is dev/test-only infrastructure and does not affect the shipped package, but the downgrade should be reverted to avoid an unintended regression in the test harness.
Recommendation:
Keep the existing stream image version on all services, including the new wiz-defend_v2 service:
wiz-defend_v2:
image: docker.elastic.co/observability/stream:v0.22.0
hostname: wiz-defend_v2
ports:
- 8090
volumes:
- ./files:/files:ro
environment:
PORT: '8090'
command:
- http-server
- --addr=:8090
- --config=/files/config-defend_v2.yml🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
TL;DRThe Remediation
Investigation detailsRoot CauseThe PR changes That downgrade is incompatible with the existing Wiz mock configs. For example, "lastDetectedAt": "{{ (now "-720h").Format "2006-01-02T15:04:05Z07:00" }}"In Evidence
Verification
What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
andrewkroh
added
the
dashboard
| - --addr=:8090 | ||
| - --config=/files/config-vulnerability.yml | ||
| wiz-defend_v2: | ||
| image: docker.elastic.co/observability/stream:v0.20.0 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/wiz/_dev/deploy/docker/docker-compose.yml:69
The new wiz-defend_v2 mock service is still pinned to stream image v0.20.0 while every other service was bumped to v0.22.0 in this same change; bump it to v0.22.0 for consistency.
Details
This PR reverted the stream mock image back to v0.22.0 for all six pre-existing services (wiz-audit, wiz-cloud_configuration_finding, wiz-cloud_configuration_finding_full_posture, wiz-issue, wiz-vulnerability, and the defend webhook services). The newly-added wiz-defend_v2 service was left at v0.20.0, so it is now the only service running an older mock image. This is an inconsistency introduced by this PR: the defend_v2 system test would run against a different stream version than every sibling data stream, which is the exact regression the version revert was intended to avoid.
Recommendation:
Align the new service with the rest of the file:
wiz-defend_v2:
image: docker.elastic.co/observability/stream:v0.22.0
hostname: wiz-defend_v2
ports:
- 8090
volumes:
- ./files:/files:ro
environment:
PORT: '8090'
command:
- http-server
- --addr=:8090
- --config=/files/config-defend_v2.yml🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
@mohitjha-elastic, please update this for consistency.
🚀 Benchmarks reportTo see the full report comment with |
| field: cloud.instance.id | ||
| tag: append_cloud_instance_id_from_triggering_events | ||
| value: '{{{_ingest._value.raw_audit_log_record.common.vm_external_id}}}' | ||
| if: ctx._ingest?._value?.raw_audit_log_record?.common?.vm_external_id != null |
There was a problem hiding this comment.
@mohitjha-elastic this looks legitimate. I don't see any of these fields in the generated samples. Please check.
1. Review comments by Dan - fix CEL code, add nav links to dashboards, and remove spaces from viz titles. 2. Review comments by Krishna - add new lines required, removed if conditions from the nested for loop, sorted readme logs section, removed orderBy filter from cel code, added more sample events in tests.
Remove blank space from viz title
|
✅ All changelog entries have the correct PR link. |
|
No issues across the latest commits 7c9db75. Review summaryIssues found across earlier commits f08f3e7 — 1 medium
Issues found across earlier commits 3e0f9a0 — 1 high, 1 low
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
|
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package wiz - 4.6.0 containing this change is available at https://epr.elastic.co/package/wiz/4.6.0/ |
4 participants
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
Related Issues