crowdstrike: add pipeline processor tags via elastic-package and expose in on_failure

[navnit-elastic]

Proposed commit message

crowdstrike: add pipeline processor tags via elastic-package and expose in on_failure

- Run `elastic-package modify -m pipeline-tag` to add a `tag` key to
  each processor in ingest pipelines (part of integration quality
  phase-1 improvements).
- Update on_failure append to error.message to include
  _ingest.on_failure_processor_tag so pipeline failures can be traced 
  by processor tag.

Note

This PR follows: #17435

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Tested all 4 packages' processors now have tags:

for pkg in ti_abusech o365 m365_defender sentinel_one; do
  find "packages/$pkg" -path '*/data_stream/*/elasticsearch/ingest_pipeline/*.yml' -type f 2>/dev/null | while read -r f; do
    missing=$(yq '.processors[] | select((to_entries[0].value | has("tag")) | not) | to_entries[0].key' "$f" 2>/dev/null)
    echo "=== $f ==="
    if [ -n "$missing" ]; then
      data_stream=$(echo "$f" | sed 's|.*/data_stream/\([^/]*\)/.*|\1|')
      pipeline=$(basename "$f" .yml)
      echo "data stream: $data_stream, pipeline: $pipeline"
      echo "$missing"
    else
      echo "all processors have tags"
    fi
    echo ""
  done
done

Output

=== packages/crowdstrike/data_stream/host/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/alert/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/categorize.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/fim_rule_matched.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/epp_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/data_protection_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/cspm_ioa.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/outbound_network.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/cspm_iom.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/inbound_network.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/incident_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/user_activity_audit.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/epp_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/auth_activity_audit.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/data_protection_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/recon_notification_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/remote_response_session_start.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/identity_protection_incident.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/firewall_match.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/customer_ioc_event.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/xdr_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/remote_response_session_end.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/ipd_detection_summary.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/scheduled_report_notification_event.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/cspm_events.yml ===
all processors have tags

=== packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/mobile_detection_summary.yml ===
all processors have tags

Related issues

• Relates Integration quality: (Phase 1) Add processor tags to reflect failing processor #17436

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[andrewkroh]

• Run elastic-package modify -m pipeline-tag to add a tag key to
  each processor in ingest pipelines (part of integration quality
  phase-1 improvements).
• Update on_failure append to error.message to include
  _ingest.on_failure_processor_tag so pipeline failures can be traced
  by processor tag.

Are these not the same thing (adding 'tag' to processors)? The first bullet encompasses the second? Or does it mean something else?

[navnit-elastic]

• Run elastic-package modify -m pipeline-tag to add a tag key to
  each processor in ingest pipelines (part of integration quality
  phase-1 improvements).
• Update on_failure append to error.message to include
  _ingest.on_failure_processor_tag so pipeline failures can be traced
  by processor tag.

Are these not the same thing (adding 'tag' to processors)? The first bullet encompasses the second? Or does it mean something else?

They are two different steps. The first bullet adds the tag key to the processors, while the second ensures that if a failure occurs, that specific tag is recorded in the error.message.

I hadn't included the change to add _ingest.on_failure_processor_tag to the on_failure block, I have now added it in this commit: a49ec0f.

[navnit-elastic]

That commit was causing the pipeline test to fail because of the sample data that is intended to fail for convert_crowdstrike_aip_to_ip_c775b545 processor (e.g. empty value for aip).

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

The changes LGTM. It's surprising how much trailing white space existed in these files.

Should we also be ensuring that every pipelines appends tag preserve_original_event in on_failure? Or was that already checked. Ref: wiki Have this helps us correct the errors when they occur.

[navnit-elastic]

Should we also be ensuring that every pipelines appends tag preserve_original_event in on_failure? Or was that already checked. Ref: wiki Have this helps us correct the errors when they occur.

Agreed, that will help. I’m not sure whether that change should go in this PR or a separate one (since the scope of the issue is limited to processor tags only), happy to do whichever you prefer.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e604a7c

History

• 💚 Build #39048 succeeded db77cab
• 💔 Build #39032 failed a49ec0f
• 💚 Build #39029 succeeded db77cab
• 💔 Build #39001 failed b5a1059
• 💚 Build #38344 succeeded a0fdaa0

cc @navnit-elastic

[andrewkroh]

The addition of preserve_original_event tagging on failure can be separate.

[navnit-elastic]

Thanks for the clarification. I've opened an issue for addition of preserve_original_event tag on failure: #17690.

[elastic-vault-github-plugin-prod]

Package crowdstrike - 3.7.0 containing this change is available at https://epr.elastic.co/package/crowdstrike/3.7.0/