[Carbon Black Cloud] - Added alerts v7 data stream and CEL input type

[ShourieG]

Type of change

• Enhancement

Proposed commit message

Added a new alerts_v7 data stream to mirror the v7 api and schema changes from carbon black cloud.
The old data stream has not been removed in order to allow smooth transition to the new v7 data stream once the old api stops working. This avoids a breaking change. In addition to this a new CEL input option has been introduced to bring more flexibility for the future. This new input has been tagged as [Beta] while the existing HTTPJSON input has been tagged as [Legacy]. This was done in order to smoothen the transition overtime and avoid breaking changes.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

1. New alert v7 data stream added with tests.
2. Alert v7 Dashboards & Search added.
3. Documentation updated.
4. Added new [Beta] CEL input type for HTTP api dependant data streams. (alert_v7, asset_vulnerability_summary, audit)

Note

1. The existing HTTPJSON input is being marked as [Legacy] with the introduction of the CEL input.
2. Please ignore commits before March 26th. Start from (initial commit, new v7 data stream under construction (ignore for review)

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package stack up command.
Move to integrations/packages/carbon_black_cloud directory.
Run the following command to run tests.

1. elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build
2. elastic-package stack up -v -d --services package-registry
3. Bind the shell using the command - eval "$(elastic-package stack shellinit)" or eval $(elastic-package stack shellinit --shell bash) on mac m1
4. Use the command - elastic-package test -v to test

Related issues

• Closes [Carbon Black Cloud] API Updates #7997
• Related https://github.com/elastic/enhancements/issues/21160

Test Results

--- Test results for package: carbon_black_cloud - START ---
╭────────────────────┬─────────────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE            │ DATA STREAM                 │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────────────┼─────────────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ carbon_black_cloud │ alert                       │ system    │ default   │ PASS   │    32.715904s │
│ carbon_black_cloud │ alert_v7                    │ system    │ httpjson  │ PASS   │ 30.962756791s │
│ carbon_black_cloud │ alert_v7                    │ system    │ cel       │ PASS   │ 32.664833208s │
│ carbon_black_cloud │ asset_vulnerability_summary │ system    │ httpjson  │ PASS   │ 28.750989958s │
│ carbon_black_cloud │ asset_vulnerability_summary │ system    │ cel       │ PASS   │ 31.468396708s │
│ carbon_black_cloud │ audit                       │ system    │ httpjson  │ PASS   │ 32.579871041s │
│ carbon_black_cloud │ audit                       │ system    │ cel       │ PASS   │ 31.452701917s │
╰────────────────────┴─────────────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: carbon_black_cloud - END   ---

Screenshots

Add here screenshots presenting:

• Kibana UI forms presenting configuration options exposed by the integration:

• Dashboards with collected metrics or logs:

  Alerts V7 (Samples):

Asset Vulnerability:

Audit Logs:

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[ShourieG]

@efd6 addressed all the PR suggestions

[kcreddy]

Sorry for the late review. I saw it late that my name was tagged 😄 .

For the ones moving from httpjson to CEL, how does the upgrade process looks like in the UI, say from 1.21.3 to 2.0.0? Do they end up still using httpjson or migrate to CEL seamlessly or prompted to configure CEL input?

[ShourieG]

Sorry for the late review. I saw it late that my name was tagged 😄 .

For the ones moving from httpjson to CEL, how does the upgrade process looks like in the UI, say from 1.21.3 to 2.0.0? Do they end up still using httpjson or migrate to CEL seamlessly or prompted to configure CEL input?

@kcreddy So atm after upgrading to 2.0, httpjson is still there and if they have httpjson configured it will still stay active, CEL will be an additional input that can they will see and choose to migrate over if they wish to.

[ShourieG]

@efd6 @kcreddy, addressed all the PR suggestions and updated the docs & tests accordingly.

[kcreddy]

LGTM 👍🏼 for my comments.

[efd6]

Nits then LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7928643

History

• 💚 Build #10318 succeeded 56ff897
• 💚 Build #10275 succeeded a63aec4
• 💚 Build #10232 succeeded 031bf0f
• 💚 Build #10191 succeeded 7debf07
• 💚 Build #10019 succeeded f7ddad6
• 💚 Build #9948 succeeded 6d5429d

cc @ShourieG

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
92.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package carbon_black_cloud - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=carbon_black_cloud