ti_rapid7_threat_command: Add IOC expiration by kcreddy · Pull Request #9925 · elastic/integrations

kcreddy · 2024-05-21T09:21:42Z

Proposed commit message

See title

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Verify if vulnerability alerts get triggered from the new rule [New Rule] Rapid7 Threat Command CVEs Correlation detection-rules#3718
Update README on how to avoid duplication (rules, alerts)
Must be merged after [Rule Tuning]: Fix threat_index and filters in Rapid7 CVE rule detection-rules#3800

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

--- Test results for package: ti_rapid7_threat_command - START ---
╭──────────────────────────┬───────────────┬───────────┬───────────────────────────────┬────────┬──────────────╮
│ PACKAGE                  │ DATA STREAM   │ TEST TYPE │ TEST NAME                     │ RESULT │ TIME ELAPSED │
├──────────────────────────┼───────────────┼───────────┼───────────────────────────────┼────────┼──────────────┤
│ ti_rapid7_threat_command │ alert         │ pipeline  │ test-alert-event.json         │ PASS   │   2.722958ms │
│ ti_rapid7_threat_command │ ioc           │ pipeline  │ test-ioc-event.json           │ PASS   │     6.5205ms │
│ ti_rapid7_threat_command │ vulnerability │ pipeline  │ test-vulnerability-event.json │ PASS   │   2.470875ms │
╰──────────────────────────┴───────────────┴───────────┴───────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_rapid7_threat_command - END   ---
Done

elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v

...
2024/06/21 11:11:28 DEBUG found 2 hits in logs-ti_rapid7_threat_command.alert-ep data stream
2024/06/21 11:12:14 DEBUG found 4 hits in logs-ti_rapid7_threat_command.ioc-ep data stream
2024/06/21 11:13:02 DEBUG found 3 hits in logs-ti_rapid7_threat_command.vulnerability-ep data stream
...
--- Test results for package: ti_rapid7_threat_command - START ---
╭──────────────────────────┬───────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE                  │ DATA STREAM   │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────────────────────┼───────────────┼───────────┼───────────┼────────┼───────────────┤
│ ti_rapid7_threat_command │ alert         │ system    │ default   │ PASS   │ 33.193950833s │
│ ti_rapid7_threat_command │ ioc           │ system    │ default   │ PASS   │ 30.954459875s │
│ ti_rapid7_threat_command │ vulnerability │ system    │ default   │ PASS   │ 33.032822708s │
╰──────────────────────────┴───────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: ti_rapid7_threat_command - END   ---
Done

Related issues

Closes [TI_Rapid7_Threat_Command] Add support for IOC expiration #7360

Screenshots

Updated dashboards (querying only on the newly created destination indices):

1. Alerts Overview

2. Vulnerability Overview

3. IOC Overview

elasticmachine · 2024-06-18T07:09:43Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

… intg upgrade)

This ensures that something is broken to the users. A required value must be set to proceed further

The transform filters out any doc with error message, so error shouldn't be written. It would be nice to capture this in some comments field though

elasticmachine · 2024-06-20T13:17:48Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2024-06-21T05:47:48Z

+      - name: ioc_expiration_duration
+        type: text
+        title: IOC Expiration Duration
+        multi: false
+        required: true
+        show_user: true
+        description: >-
+          Enforces all active IOCs to expire after this duration since their last seen time indicated in the feed. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g `10d`). If invalid units are provided, default value `90d` i.e., 90 days is used. Check `README` for more details how IOC expiration works and removal of custom transforms and views used in older versions.


Intentionally didn't add any default, so that user checks on IOC expiration and removes existing transforms, and rules. If we use default, the users may not notice the major changes during the upgrade.

kcreddy · 2024-06-21T08:07:28Z

To be merged after elastic/detection-rules#3800 is out next week

elasticmachine · 2024-06-26T07:20:13Z

💚 Build Succeeded

Buildkite Build
Commit: fb62254

History

💔 Build #12890 failed 69bc1e9
💚 Build #12700 succeeded a2a8cc7
💚 Build #12686 succeeded 7ee879d
💚 Build #12672 succeeded 87f7965
💚 Build #12669 succeeded ba1c36b

cc @kcreddy

elastic-sonarqube · 2024-06-26T07:20:15Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-06-26T07:44:06Z

Package ti_rapid7_threat_command - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=ti_rapid7_threat_command

kcreddy added 2 commits May 21, 2024 14:46

Add indicator expiry for ioc datastream

1a3a26a

Merge remote-tracking branch 'upstream/main' into rapid7_ioc_expire

0534097

kcreddy added enhancement New feature or request Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) labels May 21, 2024

kcreddy self-assigned this May 21, 2024

kcreddy added 6 commits May 21, 2024 16:22

Add unique alerts transform

093c9f2

Add unique vulnerability transform

907ef09

Update Overview dashboards

a67d921

Remove index patterns

e62d2d0

Remove other dashboards and references

a3faebb

fix pipeline tests

3d2e964

kcreddy mentioned this pull request May 30, 2024

[New Rule] Rapid7 Threat Command CVEs Correlation elastic/detection-rules#3718

Merged

Merge remote-tracking branch 'upstream/main' into rapid7_ioc_expire

e599530

update dashboards

1c829c3

kcreddy mentioned this pull request Jun 20, 2024

[Rule Tuning]: Fix threat_index and filters in Rapid7 CVE rule elastic/detection-rules#3800

Merged

kcreddy added 5 commits June 20, 2024 12:39

Add filter on source

ba1c36b

Add ioc index filter to avoid previous source indices (present during…

87f7965

… intg upgrade)

Remove default on duration

751289c

This ensures that something is broken to the users. A required value must be set to proceed further

Remove error on invalid duration

d6a18ef

The transform filters out any doc with error message, so error shouldn't be written. It would be nice to capture this in some comments field though

Update README

7ee879d

kcreddy marked this pull request as ready for review June 20, 2024 13:17

kcreddy requested a review from a team as a code owner June 20, 2024 13:17

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jun 20, 2024

efd6 reviewed Jun 21, 2024

View reviewed changes

Address PR comments and rerun tests

a2a8cc7

kcreddy commented Jun 21, 2024

View reviewed changes

kcreddy requested a review from efd6 June 21, 2024 06:11

efd6 approved these changes Jun 21, 2024

View reviewed changes

kcreddy added 2 commits June 26, 2024 12:14

Merge remote-tracking branch 'upstream/main' into rapid7_ioc_expire

69bc1e9

Update README and delete old transform files

fb62254

kcreddy merged commit 3da25c3 into elastic:main Jun 26, 2024

kcreddy deleted the rapid7_ioc_expire branch February 7, 2025 09:11

Navigation Menu

Uh oh!

ti_rapid7_threat_command: Add IOC expiration - #9925

ti_rapid7_threat_command: Add IOC expiration#9925
kcreddy merged 18 commits into
elastic:mainfrom
kcreddy:rapid7_ioc_expire

kcreddy commented May 21, 2024 •

edited

Loading

elasticmachine commented Jun 18, 2024 •

edited

Loading

elasticmachine commented Jun 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy Jun 21, 2024

kcreddy commented Jun 21, 2024

elasticmachine commented Jun 26, 2024

elastic-sonarqube Bot commented Jun 26, 2024

elasticmachine commented Jun 26, 2024

Labels

3 participants

Uh oh!

Conversation

kcreddy commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

elasticmachine commented Jun 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy Jun 21, 2024

Choose a reason for hiding this comment

kcreddy commented Jun 21, 2024

elasticmachine commented Jun 26, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Jun 26, 2024

Quality Gate passed

elasticmachine commented Jun 26, 2024

Labels

3 participants

kcreddy commented May 21, 2024 •

edited

Loading

elasticmachine commented Jun 18, 2024 •

edited

Loading