[ti_recordedfuture] Download threat data gzipped and only when new

[chrisberkhout]

Proposed commit message

[ti_recordedfuture] Download threat data gzipped and only when new

Migrates from the HTTPJSON input to the CEL input so that we can
download threat intelligence data gzipped instead of uncompressed.

If the data received is not gzipped, it will still be successfully
parsed as CSV.

Data will only be downloaded when it has changed. A HEAD request is used
to check whether new data is available (whether it returns a new value
in the Etag header) before making a GET request to actually download the
data.

If a HEAD request is unsupported and returns a 4XX response code, the
GET request will proceed.

The `threat` data stream also supports setting a custom URL to download
custom fusion files. This will continue to work.

This is a breaking change, but users who had already activated the
playbook alert or triggered alert data streams will not need to reenter
an API key, they will only need to reactivate the threat intel item and
reenter any non-default settings they used.

This version has been tested against the live API.

Discussion

Can be read commit-by-commit.

See the screenshots below for details of manifest string changes and the upgrade UX.

I put a longer changelog description that should be helpful for users who have the upcoming breaking change callout functionality.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes [TI Recorded Future] Request gzip compressed CSV #6011

Screenshots - manifest string improvements

This shows the improvements made the to manifest strings. In particular, it makes it clear that the risklist data is threat intelligence data (it goes in the threat data stream and related transform).

Screenshots - upgrade user experience

Users of the current version (1.30.0) will have the following experience when the new version (2.0.0 ) becomes available.

They will see this "Upgrade" button and press it:

They will be taken to the configuration form. The HTTPJSON input is removed.

They will need to expand the "via API" section to see to see that threat intelligence is now deactivated. They can reactivate it, and reenter any non-default settings. They will not need to reenter an API key, as long as they were already collecting playbook alerts or triggered alerts. All three data streams now share the same top-level API key variable.

After pressing "Upgrade integration", there will be one one more download of threat intel data (now gzipped), then future downloads will only happen new new data is available.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package ti_recordedfuture 👍(2) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
triggered_alert	8620.69	6535.95	-2084.74 (-24.18%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

I think we should update the README to explain 'how it works.' It's important to clarify that it uses the ETag to detect changes and will not fetch again until there is new data to download.

[chrisberkhout]

I think we should update the README to explain 'how it works.' It's important to clarify that it uses the ETag to detect changes and will not fetch again until there is new data to download.

Done. I rewrote the top section of the README. I covered this point as follows:

The threat data stream will check whether the available data has changed before actually downloading it. A short interval setting will mean that it checks frequently, but each version of the data will only be ingested once.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 6de43fb

History

• 💚 Build #26936 succeeded 0ded9a9
• 💚 Build #26935 succeeded 3c4a138

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[kcreddy]

Thank you! LGTM for my comments.

[elastic-vault-github-plugin-prod]

Package ti_recordedfuture - 2.0.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/2.0.0/