[AbuseCH] Add Commercial API support for the ThreatFox data stream - #20134
Conversation
moxarth-rathod
added
Integration:ti_abusech
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
Elastic Docs Style Checker (Vale)Summary: 3 suggestions found 💡 Suggestions (3): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![vera-review-bot[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| title: URL | ||
| description: Base URL of the abuse.ch ThreatFox API to collect threat indicators. | ||
| description: >- | ||
| URL for the ThreatFox API. For the Community API, use the full endpoint URL. For the Commercial API, use the API base URL (for example `https://api.spamhaus.com`). |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/data_stream/threatfox/manifest.yml:12
The url description now instructs Commercial users to set a different base URL, but the url var remains show_user:false, so it is hidden by default while api_type is show_user:true — commercial users can't easily set the required endpoint.
Details
The url variable's description was updated to tell users that the Commercial API needs a different base URL (e.g. https://api.spamhaus.com), while the default remains the community endpoint https://threatfox-api.abuse.ch/api/v1/. However url still has show_user: false, so it is hidden under Advanced options, even though the newly-added api_type selector is show_user: true. A user who switches api_type to Commercial via the visible control gets no equally-visible way to change the URL, so the login request goes to the wrong host by default.
Recommendation:
Surface the URL alongside the API type selector so commercial users can set the base URL:
- name: url
type: text
title: URL
description: >-
URL for the ThreatFox API. For the Community API, use the full endpoint URL. For the Commercial API, use the API base URL (for example `https://api.spamhaus.com`).
multi: false
required: true
show_user: true
default: https://threatfox-api.abuse.ch/api/v1/🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
ja3_fingerprints |
23255.81 | 7788.16 | -15467.65 (-66.51%) | 💔 |
malware |
14492.75 | 7042.25 | -7450.5 (-51.41%) | 💔 |
malware |
14492.75 | 11904.76 | -2587.99 (-17.86%) | 💔 |
To see the full report comment with /test benchmark fullreport
andrewkroh
added
the
documentation
| title: URL | ||
| description: Base URL of the abuse.ch ThreatFox API to collect threat indicators. | ||
| description: >- | ||
| URL for the ThreatFox API. For the Community API, use the full endpoint URL. For the Commercial API, use the API base URL (for example `https://api.spamhaus.com`). |
There was a problem hiding this comment.
Let's keep the commercial vars shared at the cel input level rather than following the bot's suggestion to push them back to the data stream. The bot is optimising for today's single commercial stream, but per #18996 we're adding MalwareBazaar, URLhaus, and YARAify next and it'll keep growing — if these live at the data-stream level, users re-enter the same Spamhaus username/password on every stream (4+ times today, more later). Since all six streams share the one cel input, input-level vars are the "configure once" home (that's already where auth_key lives).
Concretely, at the shared cel input in packages/ti_abusech/manifest.yml:
api_type—select(community / commercial),required: true,show_user: trueauth_key—required: false,secret: true(community)username—required: false(commercial)password—required: false,secret: true(commercial)
On the bot's point that auth_key lost required: true: we can get conditional validation back without making it unconditionally required. required_vars (available at our format_version: 3.3.2, at the input level) does exactly this — same pattern elastic_security uses. Add it to the cel input:
required_vars:
community:
- name: api_type
value: community
- name: auth_key
commercial:
- name: api_type
value: commercial
- name: username
- name: passwordNow Fleet requires auth_key only when api_type == community, and username/password only when commercial — no unconditional required, no opaque runtime CEL failure. (Note: the vars referenced here must be required: false in their own definitions, or package-spec validation errors.)
On the URL: The community endpoints are already per-stream show_user: false and differ by host/path, so they must stay per-stream and hidden. For commercial, add one shared base URL at the input level, also show_user: false (adding a second user-visible URL field would confuse users), defaulting to https://api.spamhaus.com, and have the template append the fixed per-dataset path, routing on api_type:
That way users see no URL field in normal config for either mode — just API Type and the relevant credential.
Update/add the tests accordingly.
There was a problem hiding this comment.
@moxarth-rathod, can you move the other PRs of AbuseCH to draft until this design is finalised and merged.
Also can you check and comment on Vera bot reviews if required or not?
@kcreddy done |
| description: > | ||
| The number of sightings of the IOC. | ||
| - name: comment | ||
| type: keyword |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/fields.yml:42
abusech.threatfox.comment is declared as keyword in the transform fields but as match_only_text in the data stream fields; make both match_only_text so the latest destination index and the source data stream agree.
Details
This PR adds the same five commercial-API field blocks to two files: data_stream/threatfox/fields/fields.yml and elasticsearch/transform/latest_threatfox/fields/fields.yml. Every added field matches between the two files except abusech.threatfox.comment, which is match_only_text in the data stream (line 42 of data_stream/threatfox/fields/fields.yml) and keyword here. The latest_threatfox transform reads from logs-ti_abusech.threatfox-* and writes to logs-ti_abusech_latest.dest_threatfox-4, so the same field is mapped with two different types across the two indices. Any Kibana data view or query spanning both (for example logs-ti_abusech*) reports abusech.threatfox.comment as a conflicting field, which is exactly the failure mode the comment block at the top of latest_threatfox/transform.yml warns about ('prevent query time field type conflicts').
Recommendation:
Align the transform field definition with the data stream definition:
- name: comment
type: match_only_text
description: >
An optional comment from the reporter.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| type: keyword | ||
| description: | | ||
| The configured expiration duration. | ||
| - name: sightings |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/fields.yml:37
New commercial fields are added to the latest_threatfox transform mappings but fleet_transform_version is not bumped, so existing installs keep the old destination mappings; bump it to 0.8.0.
Details
This PR adds sightings, comment, is_compromised, credits.* and malware_samples.* to the latest_threatfox transform field definitions, but elasticsearch/transform/latest_threatfox/transform.yml is not part of this change and still carries fleet_transform_version: 0.7.0. That file documents the requirement itself: 'Bump this version to delete, reinstall, and restart the transform during package. Version bump is needed if there is any code change in transform.' Without the bump, upgrading installations do not reinstall the transform or its destination index mappings, so the new commercial fields land in logs-ti_abusech_latest.dest_threatfox-4 under dynamic mapping instead of the declared types (credits_amount and sightings as long, is_compromised as boolean, malware_samples.timestamp as date). Only the version suffix on dest.index should be left alone here, since adding fields is not a breaking mapping change.
Recommendation:
Bump the transform version in packages/ti_abusech/elasticsearch/transform/latest_threatfox/transform.yml:
_meta:
managed: true
# Bump this version to delete, reinstall, and restart the transform during package.
# Version bump is needed if there is any code change in transform.
fleet_transform_version: 0.8.0🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
@moxarth-rathod, this also needs a version bump so that new mappings are applied.
| processor: | ||
| rename: | ||
| tag: rename_malware_samples_time_stamp_to_timestamp | ||
| field: _ingest._value.time_stamp |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/data_stream/threatfox/elasticsearch/ingest_pipeline/default.yml:300
The pipeline normalizes the community malware_samples.time_stamp to timestamp but leaves malware_bazaar and malware_bazaar_url as two separate fields; rename malware_bazaar to malware_bazaar_url the same way.
Details
The new foreach at this line normalizes the community API's malware_samples[].time_stamp onto the commercial API's timestamp name, so both API types produce one field. The sibling URL field is not normalized: the community shape uses malware_bazaar and the commercial shape uses malware_bazaar_url, and this PR declares both names in fields.yml rather than merging them. The new pipeline test confirms the split - in test-commercial-fields.log-expected.json the first (commercial) document has abusech.threatfox.malware_samples[].malware_bazaar_url while the second (community) document has abusech.threatfox.malware_samples[].malware_bazaar, both holding a bazaar.abuse.ch sample URL. Consumers, dashboards and detection rules must therefore query two field names for the same value, and the normalization applied to the timestamp field is inconsistent with it.
Recommendation:
Add a matching rename next to the existing time_stamp normalization, and drop the malware_bazaar entry from both fields/fields.yml files:
- foreach:
tag: foreach_rename_malware_samples_malware_bazaar_to_malware_bazaar_url
description: Normalize community API malware_bazaar to commercial malware_bazaar_url field name.
field: abusech.threatfox.malware_samples
if: ctx.abusech?.threatfox?.malware_samples instanceof List
processor:
rename:
tag: rename_malware_samples_malware_bazaar_to_malware_bazaar_url
field: _ingest._value.malware_bazaar
target_field: _ingest._value.malware_bazaar_url
ignore_missing: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
@moxarth-rathod, can you confirm this normalisation is deliberately skipped?
There was a problem hiding this comment.
Declining for now. time_stamp and timestamp are clearly the same field, and we need to merge them so the date processor can handle both API formats. malware_bazaar and malware_bazaar_url may represent the same value, but the APIs use different fieldnames, and I don't want to assume they're equivalent without vendor confirmation. Keeping both fields documented separately is the safer approach for this PR.
| - name: api_type | ||
| value: community | ||
| - name: auth_key | ||
| commercial: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/manifest.yml:64
Selecting Commercial API stops Fleet requiring Auth Key, but the url and malware data streams stay enabled and community-only, so they break; add auth_key to the commercial required_vars group and say so in the api_type description.
Details
api_type is now a shared input-level variable, but only the threatfox and malwarebazaar stream templates branch on it. The url and malware templates (data_stream/url/agent/stream/cel.yml.hbs and data_stream/malware/agent/stream/cel.yml.hbs) unconditionally send the Auth-Key header and have no commercial path, and every data stream in this package ships enabled: true. Because the commercial required_vars group lists only api_type, username and password, a user who picks Commercial API is no longer prompted for auth_key, yet url and malware keep polling the community endpoints and are rejected without it. The api_type description ('Select the abuse.ch API to use') also does not indicate that the setting applies only to the ThreatFox and MalwareBazaar datasets.
Recommendation:
Keep the Auth Key required in the commercial group as well, since the community-only datasets still need it:
required_vars:
community:
- name: api_type
value: community
- name: auth_key
commercial:
- name: api_type
value: commercial
- name: username
- name: password
- name: auth_keyand clarify the scope in the api_type description:
description: Select the abuse.ch API to use for the ThreatFox and MalwareBazaar datasets. Community API uses an Auth Key. Commercial API uses Spamhaus credentials and JWT authentication. The URLhaus, MalwareBazaar payloads, SSL Blacklist and JA3 Fingerprints datasets always use the Community API and still require an Auth Key.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| ## Troubleshooting | ||
|
|
||
| - **Upgrading to v5.0.0**: Version 5.0.0 moves **API Type**, **Username (Commercial)**, and **Password (Commercial)** from the MalwareBazaar data stream to the shared integration input. Fleet does not carry these values across that scope change. If you configured MalwareBazaar Commercial API on 4.2.0, re-enter the credentials after upgrade: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/_dev/build/docs/README.md:112
The v5.0.0 upgrade note tells 4.2.0 commercial users to re-enter credentials but not to reset the MalwareBazaar URL, which they were told to point at api.spamhaus.com and which is now hidden; add that step.
Details
In 4.2.0 the MalwareBazaar url variable was documented as 'For the Commercial API, use the API base URL (for example https://api.spamhaus.com)', so commercial users were instructed to change it. This PR narrows that variable back to 'URL of the MalwareBazaar Community API', flips it to show_user: false, and routes the commercial base URL through the new package-level commercial_url variable instead. Fleet retains the previously configured url value on upgrade, so a 4.2.0 commercial MalwareBazaar user lands on 5.0.0 with api_type back at community and url still set to https://api.spamhaus.com; the community branch then POSTs to that host and fails. Because url is now show_user: false it is only reachable under Advanced options, so the fix is not discoverable from the steps as written.
Recommendation:
Add the URL reset to the v5.0.0 upgrade steps in _dev/build/docs/README.md (and regenerate docs/README.md):
5. Under the MalwareBazaar data stream, expand **Advanced options** and reset **URL** to `https://mb-api.abuse.ch/api/v1/` if you had changed it to `https://api.spamhaus.com` for the 4.2.0 Commercial API. The Commercial API base URL is now configured once at the integration level via **Commercial API URL**.
6. Select **Save integration**.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
Thanks for pointing this out.
| conditions: | ||
| kibana: | ||
| version: "^8.19.0 || ^9.1.0" | ||
| version: "^9.1.0" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/manifest.yml:12
5.0.0 narrows conditions.kibana.version from ^8.19.0 || ^9.1.0 to ^9.1.0, dropping Kibana 8.19 support, but no changelog entry or upgrade note mentions it; add a breaking-change entry.
Details
The kibana constraint drops the ^8.19.0 branch, so 5.0.0 and later are no longer installable or upgradable on 8.x stacks. The 5.0.0 changelog block records only one breaking change - moving API Type, Username and Password to the shared input - and the new 'Upgrading to v5.0.0' README section says nothing about the stack requirement. Users on 8.19 get no explanation for why the package stops offering updates. The constraint itself is consistent with the other packages in this repo that use required_vars (elastic_security, cloud_security_posture, cloud_asset_inventory all require Kibana 9.x), so the change looks intentional - it just is not documented.
Recommendation:
Record the dropped stack support alongside the existing breaking-change entry in changelog.yml:
- version: "5.0.0"
changes:
- description: |
Require Kibana 9.1.0 or later. The integration now uses input-level required_vars to enforce Community or Commercial credentials, which is not supported on 8.x, so Kibana 8.19 is no longer a supported stack version for this package.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20134🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
kcreddy
left a comment
kcreddy
left a comment
There was a problem hiding this comment.
@moxarth-rathod, check bot's comments: #20134 (comment), #20134 (comment), #20134 (comment)
Also make following changes as well. These code changes helps users with clear errors in case they configure it wrongly. Also few improvements to README.
1. Guard url when no Auth Key is set
--- a/packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs
+++ b/packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs
@@ -46,5 +46,16 @@
},
"want_more": false,
}
+ : !has(state.auth_key) ?
+ {
+ "events": {
+ "error": {
+ "code": "400",
+ "id": "configuration_error",
+ "message": "auth_key is not configured. The URLhaus API rejects unauthenticated requests. Generate an Auth Key in the abuse.ch authentication portal (https://auth.abuse.ch) and set the Auth Key (Community) setting.",
+ },
+ },
+ "want_more": false,
+ }
:
request("GET", state.url).with({2. Same guard for malware
This stream has no configuration guards at all today, so the request needs wrapping.
--- a/packages/ti_abusech/data_stream/malware/agent/stream/cel.yml.hbs
+++ b/packages/ti_abusech/data_stream/malware/agent/stream/cel.yml.hbs
@@ -24,4 +24,16 @@
program: |
state.with(
+ !has(state.auth_key) ?
+ {
+ "events": {
+ "error": {
+ "code": "400",
+ "id": "configuration_error",
+ "message": "auth_key is not configured. The URLhaus payloads API rejects unauthenticated requests. Generate an Auth Key in the abuse.ch authentication portal (https://auth.abuse.ch) and set the Auth Key (Community) setting.",
+ },
+ },
+ "want_more": false,
+ }
+ :
request("GET", state.url).with({
"Header":{
"Content-Type": ["application/json"],Note that malware renders no state: block at all when auth_key is unset, so the guard has to work against an empty state map. I checked it with mito and it takes the error branch correctly. Once the guard is in, the ?"Auth-Key" optional handling below it is unreachable-by-construction dead weight in both files, but simplifying that is a larger diff and can wait.
3. Scope the api_type description
--- a/packages/ti_abusech/manifest.yml
+++ b/packages/ti_abusech/manifest.yml
@@ -70,7 +70,8 @@
- name: api_type
type: select
title: API Type
- description: Select the abuse.ch API to use. Community API uses an Auth Key. Commercial API uses Spamhaus credentials and JWT authentication.
+ description: >-
+ Select the abuse.ch API to use for the ThreatFox and MalwareBazaar datasets. Community API uses an Auth Key. Commercial API uses Spamhaus credentials and JWT authentication. The Malware URLs and Malware payloads datasets always use the Community API and require an Auth Key regardless of this setting.
multi: false
required: falseOne correction to the recommendation as written: sslblacklist and ja3_fingerprints do not send an Auth-Key header — both templates are a bare get(state.url) — so only url and malware need the key, and the description shouldn't claim otherwise.
4. Explain it in the README (also in the review comments below)
--- a/packages/ti_abusech/_dev/build/docs/README.md
+++ b/packages/ti_abusech/_dev/build/docs/README.md
@@ -36,8 +36,11 @@
### From abuse.ch
-Authentication depends on which API you use:
+Which credentials you need depends on which datasets you enable, not on the integration as a whole:
-- **Community API**: requires an `Auth Key` (API key). Any requests made without this key will be rejected by the abuse.ch community APIs.
-- **Commercial API**: requires Spamhaus username and password credentials. The integration uses these to obtain a short-lived JWT for API requests.
+- **ThreatFox threat indicators** (`threatfox`) and **MalwareBazaar payloads** (`malwarebazaar`) can use either API, selected with the shared **API Type** setting. The Community API requires an `Auth Key`. The Commercial API requires Spamhaus username and password credentials, which the integration exchanges for a short-lived JWT.
+- **Malware URLs** (`url`) and **Malware payloads** (`malware`) always query the URLhaus Community API and require an `Auth Key` regardless of **API Type**. Requests without the key are rejected.
+- **SSL Blacklisted Certificates** (`sslblacklist`) and **JA3 Fingerprints** (`ja3_fingerprints`) read the SSLBL feeds and use neither credential.
+
+Because **API Type** applies to the whole integration, selecting **Commercial API** does not remove the need for an `Auth Key` unless you also disable **Malware URLs** and **Malware payloads**.
#### Obtain `Auth Key` (Community API)The Setup step has the same blind spot — it currently reads as if Commercial API replaces the Auth Key:
--- a/packages/ti_abusech/_dev/build/docs/README.md
+++ b/packages/ti_abusech/_dev/build/docs/README.md
@@ -86,6 +86,6 @@
* To **Collect abuse.ch logs via API**, you'll need to:
- - Configure **Auth Key** for Community API datasets.
- - For Commercial API collection, set **API Type** to **Commercial API** and configure **Username** and **Password**.
+ - Configure **Auth Key (Community)**. The **Malware URLs** and **Malware payloads** datasets always need it, and **ThreatFox threat indicators** and **MalwareBazaar payloads** need it when **API Type** is **Community API**.
+ - To use the Commercial API for **ThreatFox threat indicators** and **MalwareBazaar payloads**, set **API Type** to **Commercial API** and configure **Username (Commercial)** and **Password (Commercial)**.
- Enable/Disable the required datasets.
- For each dataset, adjust the integration configuration parameters if required, including the URL, Interval, etc. to enable data collection.Follow-ups after applying
elastic-package test policy -g -d url,malwareto refresh the policy expectations, since the rendered programs change.elastic-package buildto regeneratedocs/README.mdfrom_dev/build/docs/README.md.- A changelog line for 5.0.0, something like "Emit a configuration error when the Auth Key is not set for the Malware URLs and Malware payloads datasets."
- Optional but consistent with what
urlalready does: amissing_auth_key.txtscript test alongsidedeprecated_url.txtandinterval_too_long.txt, proving the guard fires. - Update commit message accordingly
| { | ||
| "events": { | ||
| "error": { | ||
| "code": 400, |
There was a problem hiding this comment.
| "code": 400, | |
| "code": "400", |
This field is keyword
|
🟢 Reviewed the latest commits f98d126 — nothing new beyond already posted comments. Review summaryIssues found across earlier commits d47d898 — 1 low
Issues found across earlier commits e6e6105 — 2 lowIssues found across earlier commits e864342 — 2 high, 4 medium
Issues found across earlier commits 4ba3db9 — 1 high, 4 medium
Issues found across earlier commits 9e8902a — 3 high, 3 medium, 1 low
Issues found across earlier commits 35e30bf…a138839 (75 commits) — 1 high, 2 medium
Issues found across earlier commits 00f5a3f…a17330c (3 commits) — 5 medium, 1 low
Issues found across earlier commits 8ce4362 — 1 medium
Issues found across earlier commits 1e37f63…22d993d (3 commits) — 1 low
Issues found across earlier commits dc62db4 — 2 medium, 3 low
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
|
| ?"Auth-Key": has(state.auth_key) ? | ||
| optional.of([state.auth_key]) | ||
| : | ||
| optional.none(), |
There was a problem hiding this comment.
| ?"Auth-Key": has(state.auth_key) ? | |
| optional.of([state.auth_key]) | |
| : | |
| optional.none(), | |
| ?"Auth-Key": state.?auth_key.optMap(v, [v]), |
(check this; it may not be available on this stack version)
| ).as(req, | ||
| req.do_request().as(resp, resp.StatusCode == 200 ? |
There was a problem hiding this comment.
| ).as(req, | |
| req.do_request().as(resp, resp.StatusCode == 200 ? | |
| ).do_request().as(resp, resp.StatusCode == 200 ? |
and clean up. The binding was not necessary.
| body.items.size() > 0 ? | ||
| optional.of( | ||
| ( | ||
| has(state.?cursor.first_seen) ? | ||
| max(timestamp(state.cursor.first_seen), body.items.map(ind, timestamp(ind.first_seen)).max()) | ||
| : | ||
| body.items.map(ind, timestamp(ind.first_seen)).max() | ||
| ).format(time_layout.RFC3339) | ||
| ) | ||
| : | ||
| state.?cursor.first_seen | ||
| ).as(first_seen, |
There was a problem hiding this comment.
You don't need to bind this. It's only used once.
There was a problem hiding this comment.
Thanks @moxarth-rathod. LGTM for my comments.
Please wait for @efd6 approval.
|
Tick the box to add this pull request to the merge queue (same as
|
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
|
Package ti_abusech - 5.0.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/5.0.0/ |
5 participants
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
Related issues