Skip to content
Merged
35 changes: 32 additions & 3 deletions packages/ti_abusech/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,15 @@

### From abuse.ch

abuse.ch requires an `Auth Key` (API key) for request authentication. Any requests made without this key will be rejected by the abuse.ch APIs.
Which credentials you need depends on which datasets you enable, not on the integration as a whole:

#### Obtain `Auth Key`
- **ThreatFox threat indicators** (`threatfox`) and **MalwareBazaar payloads** (`malwarebazaar`) can use either API, selected with the shared **API Type** setting. The Community API requires an **Auth Key**. The Commercial API requires Spamhaus username and password credentials, which the integration exchanges for a short-lived JWT.
- **Malware URLs** (`url`) and **Malware payloads** (`malware`) always query the URLhaus Community API and require an **Auth Key** regardless of **API Type**. Requests without the key are rejected.
- **SSL Blacklisted Certificates** (`sslblacklist`) and **JA3 Fingerprints** (`ja3_fingerprints`) read the SSLBL feeds and use neither credential.

Because **API Type** applies to the whole integration, selecting **Commercial API** does not remove the need for an **Auth Key** unless you also disable **Malware URLs** and **Malware payloads**.

#### Obtain `Auth Key` (Community API)

1. Sign up for a new account, or login into the [abuse.ch authentication portal](https://auth.abuse.ch).
2. Connect with at least one authentication provider: Google, Github, X, or LinkedIn.
Expand All @@ -47,6 +53,17 @@

For more details, check the abuse.ch [Community First - New Authentication](https://abuse.ch/blog/community-first/) blog.

#### Obtain Commercial API credentials

Commercial API access uses JWT authentication. Create credentials in the Spamhaus Customer Portal, then configure the username and password in the integration. The integration authenticates to `/v1/login` and refreshes the JWT as needed.

1. Log in to the Spamhaus [Customer Portal](https://portal.spamhaus.com).
2. Navigate to **Product** > **abuse.ch API**.
3. Under **Generate new credentials for JWT authentication**, fill out the required information and follow the on-screen instructions.
4. Copy the generated **username** and **password**.

For more details, check the abuse.ch commercial API documentation on [JWT authentication for endpoints available to query](https://abusech.docs.spamhaus.com/api-reference#description/jwt-authentication-for-endpoints-available-to-query).

Check notice on line 65 in packages/ti_abusech/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint user-facing content

Elastic.WordChoice: Consider using 'misuse' instead of 'abuse', unless the term is in the UI.

## How do I deploy this integration?

This integration supports both Elastic Agentless-based and Agent-based installations.
Expand All @@ -71,7 +88,8 @@

* To **Collect abuse.ch logs via API**, you'll need to:

- Configure **Auth Key**.
- Configure **Auth Key (Community)**. The **Malware URLs** and **Malware payloads** datasets always need it, and **ThreatFox threat indicators** and **MalwareBazaar payloads** need it when **API Type** is **Community API**.
- To use the Commercial API for **ThreatFox threat indicators** and **MalwareBazaar payloads**, set **API Type** to **Commercial API** and configure **Username (Commercial)** and **Password (Commercial)**.
- Enable/Disable the required datasets.
- For each dataset, adjust the integration configuration parameters if required, including the URL, Interval, etc. to enable data collection.

Expand All @@ -94,6 +112,15 @@

## Troubleshooting

- **Upgrading to v5.0.0**: Version 5.0.0 moves **API Type**, **Username (Commercial)**, and **Password (Commercial)** from the MalwareBazaar data stream to the shared integration input. Fleet does not carry these values across that scope change. If you configured MalwareBazaar Commercial API on 4.2.0, re-enter the credentials after upgrade:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/_dev/build/docs/README.md:112

The v5.0.0 upgrade note tells 4.2.0 commercial users to re-enter credentials but not to reset the MalwareBazaar URL, which they were told to point at api.spamhaus.com and which is now hidden; add that step.

Details

In 4.2.0 the MalwareBazaar url variable was documented as 'For the Commercial API, use the API base URL (for example https://api.spamhaus.com)', so commercial users were instructed to change it. This PR narrows that variable back to 'URL of the MalwareBazaar Community API', flips it to show_user: false, and routes the commercial base URL through the new package-level commercial_url variable instead. Fleet retains the previously configured url value on upgrade, so a 4.2.0 commercial MalwareBazaar user lands on 5.0.0 with api_type back at community and url still set to https://api.spamhaus.com; the community branch then POSTs to that host and fails. Because url is now show_user: false it is only reachable under Advanced options, so the fix is not discoverable from the steps as written.

Recommendation:

Add the URL reset to the v5.0.0 upgrade steps in _dev/build/docs/README.md (and regenerate docs/README.md):

    5. Under the MalwareBazaar data stream, expand **Advanced options** and reset **URL** to `https://mb-api.abuse.ch/api/v1/` if you had changed it to `https://api.spamhaus.com` for the 4.2.0 Commercial API. The Commercial API base URL is now configured once at the integration level via **Commercial API URL**.
    6. Select **Save integration**.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moxarth-rathod, this is worth doing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out.

1. In Kibana, navigate to **Fleet** > **Agent policies**.
2. Select the policy containing the abuse.ch integration.

Check notice on line 117 in packages/ti_abusech/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint user-facing content

Elastic.WordChoice: Consider using 'misuse' instead of 'abuse', unless the term is in the UI.
3. Edit the abuse.ch integration.

Check notice on line 118 in packages/ti_abusech/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint user-facing content

Elastic.WordChoice: Consider using 'misuse' instead of 'abuse', unless the term is in the UI.
4. Set **API Type** to **Commercial API**, then enter **Username (Commercial)** and **Password (Commercial)** at the integration level (not under the MalwareBazaar data stream).
5. Under the MalwareBazaar data stream, expand **Advanced options** and reset **URL** to `https://mb-api.abuse.ch/api/v1/` if you had changed it to `https://api.spamhaus.com` for the 4.2.0 Commercial API. The Commercial API base URL is now configured once at the integration level via **Commercial API URL**.
6. Select **Save integration**.

Community API users who only use an Auth Key are unaffected. After this change, the same Spamhaus credentials apply to all commercial datasets (MalwareBazaar, ThreatFox, and future commercial data streams).
- **Upgrading to v4.0.0**: Version 4.0.0 switches the URL data stream from the full export ZIP endpoint (`/downloads/json`) to the incremental JSON API (`/v1/urls/recent/`). When upgrading from a previous version, the URL setting in your integration policy retains the old value and must be updated manually:
1. In Kibana, navigate to **Fleet** > **Agent policies**.
2. Select the policy containing the abuse.ch integration.
Expand All @@ -112,6 +139,8 @@
All the abusec.ch API errors are captured inside the `error` fields.
1. abuse.ch APIs return HTTP status `403 Forbidden` when the Auth Key is invalid. In such case, the `error.message` field is populated with message `query_status: unknown_auth_key` and `error.id` with `403 Forbidden`. To fix this, you need to regenerate the Auth Key in the [abuse.ch authentication portal](https://auth.abuse.ch/) and update the integration policy with newly generated Auth Key.
2. abuse.ch APIs return HTTP status `500 Internal Server Error` when experiencing problem on the abuse.ch service. In such case, `error.message` field is populated with message `POST:500 Internal Server Error (500)` and `error.id` with `500 Internal Server Error`. This is likely a one-off scenario and the ingestion should resume normally in the subsequent request.
3. When **API Type** is **Community API** and **Auth Key (Community)** is not configured, MalwareBazaar and ThreatFox log `api_type is 'community' but auth_key is not configured` in `error.message` and `configuration_error` in `error.id`. Add **Auth Key (Community)** at the integration level.
4. When **API Type** is **Commercial API** and **Username (Commercial)** or **Password (Commercial)** is not configured, MalwareBazaar and ThreatFox log `api_type is 'commercial' but username/password are not configured` in `error.message` and `configuration_error` in `error.id`. Configure both fields at the integration level.
- Since this integration supports the expiration of Indicators of Compromise (IoCs) using Elastic latest transform, the threat indicators are present in both source and destination indices. While this may appear to be duplicate ingestion, it is an implementation detail necessary for properly expiring threat indicators.
- Because the latest copy of threat indicators is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly. The ILM policies on source indices can be tuned to manage their data retention period.
- For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
Expand Down
56 changes: 41 additions & 15 deletions packages/ti_abusech/_dev/deploy/docker/files/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,17 @@ rules:
- "Bearer test-commercial-token"
responses:
- status_code: 200
headers:
Content-Type:
- "application/json"
body: |-
{
"items": [
{
"anonymous": false,
"comment": "Commercial API test sample",
"comment": "Before token refresh",
"delivery_method": "email_attachment",
"file_name": "7a6c03013a2f2ab8b9e8e7e5d226ea89e75da72c1519e78fd28b2253ea755c28.exe",
"file_name": "before-refresh.exe",
"file_size": 432640,
"file_type": "exe",
"file_type_mime": "application/x-dosexec",
Expand All @@ -52,8 +55,10 @@ rules:
"tlsh": "T13794242864BFC05994E3EEA12DDCA8FBD99A55E3640C743301B4633B8B52B84DE4F479"
},
{
"anonymous": true,
"file_name": "657edf01cd9100a52abd7a9d8e585a28",
"anonymous": false,
"comment": "After token refresh",
"delivery_method": "web_download",
"file_name": "after-refresh.exe",
"file_size": 414720,
"file_type": "exe",
"file_type_mime": "application/x-dosexec",
Expand All @@ -76,6 +81,7 @@ rules:
},
{
"anonymous": false,
"delivery_method": "web_download",
"file_name": "dummy_malware.exe",
"file_size": 512345,
"file_type": "exe",
Expand Down Expand Up @@ -108,18 +114,16 @@ rules:
request_headers:
Content-Type: "application/json"
Auth-Key: "test_auth_key"
body:
query: "get_iocs"
days: /[1-7]+/
request_body: '/.*"query": "get_iocs".*"days": [1-7]+.*/'
responses:
- status_code: 200
body: |-
{
"query_status": "ok",
"data": [
{"id":"841537","ioc":"wizzy.hopto.org","threat_type":"botnet_cc","threat_type_desc":"Indicator that identifies a botnet command&control server (C&C)","ioc_type":"domain","ioc_type_desc":"Domain that is used for botnet Command&control (C&C)","malware":"win.asyncrat","malware_printable":"AsyncRAT","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.asyncrat","confidence_level":100,"first_seen":"2022-08-05 19:43:08 UTC","last_seen":null,"reference":"https://tria.ge/220805-w57pxsgae2","reporter":"AndreGironda","tags":["asyncrat"]},
{"id":"839586","ioc":"872ff530d50579ae6bdc7cb4d658324b1d0e7a3e","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.vidar","malware_printable":"Vidar","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.vidar","confidence_level":75,"first_seen":"2022-07-25 22:27:09 UTC","last_seen":null,"reference":"","reporter":"crep1x","tags":["Vidar"]},
{"id":"839587","ioc":"a3b5c6d7e8f9g0h1i2j3k4l5m6n7o8p9q0r1s2t3","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.redline","malware_printable":"RedLine","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.redline","confidence_level":80,"first_seen":"2025-03-15 10:15:00 UTC","last_seen":null,"reference":"","reporter":"cyberhunter","tags":["RedLine"]}
{"id":"841537","ioc":"wizzy.hopto.org","threat_type":"botnet_cc","threat_type_desc":"Indicator that identifies a botnet command&control server (C&C)","ioc_type":"domain","ioc_type_desc":"Domain that is used for botnet Command&control (C&C)","malware":"win.asyncrat","malware_printable":"AsyncRAT","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.asyncrat","confidence_level":100,"first_seen":"2022-08-05 19:43:08 UTC","last_seen":null,"reference":"https://tria.ge/220805-w57pxsgae2","reporter":"test_reporter","tags":["asyncrat"]},
{"id":"839586","ioc":"872ff530d50579ae6bdc7cb4d658324b1d0e7a3e","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.vidar","malware_printable":"Vidar","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.vidar","confidence_level":75,"first_seen":"2022-07-25 22:27:09 UTC","last_seen":null,"reference":"","reporter":"test_reporter_2","tags":["Vidar"]},
{"id":"839587","ioc":"a3b5c6d7e8f9g0h1i2j3k4l5m6n7o8p9q0r1s2t3","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.redline","malware_printable":"RedLine","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.redline","confidence_level":80,"first_seen":"2025-03-15 10:15:00 UTC","last_seen":null,"reference":"","reporter":"test_reporter_3","tags":["RedLine"]}
]
}
- path: /v1/payloads/recent/
Expand Down Expand Up @@ -195,7 +199,7 @@ rules:
"spamhaus_dbl": "not listed",
"surbl": "not listed"
},
"reporter": "tammeto",
"reporter": "test_reporter",
"larted": "true",
"tags": null
},
Expand All @@ -211,7 +215,7 @@ rules:
"spamhaus_dbl": "not listed",
"surbl": "not listed"
},
"reporter": "geenensp",
"reporter": "test_reporter_2",
"larted": "true",
"tags": [
"32-bit",
Expand All @@ -237,8 +241,8 @@ rules:
Content-Type: "application/x-www-form-urlencoded"
Auth-Key: "test_auth_key"
query_params:
query: "get_recent"
selector: "time"
query: ["get_recent"]
selector: ["time"]
responses:
- status_code: 200
body: |-
Expand Down Expand Up @@ -287,7 +291,7 @@ rules:
"file_size": 414720,
"file_type_mime": "application/x-dosexec",
"file_type": "exe",
"reporter": "zbetcheckin",
"reporter": "test_reporter_2",
"origin_country": "FR",
"anonymous": 0,
"signature": "Loki",
Expand Down Expand Up @@ -395,3 +399,25 @@ rules:
d6f04b5a910115f4b50ecec09d40a1df,2017-07-15 19:42:24,2018-10-14 08:12:51,Dridex
35c0a31c481927f022a3b530255ac080,2017-07-15 19:43:19,2021-04-10 12:54:04,Tofsee
e330bca99c8a5256ae126a55c4c725c5,2017-07-15 19:59:29,2021-01-13 00:29:37,Adware
- path: /threatfox/v1/iocs
methods: ["GET"]
query_params:
since: ["{since:.*}"]
page_size: ["{page_size:.*}"]
request_headers:
Authorization:
- "Bearer test-commercial-token"
responses:
- status_code: 200
body: |-
{
"items": [
{"id":841537,"ioc":"wizzy.hopto.org","threat_type":"botnet_cc","threat_type_desc":"Indicator that identifies a botnet command&control server (C&C)","ioc_type":"domain","ioc_type_desc":"Domain that is used for botnet Command&control (C&C)","malware":"win.asyncrat","malware_printable":"AsyncRAT","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.asyncrat","confidence_level":100,"first_seen":"2022-08-05T19:43:08Z","last_seen":"2022-08-05T19:43:08Z","reference":"https://example.com/ref/1","reporter":"test_reporter","tags":["asyncrat"]},
{"id":839586,"ioc":"872ff530d50579ae6bdc7cb4d658324b1d0e7a3e","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.vidar","malware_printable":"Vidar","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.vidar","confidence_level":75,"first_seen":"2022-07-25T22:27:09Z","last_seen":"2022-07-25T22:27:09Z","reference":"","reporter":"test_reporter_2","tags":["Vidar"]},
{"id":839587,"ioc":"a3b5c6d7e8f9g0h1i2j3k4l5m6n7o8p9q0r1s2t3","threat_type":"payload","threat_type_desc":"Indicator that identifies a malware sample (payload)","ioc_type":"sha1_hash","ioc_type_desc":"SHA1 hash of a malware sample (payload)","malware":"win.redline","malware_printable":"RedLine","malware_alias":null,"malware_malpedia":"https://malpedia.caad.fkie.fraunhofer.de/details/win.redline","confidence_level":80,"first_seen":"2025-03-15T10:15:00Z","last_seen":"2025-03-15T10:15:00Z","reference":"","reporter":"test_reporter_3","tags":["RedLine"]}
],
"meta": {
"page_size": 100,
"total_items": 3
}
}
13 changes: 13 additions & 0 deletions packages/ti_abusech/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
# newer versions go on top
- version: "5.0.0"
changes:
- description: Emit a configuration error when the Auth Key is not set for the Malware URLs and Malware payloads datasets.
type: bugfix
link: https://github.com/elastic/integrations/pull/20134
- description: Add Commercial API support for the ThreatFox data stream.
Comment thread
moxarth-rathod marked this conversation as resolved.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: medium path: packages/ti_abusech/changelog.yml:4

MalwareBazaar's api_type/username/password move from data-stream scope to policy-template scope, so 4.2.0 commercial policies lose those values on upgrade - add a breaking-change entry (or var migrate_from) covering it.

Details

Version 4.2.0 (already on main, PR 20199) shipped api_type, username and password as vars of the malwarebazaar data stream. This PR deletes them from packages/ti_abusech/data_stream/malwarebazaar/manifest.yml and re-declares them on the policy-template input in the root manifest. Fleet does not carry values across that scope change on its own, so an existing MalwareBazaar Commercial API policy upgrading 4.2.0 -> 4.3.0 falls back to the new policy-level default api_type=community with no auth_key configured, and collection breaks until the user re-enters the settings. The 4.3.0 changelog contains only a single enhancement line about ThreatFox and gives users no warning. The same PR also flips the malwarebazaar url var from show_user: true to show_user: false, which is another silent UI change.

Recommendation:

Document the migration explicitly, following the precedent set by the 4.0.0 entry in this same file:

- version: "4.3.0"
  changes:
    - description: Add Commercial API support for the ThreatFox data stream.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/20134
    - description: |
        The MalwareBazaar Commercial API settings (API Type, Username, Password) moved from the
        MalwareBazaar data stream settings to the integration-level settings. Users who configured
        the MalwareBazaar Commercial API on 4.2.0 must re-enter API Type, Username and Password
        after upgrading.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/20134

Alternatively, let Fleet carry the values over automatically by bumping format_version to "3.7.0" and declaring the old scope on each relocated var in the root manifest:

          - name: api_type
            type: select
            title: API Type
            migrate_from:
              scope: stream
              stream: malwarebazaar

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moxarth-rathod, we need breaking-change for this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

type: enhancement
link: https://github.com/elastic/integrations/pull/20134
Comment thread
efd6 marked this conversation as resolved.
Comment thread
efd6 marked this conversation as resolved.
- description: |
Move API Type, Username, and Password from the MalwareBazaar data stream to the shared integration input so one Spamhaus credential pair covers all commercial datasets.
Users who configured MalwareBazaar Commercial API on 4.2.0 must re-enter API Type, Username, and Password at the integration level after upgrade. See the Troubleshooting section in the README.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20134
- version: "4.2.0"
changes:
- description: Add Commercial API support for the MalwareBazaar data stream.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ inputs:
- config_version: 2
data_stream:
dataset: ti_abusech.ja3_fingerprints
type: logs
fields:
_conf:
interval: 1h
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ inputs:
- config_version: 2
data_stream:
dataset: ti_abusech.ja3_fingerprints
type: logs
fields:
_conf:
interval: 1h
Expand Down
Loading
Loading