abuse.ch integration for Elastic

Version 5.0.0 (View all)
Managed integration release status
Subscription level
What's this?
Basic
Developed by
What's this?
Elastic
Ingestion method(s) API
Minimum Kibana version(s) 9.1.0
8.19.0
Important - Managed integrations on Elastic Cloud Hosted

You can use this integration as an Elastic Managed integration on Elastic Cloud Hosted deployments running this version or later.

The abuse.ch integration for Elastic allows you to collect logs from abuse.ch, which provides actionable, community-driven threat intelligence data and helps identify, track, and mitigate malware and botnet-related cyber threats. With this integration, threat intelligence indicators can be ingested into Elastic for enhanced threat detection and event enrichment.

The abuse.ch integration is compatible with v1 version of abuse.ch URLhaus, MalwareBazaar, ThreatFox, and SSLBL APIs.

This integration periodically queries the abuse.ch APIs to retrieve threat intelligence indicators.

This integration collects threat intelligence indicators into the following datasets:

  • ja3_fingerprints: Collects JA3 fingerprint based threat indicators identified by SSLBL via SSLBL API endpoint.
  • malware: Collects malware payloads from URLs tracked by URLhaus via URLhaus Bulk API.
  • malwarebazaar: Collects malware payloads from MalwareBazaar via MalwareBazaar API.
  • sslblacklist: Collects SSL certificate based threat indicators blacklisted on SSLBL via SSLBL API endpoint.
  • threatfox: Collects threat indicators from ThreatFox via ThreatFox API.
  • url: Collects recently added malware URL based threat indicators from URLhaus via URLhaus API. The API returns at most 1000 entries from the last 3 days. The Interval setting must be short enough to avoid exceeding the 1000-entry limit between polls; otherwise the oldest URLs added in that window will be lost.

The abuse.ch integration brings threat intel into Elastic Security, enabling detection alerts when Indicators of Compromise (IoCs) like malicious IPs, domains, or hashes match your event or alert data. This data can also support threat hunting, enrich alerts with threat context, and power dashboards to track known threats in your environment.

This integration installs Elastic latest transforms. For more details, check the Transform setup and requirements.

Which credentials you need depends on which datasets you enable, not on the integration as a whole:

  • ThreatFox threat indicators (threatfox) and MalwareBazaar payloads (malwarebazaar) can use either API, selected with the shared API Type setting. The Community API requires an Auth Key. The Commercial API requires Spamhaus username and password credentials, which the integration exchanges for a short-lived JWT.
  • Malware URLs (url) and Malware payloads (malware) always query the URLhaus Community API and require an Auth Key regardless of API Type. Requests without the key are rejected.
  • SSL Blacklisted Certificates (sslblacklist) and JA3 Fingerprints (ja3_fingerprints) read the SSLBL feeds and use neither credential.

Because API Type applies to the whole integration, selecting Commercial API does not remove the need for an Auth Key unless you also disable Malware URLs and Malware payloads.

  1. Sign up for a new account, or login into the abuse.ch authentication portal.
  2. Connect with at least one authentication provider: Google, Github, X, or LinkedIn.
  3. Select Save profile.
  4. In the Optional section, click the Generate Key button to generate Auth Key.
  5. Copy the generated Auth Key.

For more details, check the abuse.ch Community First - New Authentication blog.

Commercial API access uses JWT authentication. Create credentials in the Spamhaus Customer Portal, then configure the username and password in the integration. The integration authenticates to /v1/login and refreshes the JWT as needed.

  1. Log in to the Spamhaus Customer Portal.
  2. Navigate to Product > abuse.ch API.
  3. Under Generate new credentials for JWT authentication, fill out the required information and follow the on-screen instructions.
  4. Copy the generated username and password.

For more details, check the abuse.ch commercial API documentation on JWT authentication for endpoints available to query.

This integration supports both Elastic Agentless-based and Agent-based installations.

Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Agentless integrations and the Agentless integrations FAQ.

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

  1. In the top search bar in Kibana, search for Integrations.

  2. In the search bar, type abuse.ch.

  3. Select the abuse.ch integration from the search results.

  4. Select Add abuse.ch to add the integration.

  5. Enable and configure only the collection methods which you will use.

    • To Collect abuse.ch logs via API, you'll need to:

      • Configure Auth Key (Community). The Malware URLs and Malware payloads datasets always need it, and ThreatFox threat indicators and MalwareBazaar payloads need it when API Type is Community API.
      • To use the Commercial API for ThreatFox threat indicators and MalwareBazaar payloads, set API Type to Commercial API and configure Username (Commercial) and Password (Commercial).
      • Enable/Disable the required datasets.
      • For each dataset, adjust the integration configuration parameters if required, including the URL, Interval, etc. to enable data collection.
  6. Select Save and continue to save the integration.

  1. In Kibana, navigate to Dashboards.
  2. In the search bar, type abuse.ch.
  3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated.
  1. In Kibana, navigate to Management > Stack Management.
  2. Under Data, select Transforms.
  3. In the search bar, type abuse.ch.
  4. All transforms from the search results should indicate Healthy under the Health column.
  • Upgrading to v5.0.0: Version 5.0.0 moves API Type, Username (Commercial), and Password (Commercial) from the MalwareBazaar data stream to the shared integration input. Fleet does not carry these values across that scope change. If you configured MalwareBazaar Commercial API on 4.2.0, re-enter the credentials after upgrade:

    1. In Kibana, navigate to Fleet > Agent policies.
    2. Select the policy containing the abuse.ch integration.
    3. Edit the abuse.ch integration.
    4. Set API Type to Commercial API, then enter Username (Commercial) and Password (Commercial) at the integration level (not under the MalwareBazaar data stream).
    5. Under the MalwareBazaar data stream, expand Advanced options and reset URL to https://mb-api.abuse.ch/api/v1/ if you had changed it to https://api.spamhaus.com for the 4.2.0 Commercial API. The Commercial API base URL is now configured once at the integration level via Commercial API URL.
    6. Select Save integration.

    Community API users who only use an Auth Key are unaffected. After this change, the same Spamhaus credentials apply to all commercial datasets (MalwareBazaar, ThreatFox, and future commercial data streams).

  • Upgrading to v4.0.0: Version 4.0.0 switches the URL data stream from the full export ZIP endpoint (/downloads/json) to the incremental JSON API (/v1/urls/recent/). When upgrading from a previous version, the URL setting in your integration policy retains the old value and must be updated manually:

    1. In Kibana, navigate to Fleet > Agent policies.
    2. Select the policy containing the abuse.ch integration.
    3. Edit the abuse.ch integration.
    4. Under the Malware URLs data stream, change the URL setting from https://urlhaus.abuse.ch/downloads/json to https://urlhaus-api.abuse.ch/v1/urls/recent/.
    5. Select Save integration.

    If the URL is not updated, the integration will log an error: the URL is set to the deprecated full export endpoint (/downloads/json) which is no longer supported. Additionally, the labels.interval field has been removed from URL data stream documents. If you have saved queries, detection rules, or dashboards that reference this field, remove those references. A new IOC Expiration Duration setting (default 90d) now controls how long indicators remain active. Review this value and adjust it to match your organization's retention requirements.

  • When creating the Auth Key inside the abuse.ch authentication portal, make sure you connect at least one additional authentication provider to ensure seemless access to the abuse.ch platform.

  • Check for captured ingestion errors inside Kibana. Ingestion errors, including API errors, are captured into error.message field.

    1. Navigate to Analytics > Discover.
    2. In Search field names, search and add fields error.message and data_stream.dataset into the Discover view. For more details on adding fields inside Discover, check Discover getting started.
    3. Search for the dataset(s) that are enabled by this integration. For example, in the KQL query bar, use the KQL query data_stream.dataset: ti_abusech.url to search on specific dataset or KQL query data_stream.dataset: ti_abusech.* to search on all datasets.
    4. Search for errors that are captured into error.message field using KQL query error.message: *. You can combine queries using KQL boolean expressions, such as AND. For example, to search for errors inside url dataset, you can use KQL query: data_stream.dataset: ti_abusech.url AND error.message: *.
  • Common API errors: All the abusec.ch API errors are captured inside the error fields.

    1. abuse.ch APIs return HTTP status 403 Forbidden when the Auth Key is invalid. In such case, the error.message field is populated with message query_status: unknown_auth_key and error.id with 403 Forbidden. To fix this, you need to regenerate the Auth Key in the abuse.ch authentication portal and update the integration policy with newly generated Auth Key.
    2. abuse.ch APIs return HTTP status 500 Internal Server Error when experiencing problem on the abuse.ch service. In such case, error.message field is populated with message POST:500 Internal Server Error (500) and error.id with 500 Internal Server Error. This is likely a one-off scenario and the ingestion should resume normally in the subsequent request.
    3. When API Type is Community API and Auth Key (Community) is not configured, MalwareBazaar and ThreatFox log api_type is 'community' but auth_key is not configured in error.message and configuration_error in error.id. Add Auth Key (Community) at the integration level.
    4. When API Type is Commercial API and Username (Commercial) or Password (Commercial) is not configured, MalwareBazaar and ThreatFox log api_type is 'commercial' but username/password are not configured in error.message and configuration_error in error.id. Configure both fields at the integration level.
  • Since this integration supports the expiration of Indicators of Compromise (IoCs) using Elastic latest transform, the threat indicators are present in both source and destination indices. While this may appear to be duplicate ingestion, it is an implementation detail necessary for properly expiring threat indicators.

  • Because the latest copy of threat indicators is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly. The ILM policies on source indices can be tuned to manage their data retention period.

  • For help with Elastic ingest tools, check Common problems.

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

These inputs can be used in this integration:

This integration datasets use the following APIs:

All abuse.ch datasets now support indicator expiration. The URL, Malware, MalwareBazaar, and ThreatFox datasets expire threat indicators after the duration configured in the IOC Expiration Duration setting (default 90d). An Elastic Transform is created for every source index to make sure only active threat indicators are available to the end users. Each transform creates a destination index named logs-ti_abusech_latest.dest_* which only contains active and unexpired threat indicators. The indicator match rules and dashboards are updated to list only active threat indicators. Destinations indices are aliased to logs-ti_abusech_latest.<data_stream_name>.

Source Data stream Destination Index Pattern Destination Alias
logs-ti_abusech.url-* logs-ti_abusech_latest.dest_url-* logs-ti_abusech_latest.url
logs-ti_abusech.malware-* logs-ti_abusech_latest.dest_malware-* logs-ti_abusech_latest.malware
logs-ti_abusech.malwarebazaar-* logs-ti_abusech_latest.dest_malwarebazaar-* logs-ti_abusech_latest.malwarebazaar
logs-ti_abusech.threatfox-* logs-ti_abusech_latest.dest_threatfox-* logs-ti_abusech_latest.threatfox

To facilitate IoC expiration, source data stream-backed indices .ds-logs-ti_abusech.<data_stream_name>-* are allowed to contain duplicates from each polling interval. ILM policy logs-ti_abusech.<data_stream_name>-default_policy is added to these source indices, so it doesn't lead to unbounded growth. This means that in these source indices data will be deleted after 5 days from ingested date.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.