ti_abusech: switch URL data stream to incremental API - #18945
Conversation
Replace the full-export ZIP download (/downloads/json) with the incremental /v1/urls/recent/ endpoint. The full export ingested all active indicators every interval as a single batch, causing large memory spikes. The incremental API returns only recently added URLs as plain JSON, eliminating ZIP decompression and bulk allocation. IOC expiration is now user-configurable via a new ioc_expiration_duration setting (default 90d), computed from first_seen rather than event.ingested. This matches the pattern already used by the malware, threatfox, and malwarebazaar sibling data streams. Changes: - CEL program: GET to /v1/urls/recent/, decode body.urls, no ZIP - Manifest: new default URL, ioc_expiration_duration var, 10m interval - Pipeline: replace interval-based expiration with duration-based, rename _conf.ioc_expiration_duration to abusech.url field - Fields: replace labels.interval with abusech.url.ioc_expiration_duration - Tests: update fixtures to incremental API response format - Version: 3.6.0 -> 4.0.0 (breaking: labels.interval field removed)
Vale Linting ResultsSummary: 11 suggestions found 💡 Suggestions (11)
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
kcreddy
added
Integration:ti_abusech
🚀 Benchmarks reportTo see the full report comment with |
andrewkroh
added
the
documentation
Add a CEL program guard that detects when the URL is still set to the old full export endpoint (/downloads/json) after an upgrade and returns a clear error message instead of failing with a cryptic JSON unmarshal error. Update the README troubleshooting section with v4.0.0 upgrade steps and revise the breaking-change changelog entry to cover the URL migration, labels.interval removal, and the new IOC Expiration Duration setting. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| required: true | ||
| show_user: false | ||
| default: https://urlhaus.abuse.ch/downloads/json | ||
| default: https://urlhaus-api.abuse.ch/v1/urls/recent/ |
There was a problem hiding this comment.
Similar affect applies here as well.
Existing installations are unaffected. Fleet's package_policy API copies the current input defaults into the policy at creation time, so only package policies created after this change will pick up the new default.
A specific surface level error message is added to CEL program to help with user intervention.
There was a problem hiding this comment.
I think we need to flag the two maxima in play here: three days, and 1000 entries. If the interval is greater than three days (or slightly less than three days for grace), we should raise an error (or at the very least, warn in the UI docs). But we still have the issue that if the entries come in at greater than 1000 entries within the configured interval, the oldest ones will be lost. Do we have an estimate of document addition to the service and the degree of burstiness?
There was a problem hiding this comment.
Updated README, manifest and also CEL with an error 122a7c2.
Do we have an estimate of document addition to the service and the degree of burstiness?
During steady-state, I noticed anywhere between 1-6 URLs being fetched every 10 min. However, the burst could vary significantly as it is community-driven.
| show_user: true | ||
| default: 1h | ||
| description: Duration between requests to the URLhaus API. Supported units for this parameter are h/m/s. Example `24h`. As data dump is generated every 5 minutes, it should be greater than 5 minutes. | ||
| default: 10m |
There was a problem hiding this comment.
This value has significant implications for data loss and network costs. I think that needs to be made clear; too long and we lose documents, too short and re-collect the same documents repeatedly without benefit.
There was a problem hiding this comment.
I've updated the documentation and manifest description with relevant information: 122a7c2
The URLhaus /v1/urls/recent/ endpoint returns at most 1000 entries from the last 3 days. Add a CEL guard that rejects intervals longer than 72h, since the API cannot serve data beyond that window. Update the interval description in the manifest and the README to explain the 1000-entry and 3-day constraints and the trade-off between too-long (data loss) and too-short (redundant collection) intervals. Add script tests for both CEL guards: deprecated URL detection and interval-too-long rejection.
TL;DRThe Remediation
Investigation detailsRoot Cause
But in the same failing commit, policy expected files still contain the older program form and do not include the new guard branch nor
A subsequent PR commit ( Evidence
Verification
Follow-upIf CI still fails after snapshot update, share the full failing section from the Buildkite test phase (before teardown) to confirm whether there is an additional non-policy failure. Note 🔒 Integrity filter blocked 2 itemsThe following items were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | noneWhat is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
💚 Build Succeeded
History
cc @kcreddy |
|
Package ti_abusech - 4.0.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/4.0.0/ |
Replace the full-export ZIP download (/downloads/json) with the incremental /v1/urls/recent/ endpoint. The full export ingested all active indicators every interval as a single batch, causing large memory spikes. The incremental API returns only recently added URLs as plain JSON, eliminating ZIP decompression and bulk allocation. IOC expiration is now user-configurable via a new ioc_expiration_duration setting (default 90d), computed from first_seen rather than event.ingested. This matches the pattern already used by the malware, threatfox, and malwarebazaar sibling data streams. The /v1/urls/recent/ endpoint returns at most 1000 entries from the last 3 days. CEL guards reject the deprecated /downloads/json URL and intervals longer than 72h with clear error messages instead of silent failures. The interval description and README document the 1000-entry and 3-day constraints. Changes: - CEL program: GET to /v1/urls/recent/, decode body.urls, no ZIP - CEL guards: reject deprecated URL, reject interval > 72h - Manifest: new default URL, ioc_expiration_duration var, 10m interval with documented API limits - Pipeline: replace interval-based expiration with duration-based, rename _conf.ioc_expiration_duration to abusech.url field - Fields: replace labels.interval with abusech.url.ioc_expiration_duration - Tests: update fixtures to incremental API response format - Script tests: deprecated URL, interval too long - README: upgrade guidance, API limits, IOC expiration duration - Version: 3.6.0 -> 4.0.0 (breaking: labels.interval field removed)
4 participants
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
System tests (with incremental API) are successful
Updated policy tests pass
Added new script tests