ThreatQ Integration | Fixes Issue Pulling Sources & Attributes

[zach-threatq]

Bug Fix

Proposed commit message

fix: issue returning sources & attributes from ThreatQ

ThreatQ recently updated the default API response from the Threat Library to exclude attributes and sources by default. The original API response can be maintained by adding the fields URL parameter. In addition to the default fields (*), we can ask for the attributes and sources fields, which will restore full functionality of the integration & mappings. This commit also specifically asks for attribute names and values to de-clutter the API response.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

N/A

How to test this PR locally

Using a ThreatQ instance on the latest version, ensure that the list of sources and attributes properly get parsed into an Elastic index record with the default integration configuration.

Related issues

N/A

Screenshots

N/A

[cla-checker-service]

💚 CLA has been signed

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

Is this change backward compatible with earlier versions of ThreatQ? Let's say a user is connected to an earlier version, then will the new query param on the GET request work the same?

The original API response can be maintained by adding the fields URL parameter.

Can you point me at the documentation for the API that this integration is using? I was looking at https://helpcenter.threatq.com/assets/HTML/api/ThreatQ_REST_API_Reference_v6.16.0.html and didn't see the API endpoint.

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[zach-threatq]

Is this change backward compatible with earlier versions of ThreatQ? Let's say a user is connected to an earlier version, then will the new query param on the GET request work the same?

The original API response can be maintained by adding the fields URL parameter.

Can you point me at the documentation for the API that this integration is using? I was looking at https://helpcenter.threatq.com/assets/HTML/api/ThreatQ_REST_API_Reference_v6.16.0.html and didn't see the API endpoint.

The fields parameter has existed in the Threat Library API for quite some time, dating back to 2023 (ThreatQ v5.23.0). So this should be backwards compatible with most current versions of ThreatQ. I made the fields list configurable just in case there needs to be some tweaks to it for older versions or newer versions, but from what I understand, it should be compatible with any ThreatQ version >= v5.23.0.

As far as documentation goes, I don't believe there are public docs outlining the endpoints in full detail, just high level.

[andrewkroh]

As far as documentation goes, I don't believe there are public docs outlining the endpoints in full detail, just high level.

Even if it's high level info, can you point me toward where I would find a doc for /api/indicators/query/hash/{{data_collection_id}} if any exists. Thanks.

[jamiehynds]

@zach-threatq are you ok to proceed with the suggestions made by @andrewkroh and also provide the documentation he requested here. We have a few customers awaiting the integration update to support your latest API, so keen to get this PR merged. Thanks!

[zach-threatq]

@zach-threatq are you ok to proceed with the suggestions made by @andrewkroh and also provide the documentation he requested here. We have a few customers awaiting the integration update to support your latest API, so keen to get this PR merged. Thanks!

Yes, sorry, just been busy on my end and haven't had a chance to look back at this. I'll check it out and make the changes!

[vera-review-bot]

No issues across the latest commits 1c67a22, d0eb5c8.

Review summary

Issues found across earlier commits aa6759b — 1 high

• 🟠 The new url.params.fields value is rendered unquoted (link) (Resolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[andrewkroh]

/test

[andrewkroh]

Tests are failing with:

error: could not add data stream config to policy: could not create package policy (req {"name":"ti_threatq-threat-73707","description":"","namespace":"73707","policy_id":"424faf96-d64a-4e67-a61e-7fd26f37f9c9","package":{"name":"ti_threatq","version":"1.39.2"},"inputs":{"ti_threatq-httpjson":{"enabled":true,"streams":{"ti_threatq.threat":{"enabled":true,"vars":{"client_id":"test","client_secret":"test","data_collection_id":"someid","enable_request_tracer":true,"host":"http://svc-threatq:8080","interval":"10m","ioc_expiration_duration":"5d","preserve_original_event":true,"token_url":"http://svc-threatq:8080/api/token"}}}}},"force":false}); API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.httpjson.streams.ti_threatq.threat.vars.threat_library_fields: Strings starting with special YAML characters like * or & need to be enclosed in double quotes."}

[andrewkroh]

This is a known Fleet issue: elastic/kibana#184972

Fleet rejects text vars whose value starts with * or & at package-policy create time. Quoting the value in the UI does not work, because the quotes become part of the stored string.

Workaround: reorder the default so it does not start with *:

default: "sources,attributes,attributes.name,attributes.value,*"

instead of:

default: "*,sources,attributes,attributes.name,attributes.value"

[andrewkroh]

/test

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 247d3b8

History

• 💔 Build #46604 failed d0eb5c8

[elastic-vault-github-plugin-prod]

Package ti_threatq - 1.39.2 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.39.2/