Skip to content

[aws] Ingest managed insights from Security Hub - #14354

Merged
chrisberkhout merged 3 commits into
elastic:mainfrom
chrisberkhout:aws-securityhub_insights-managed-insights
Jul 1, 2025
Merged

[aws] Ingest managed insights from Security Hub#14354
chrisberkhout merged 3 commits into
elastic:mainfrom
chrisberkhout:aws-securityhub_insights-managed-insights

Conversation

@chrisberkhout

@chrisberkhout chrisberkhout commented Jun 27, 2025

Copy link
Copy Markdown
Contributor

Proposed commit message

[aws] Ingest managed insights from Security Hub

The AWS Security Hub API's GetInsights endpoint[1] returns only custom
insights. Managed insights[2] are built-in, and can only be retrieved
by ARN.

Fetch a hard-coded list of managed insights first, then fetch pages of
custom insights as before.

This has been tested against the live API.

[1]: https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetInsights.html
[2]: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-managed-insights.html

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

@chrisberkhout chrisberkhout self-assigned this Jun 27, 2025
@chrisberkhout
chrisberkhout requested a review from a team as a code owner June 27, 2025 16:50
@chrisberkhout chrisberkhout added enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jun 27, 2025
@elasticmachine

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@chrisberkhout
chrisberkhout requested review from a team as code owners June 27, 2025 16:52
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 27, 2025

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package aws 👍(13) 💚(7) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
guardduty 1061.57 887.31 -174.26 (-16.42%) 💔
inspector 2012.07 1492.54 -519.53 (-25.82%) 💔

To see the full report comment with /test benchmark fullreport

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit message, s/retieved/retrieved/.

Comment on lines +81 to +84
# This transform's job is to fail and end pagination when finished.
# It will set a value if the last request was the first in the sequence
# (which collects the managed insights), or if there is another page of
# custom insights to fetch.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's enough HTTPJSON magic here that I think the comment needs to explain more about how it is actually working.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a sentence. Here's the full new comment:

      # This transform's job is to fail and end pagination when finished.
      # It will set a value if the last request was the first in the sequence
      # (which collects the managed insights), or if there is another page of
      # custom insights to fetch. Otherwise, there will be a value template
      # error (.last_response.body.NextToken doesn't exist) and the pagination
      # sequence will end (because fail_on_template_error is set).
@chrisberkhout
chrisberkhout force-pushed the aws-securityhub_insights-managed-insights branch from ea0efa3 to 0444f3e Compare June 30, 2025 08:43
@chrisberkhout
chrisberkhout requested a review from efd6 June 30, 2025 08:44
@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @chrisberkhout

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@chrisberkhout
chrisberkhout merged commit 08dd8a2 into elastic:main Jul 1, 2025
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package aws - 3.9.0 containing this change is available at https://epr.elastic.co/package/aws/3.9.0/

robester0403 pushed a commit to robester0403/integrations that referenced this pull request Jul 8, 2025
The AWS Security Hub API's GetInsights endpoint[1] returns only custom
insights. Managed insights[2] are built-in, and can only be retrieved
by ARN.

Fetch a hard-coded list of managed insights first, then fetch pages of
custom insights as before.

This has been tested against the live API.

[1]: https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetInsights.html
[2]: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-managed-insights.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

3 participants