[sysdig] Add support for vulnerability datastream

[brijesh-elastic]

Proposed commit message

sysdig: add support for vulnerability datastream.

This vulnerability logs provides an overview of scan results from your
runtime, pipeline, and container registries to provide highly accurate views
of vulnerability risk, access to public exploits, and risk management.

Sanitized test case inputs were obtained from live Sysdig Secure instance
using the Sysdig Next Gen API.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/sysdig directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes [Sysdig Secure] New data stream: Vulnerabilities #12269

Screenshots

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

Lots of good work. I think that some significant simplifications can be made before we merge it.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chrisberkhout]

Here's a summary (see other comments for details)...

Some outstanding questions:

• Paginate/query for continuous operation and no repeats?
• One stage only?

Some things to do:

• User documentation regarding endpoint responses (scan result) vs ingested events (scan results per package-vulnerability pair).
• Code comment re relationships between entities and intended split/resolve logic.

Something to maybe do:

• Possible CEL simplification

[chrisberkhout]

Regarding pagination, is it not possible to request scan results that are after scan results we have seen so far?

I wonder if the next.page cursor is long-lived, or if there are other filtering parameters. It would be good to avoid fetching everything each day, and also to avoid waiting a day to fetch new things.

I saw in the issue discussion:

Workloads that are no longer running will be removed from the logs, as mentioned in the runtime documentation.

But in the documentation it sounds like the old runtime workloads will be removed from a UI view, not that earlier scan results from such workloads will immediately become unavailable from the UI.

• Runtime workloads will offer a live, auto-refreshing state. This means:
  • Workloads that are no longer running will be removed from the runtime view

[brijesh-elastic]

I agree.
We don't have a filter to collect only the latest scan results, so we have no choice but to gather all the results at every interval.

We can offer a filter option to users where they can apply filtering on supported fields (for example, kubernetes.cluster.name="production" and kubernetes.pod.container.name="docker.internal.sysdig.tools"). This will reduce the amount of data fetched and can be useful for users who need data based on specific criteria.

[chrisberkhout]

Okay.

Here's what I meant this part:

I wonder if the next.page cursor is long-lived, or if there are other filtering parameters. It would be good to avoid fetching everything each day, and also to avoid waiting a day to fetch new things.

In the system test I see cursor values like abcdefgabcdefg01234567899876543210==, which looks like a fake version of a base64-encoded value.

Often APIs have cursors like dGltZXN0YW1wPjE3NDQ3NzI0NjExOTUxNDk4MDA=, which can be base64-decoded to timestamp>1744772461195149800. A cursor like that is something we can reuse many times to keep checking for data after that point. We could even construct a cursor value like that ourselves using the timestamp of the last data we saw.

I haven't seen a real cursor value from the live API, so I'm not sure if this is one of the cases where it has a clearly identifiable expression or ID that could be used in that way.

[brijesh-elastic]

Often APIs have cursors like dGltZXN0YW1wPjE3NDQ3NzI0NjExOTUxNDk4MDA=, which can be base64-decoded to timestamp>1744772461195149800. A cursor like that is something we can reuse many times to keep checking for data after that point. We could even construct a cursor value like that ourselves using the timestamp of the last data we saw.

Okay, I got it. The real cursor from the live API doesn't include the case you mentioned. I'm only receiving strings such as NDAw, MzEw, MTQw, etc.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 424ac6a

History

• 💚 Build #29888 succeeded 370baf8
• 💚 Build #29291 succeeded c9064a5
• 💚 Build #27907 succeeded fd2f491
• 💔 Build #27860 failed 05309a0
• 💔 Build #27076 failed 48b0930
• 💔 Build #26715 failed e4960fb

cc @brijesh-elastic

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[chrisberkhout]

Thanks for addressing all the points I raised.

[elastic-vault-github-plugin-prod]

Package sysdig - 2.1.0 containing this change is available at https://epr.elastic.co/package/sysdig/2.1.0/