Skip to content

Add annotation control to the profiling feature #1632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jun 6, 2025

Conversation

petewall
Copy link
Collaborator

@petewall petewall commented Jun 3, 2025

Now, the following annotations can be used and are on by default:

  • profiles.grafana.com/cpu.ebpf.enabled: This Pod should have CPU profiles collected using eBPF.
  • profiles.grafana.com/java.enabled: This Pod should have Java profiles collected.

For each enabled type (memory, block, goroutine, mutex, cpu, fgprof, godeltaprof_memory,
godeltaprof_mutex, godeltaprof_block), you can use the following annotations to control profiling:

  • profiles.grafana.com/<type>.scrape: This Pod should have pprof profiles collected for the specified type.
  • profiles.grafana.com/<type>.port: Profiles for the specified type should be collected from this port number.
  • profiles.grafana.com/<type>.port_name: Profiles for the specified type should be collected from this named port.
  • profiles.grafana.com/<type>.path: Profiles for the specified type should be collected from this path.
  • profiles.grafana.com/<type>.scheme: The scheme to use when scraping profiles for the specified type. Defaults to http.

Also create the start of a "Targeting Data Collection" document that shows the various features that can be controlled and configured with annotations.

Finally, some clean up for other readme and tests.

  • Remove white space
  • Remove "helm badges" from feature charts, where they are not necessary.
  • Update the Annotation Autodiscovery feature's list of annotations to include some that weren't there before.
@petewall petewall self-assigned this Jun 3, 2025
@petewall petewall requested review from a team, simonswine and rlankfo as code owners June 3, 2025 22:06
@petewall petewall linked an issue Jun 3, 2025 that may be closed by this pull request
@petewall petewall requested a review from cyrille-leclerc June 3, 2025 22:07
Copy link
Member

@rlankfo rlankfo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Collaborator

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Great work! Just a few nits that I have found in the comments really.

Thanks for your work on this. You were asking for small services to integration test java, I think the adservice from the otel demo might be a good candidate: https://opentelemetry.io/docs/demo/services/ad/

If you have any questions please reach out on slack, I also would like to give this a spin if I find a moment, but that shouldn't block you.

regex = "false"
action = "drop"
}
{{- end }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be super helpful to aid debugging once we have profiles ingested. Futhermore it also helps to make sure that if we double collect, we don't have a series conflict.

If you agree with this let's also add it to _java.tpl and _pprof.tpl

          rule {
            replacement = "alloy/pyroscope.ebpf"
            target_label = "profile_source"
          }
@petewall petewall added the integration-test-skip Do not run integration tests on this PR. label Jun 6, 2025
petewall and others added 10 commits June 6, 2025 12:32
Signed-off-by: Pete Wall <pete.wall@grafana.com>
Signed-off-by: Pete Wall <pete.wall@grafana.com>
Also remove badges from feature charts, as it's not helpful information

Signed-off-by: Pete Wall <pete.wall@grafana.com>
Signed-off-by: Pete Wall <pete.wall@grafana.com>
…to grafana to get it picked up by pyroscope

Signed-off-by: Pete Wall <pete.wall@grafana.com>
Signed-off-by: Pete Wall <pete.wall@grafana.com>
Co-authored-by: Christian Simon <simon@swine.de>
Co-authored-by: Christian Simon <simon@swine.de>
Co-authored-by: Christian Simon <simon@swine.de>
Co-authored-by: Christian Simon <simon@swine.de>
@petewall petewall force-pushed the feat/pyroscope-annotation-targeting branch from 2f0bf14 to af57a96 Compare June 6, 2025 17:33
Signed-off-by: Pete Wall <pete.wall@grafana.com>
@petewall petewall removed the integration-test-skip Do not run integration tests on this PR. label Jun 6, 2025
@petewall petewall merged commit 7dae4f8 into main Jun 6, 2025
25 of 26 checks passed
@petewall petewall deleted the feat/pyroscope-annotation-targeting branch June 6, 2025 17:54
| java.enabled | bool | `true` | Gather profiles by scraping java HTTP endpoints |
| java.annotationSelectors | object | `{}` | Select pods to profile based on pod annotations. Example: `color: "green"` will select pods with the annotation `color="green"`. Example with multiple values: `color: ["blue", "green"]` will select pods with the annotation `color="blue"` or `color="green"`. |
| java.annotations.enable | string | `"enabled"` | The annotation action for enabling or disabling of Java profile collection. |
| java.enabled | bool | `true` | Gather profiles by scraping Java HTTP endpoints |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonswine is it HTTP scrapping for the Java async profiler?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No that's wrong, I created this: #1683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants