Skip to content

Refactor benchmark CI - #7873

Merged
XSAM merged 15 commits into
open-telemetry:mainfrom
XSAM:refactor-benchmark-ci
Mar 3, 2026
Merged

Refactor benchmark CI#7873
XSAM merged 15 commits into
open-telemetry:mainfrom
XSAM:refactor-benchmark-ci

Conversation

@XSAM

@XSAM XSAM commented Feb 3, 2026

Copy link
Copy Markdown
Member

Resolves #4537

This PR replaces the github-action-benchmark approach with CodSpeed.

CodSpeed addresses:

An example of benchmark result: https://codspeed.io/XSAM/opentelemetry-go/runs/697aa351f5662999a88a8bf3

Benchmark page for opentelemetry-go: https://codspeed.io/open-telemetry/opentelemetry-go

@XSAM XSAM added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Feb 3, 2026
@codecov

codecov Bot commented Feb 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.7%. Comparing base (4ba8200) to head (a1c2448).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@          Coverage Diff          @@
##            main   #7873   +/-   ##
=====================================
  Coverage   81.7%   81.7%           
=====================================
  Files        304     304           
  Lines      23342   23342           
=====================================
+ Hits       19086   19090    +4     
+ Misses      3867    3864    -3     
+ Partials     389     388    -1     

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@dmathieu

dmathieu commented Feb 4, 2026

Copy link
Copy Markdown
Member

While I have no objection for this (it fixes the conflict issue which is a big improvement), I want to note that it also adds a dependency on an external private company (we've seen cases where external things like this dropped our access overnight).
Also, on the collector, CodSpeed ends up not being very relevant (see open-telemetry/opentelemetry-collector#14521 for example).

@XSAM

XSAM commented Feb 4, 2026

Copy link
Copy Markdown
Member Author

@dmathieu

Also, on the collector, CodSpeed ends up not being very relevant (see open-telemetry/opentelemetry-collector#14521 for example).

My assumption is because the collector repo uses ubuntu-latest, which is VM where it is a noisy environment, to run benchmarks.

https://github.com/open-telemetry/opentelemetry-collector/blob/5e3bf40de1335e61ab1f9550ccf1d9bcf8a8b58f/.github/workflows/go-benchmarks.yml#L13

We use bare-metal to run benchmarks instead. This would give us a stable result.

Comment thread Makefile Outdated
Comment thread .github/workflows/benchmark.yml
@pellared

Copy link
Copy Markdown
Member

@XSAM, bump

@XSAM
XSAM requested a review from pellared March 3, 2026 12:53

@pellared pellared left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add https://codspeed.io/open-telemetry/opentelemetry-go somewhere in README.md?

@XSAM

XSAM commented Mar 3, 2026

Copy link
Copy Markdown
Member Author

Should we add https://codspeed.io/open-telemetry/opentelemetry-go somewhere in README.md?

Yeah, I am planning to do that once we can confirm the benchmark result is stable enough to be useful.

@XSAM
XSAM merged commit e413dfc into open-telemetry:main Mar 3, 2026
29 checks passed
@XSAM
XSAM deleted the refactor-benchmark-ci branch March 3, 2026 18:42
@pellared pellared added this to the v1.42.0 milestone Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skip Changelog PRs that do not require a CHANGELOG.md entry

3 participants