capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… by stefans-elastic · Pull Request #44205 · elastic/beats

stefans-elastic · 2025-05-05T13:56:12Z

…umber, memory usage %)

Proposed commit message

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Beats. vSphere metrics to be added #44325

Use cases

Screenshots

Using vcsim simulation of vSphere API not all metrics were available to check (it appears that vcsim doesn't generate disk usage metrics).
So only 2 of the new metrics were generated by vsim (cpu.percent and memory.percent):

However when I tried commenting out the check if a metric is in list of "available" metrics in the code (line 70 in metricbeat/module/vsphere/client/performance.go) it showed more metrics:

So only disk.numberWrite and disk.numberRead I wasn't able to test at all.

Logs

…umber, memory usage %)

mergify · 2025-05-05T13:56:58Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @stefans-elastic? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

mergify · 2025-05-17T11:21:37Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b vsphere-metrics upstream/vsphere-metrics
git merge upstream/main
git push upstream vsphere-metrics

…metrics

ishleenk17 · 2025-05-22T11:59:04Z

@stefans-elastic
Since, we have moved the perf manager code to a common place (good step).
I hope we have tested the metrics of host and datastore that were using the previous implementation.

stefans-elastic · 2025-05-22T12:25:17Z

@stefans-elastic Since, we have moved the perf manager code to a common place (good step). I hope we have tested the metrics of host and datastore that were using the previous implementation.

I've just tried it again and host and datastore streams appear to be working fine. Here are screenshots with some of the properties:

metricbeat/module/vsphere/client/performance.go

metricbeat/module/vsphere/virtualmachine/virtualmachine.go

metricbeat/module/vsphere/virtualmachine/data.go

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml

…to vsphere-metrics

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml

ishleenk17

Thank you for addressing all the comments.
GTG!

#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057)

#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057) # Conflicts: # auditbeat/module/file_integrity/mock_performance.go # auditbeat/module/file_integrity/performance.go # auditbeat/module/file_integrity/performance_test.go

#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057)

…erage, disk rate, disk n… (#44494) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057) * clean up changelog * fix logger in tests * mage update --------- Co-authored-by: stefans-elastic <stefan.stas@elastic.co> Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>

…erage, disk rate, disk n… (#44495) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057) # Conflicts: # auditbeat/module/file_integrity/mock_performance.go # auditbeat/module/file_integrity/performance.go # auditbeat/module/file_integrity/performance_test.go * resolve merge * clean up changelog * fix missing imports * update docs/fields.asciidoc * fix linter --------- Co-authored-by: stefans-elastic <stefan.stas@elastic.co>

…rage, disk rate, disk n… (#44496) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com> (cherry picked from commit 14ac057) * clean up changelog * go mod tidy * update NOTICE.txt --------- Co-authored-by: stefans-elastic <stefan.stas@elastic.co> Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>

elastic#44205) * capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %) * add missing license headers * mage fmt * convert percentage values from fixed point integers * add CHANGELOG.next.asciidoc entry * refactor performance data fetching for datastore * document new fields * remove unused function * remove debug log * add unit tests * mage fmt * fixed precision issue for percentage performance manager values * correct percentage property names * converts kilobytes per sec to bytes per sec * update fields.go * fix linter * fix type assertion pabic * fix unit test failure * address PR comments * changed percentage fields to scaled_float type * make field names same between fields.go and fields.yml --------- Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>

capture extra vsphere metrics (cpu %, disk average, disk rate, disk n…

9534dfa

…umber, memory usage %)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 5, 2025

mergify bot assigned stefans-elastic May 5, 2025

stefans-elastic added the Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team label May 5, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 5, 2025

stefans-elastic added 5 commits May 5, 2025 16:58

Merge branch 'main' into vsphere-metrics

ff0e032

add missing license headers

4249986

mage fmt

247cd5a

Merge branch 'main' into vsphere-metrics

04eb69d

convert percentage values from fixed point integers

fc31132

stefans-elastic added the enhancement label May 6, 2025

stefans-elastic added 9 commits May 6, 2025 12:28

add CHANGELOG.next.asciidoc entry

5cad1e3

refactor performance data fetching for datastore

ea0be4f

document new fields

00135e3

remove unused function

af410f9

remove debug log

0357c15

Merge branch 'main' into vsphere-metrics

35614f0

add unit tests

a728e24

mage fmt

5273df4

Merge branch 'main' into vsphere-metrics

c769e74

Merge branch 'main' of github.com:stefans-elastic/beats into vsphere-…

f6b3bf8

…metrics

stefans-elastic marked this pull request as ready for review May 21, 2025 07:56

stefans-elastic requested a review from a team as a code owner May 21, 2025 07:56

lalit-satapathy requested a review from ishleenk17 May 21, 2025 08:26

ishleenk17 reviewed May 22, 2025

View reviewed changes

metricbeat/module/vsphere/client/performance.go Show resolved Hide resolved

metricbeat/module/vsphere/client/performance.go Outdated Show resolved Hide resolved

fixed precision issue for percentage performance manager values

764c34a

ishleenk17 reviewed May 22, 2025

View reviewed changes

stefans-elastic added 6 commits May 22, 2025 16:19

correct percentage property names

ce7e055

converts kilobytes per sec to bytes per sec

ab81503

update fields.go

26d9f49

fix linter

17a459d

fix type assertion pabic

c366bf9

fix unit test failure

cf2ae15

ishleenk17 reviewed May 23, 2025

View reviewed changes

metricbeat/module/vsphere/virtualmachine/data.go Outdated Show resolved Hide resolved

ishleenk17 and others added 2 commits May 23, 2025 00:47

Merge branch 'main' into vsphere-metrics

fe58854

address PR comments

09ff7ab

ishleenk17 reviewed May 23, 2025

View reviewed changes

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml Show resolved Hide resolved

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml Outdated Show resolved Hide resolved

stefans-elastic added 2 commits May 23, 2025 10:53

Merge branch 'vsphere-metrics' of github.com:stefans-elastic/beats in…

b86c4ab

…to vsphere-metrics

changed percentage fields to scaled_float type

b53555f

ishleenk17 reviewed May 23, 2025

View reviewed changes

metricbeat/module/vsphere/virtualmachine/_meta/fields.yml Outdated Show resolved Hide resolved

make field names same between fields.go and fields.yml

8e3a69b

ishleenk17 approved these changes May 23, 2025

View reviewed changes

stefans-elastic merged commit 14ac057 into elastic:main May 26, 2025
46 checks passed

stefans-elastic added backport-8.18 Automated backport to the 8.18 branch backport-9.0 Automated backport to the 9.0 branch backport-8.19 Automated backport to the 8.19 branch labels May 26, 2025

mergify bot mentioned this pull request May 26, 2025

[8.18](backport #44205) capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… #44494

Merged

6 tasks

mergify bot mentioned this pull request May 26, 2025

[8.19](backport #44205) capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… #44495

Merged

6 tasks

mergify bot mentioned this pull request May 26, 2025

[9.0](backport #44205) capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… #44496

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

capture extra vsphere metrics (cpu %, disk average, disk rate, disk n…#44205

capture extra vsphere metrics (cpu %, disk average, disk rate, disk n…#44205
stefans-elastic merged 28 commits intoelastic:mainfrom
stefans-elastic:vsphere-metrics

stefans-elastic commented May 5, 2025 •

edited

Loading

mergify bot commented May 5, 2025

mergify bot commented May 17, 2025

ishleenk17 commented May 22, 2025

stefans-elastic commented May 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ishleenk17 left a comment

Uh oh!

Labels

2 participants

Conversation

stefans-elastic commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented May 5, 2025

mergify bot commented May 17, 2025

ishleenk17 commented May 22, 2025

stefans-elastic commented May 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

2 participants

stefans-elastic commented May 5, 2025 •

edited

Loading