Skip to content

[vSphere][host] Add support for new metrics in Host datastream - #10894

Merged
ishleenk17 merged 19 commits into
elastic:mainfrom
kush-elastic:28-implement-the-change-to-hostsystem-datastream-in-integrations
Sep 19, 2024
Merged

[vSphere][host] Add support for new metrics in Host datastream#10894
ishleenk17 merged 19 commits into
elastic:mainfrom
kush-elastic:28-implement-the-change-to-hostsystem-datastream-in-integrations

Conversation

@kush-elastic

@kush-elastic kush-elastic commented Aug 27, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Here are the following metrics to be added for the host data stream in the vSphere Integration. Here we added a new performance API to get more detailed information from the vSphere.

Metrics Type Metrics API Field Mappings
Host Disk Performance      
  disk.deviceLatency.average (ms) Performance disk.devicelatency.average.ms
  disk.maxTotalLatency.latest (ms) Performance disk.latency.total.ms
  disk.usage.average (KBps) Performance disk.total.bytes
  disk.read.average (KBps) Performance disk.read.bytes
  disk.write.average (KBps) Performance disk.write.bytes
  disk.capacity.usage.average (KB) Performance disk.capacity.usage.bytes
       
Host uptime/status      
  host.Summary.OverallStatus Summary status
  host.Summary.QuickStats.Uptime Summary uptime
  host.Datastore Extra datastore.names
      datastore.count
  host.Vm Extra vm.names
      vm.count
       
Host Network Stats host.Netowork Extra network.names
      network.count
  net.transmitted.average (KBps) Performance network.bandwidth.transmitted.bytes
  net.received.average (KBps) Performance network.bandwidth.received.bytes
  net.usage.average (KBps) Performance network.bandwidth.total.bytes
  net.packetsTx.summation (num) Performance network.packets.transmitted.count
  net.packetsRx.summation (num) Performance network.packets.received.count
  net.errorsRx.summation (num) Performance network.packets.errors.received.count
  net.errorsTx.summation (num) Performance network.packets.errors.transmitted.count
      network.packets.errors.total.count
  net.multicastTx.summation (num) Performance network.packets.multicast.transmitted.count
  net.multicastRx.summation (num) Performance network.packets.multicast.received.count
      network.packets.multicast.total.count
  net.droppedTx.summation (num) Performance network.packets.dropped.transmitted.count
  net.droppedRx.summation (num) Performance network.packets.dropped.received.count
      network.packets.dropped.total.count
Alerts  
triggerd_alarms.name Name of the alarm.
triggerd_alarms.id Unique key that identifies the alarm.
triggerd_alarms.status Overall status of the alarm object.
triggerd_alarms.triggered_time Time the alarm triggered.
triggerd_alarms.description Description of the alarm.
triggerd_alarms.entity_name alarm happened in respective entity

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

TSDB

Values being used:
        elasticsearch_host = https://localhost:9200
        elasticsearch_ca_path = /root/.elastic-package/profiles/default/certs/elasticsearch/ca-cert.pem
        elasticsearch_user = elastic
        elasticsearch_pwd = changeme
        data_stream = metrics-vsphere.host-default
        docs_index = First index of the data stream
        settings_mappings_index = Last index of the data stream
        max_docs = All documents
        get_overlapping_files = True
        directory_overlapping_files = overwritten-docs-metrics-vsphere.host-default
        display_docs = 10
        copy_docs_per_dimension = 2

You're testing with version 8.15.2-SNAPSHOT.

Testing data stream metrics-vsphere.host-default.
Index being used for the documents is .ds-metrics-vsphere.host-default-2024.09.19-000001.
Index being used for the settings and mappings is .ds-metrics-vsphere.host-default-2024.09.19-000001.

The time series fields for the TSDB index are: 
        - dimension (10 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name
                - service.address
                - vsphere.host.name
        - counter (1 fields):
                - vsphere.host.cpu.total.mhz
        - gauge (31 fields):
                - vsphere.host.cpu.free.mhz
                - vsphere.host.cpu.used.mhz
                - vsphere.host.cpu.used.pct
                - vsphere.host.datastore.count
                - vsphere.host.disk.capacity.usage.bytes
                - vsphere.host.disk.devicelatency.average.ms
                - vsphere.host.disk.latency.total.ms
                - vsphere.host.disk.read.bytes
                - vsphere.host.disk.total.bytes
                - vsphere.host.disk.write.bytes
                - vsphere.host.memory.free.bytes
                - vsphere.host.memory.total.bytes
                - vsphere.host.memory.used.bytes
                - vsphere.host.memory.used.pct
                - vsphere.host.network.bandwidth.received.bytes
                - vsphere.host.network.bandwidth.total.bytes
                - vsphere.host.network.bandwidth.transmitted.bytes
                - vsphere.host.network.count
                - vsphere.host.network.packets.dropped.received.count
                - vsphere.host.network.packets.dropped.total.count
                - vsphere.host.network.packets.dropped.transmitted.count
                - vsphere.host.network.packets.errors.received.count
                - vsphere.host.network.packets.errors.total.count
                - vsphere.host.network.packets.errors.transmitted.count
                - vsphere.host.network.packets.multicast.received.count
                - vsphere.host.network.packets.multicast.total.count
                - vsphere.host.network.packets.multicast.transmitted.count
                - vsphere.host.network.packets.received.count
                - vsphere.host.network.packets.transmitted.count
                - vsphere.host.uptime
                - vsphere.host.vm.count
        - routing_path (10 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name
                - service.address
                - vsphere.host.name

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-vsphere.host-default-2024.09.19-000001 to tsdb-index-enabled...
All 8 documents taken from index .ds-metrics-vsphere.host-default-2024.09.19-000001 were successfully placed to index tsdb-index-enabled.

Related issues
@kush-elastic
kush-elastic requested a review from a team as a code owner August 27, 2024 10:52
@kush-elastic kush-elastic self-assigned this Aug 27, 2024
@kush-elastic kush-elastic added enhancement New feature or request Integration:vsphere VMware vSphere Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Aug 27, 2024
ishleenk17
ishleenk17 reviewed Aug 30, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/host/fields/agent.yml Outdated
lucian-ioan
lucian-ioan reviewed Sep 11, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/datastore/manifest.yml
ishleenk17
ishleenk17 reviewed Sep 17, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/datastore/manifest.yml Outdated
ishleenk17
ishleenk17 reviewed Sep 17, 2024
View reviewed changes
Comment thread packages/vsphere/_dev/build/docs/README.md Outdated
Comment thread packages/vsphere/_dev/build/docs/README.md Outdated
Comment thread packages/vsphere/_dev/build/docs/README.md Outdated
Comment thread packages/vsphere/_dev/build/docs/README.md Outdated
Comment thread packages/vsphere/data_stream/host/fields/fields.yml
Comment thread packages/vsphere/data_stream/host/fields/fields.yml
Comment thread packages/vsphere/data_stream/host/fields/fields.yml
Comment thread packages/vsphere/data_stream/host/manifest.yml Outdated
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 17, 2024

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
ishleenk17
ishleenk17 reviewed Sep 18, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/host/fields/agent.yml
ishleenk17
ishleenk17 reviewed Sep 18, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/host/fields/fields.yml
Comment thread packages/vsphere/data_stream/host/fields/fields.yml
ishleenk17
ishleenk17 reviewed Sep 19, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/host/elasticsearch/ingest_pipeline/default.yml Outdated
@kush-elastic

kush-elastic commented Sep 19, 2024

Copy link
Copy Markdown
Contributor Author

/test
ishleenk17
ishleenk17 reviewed Sep 19, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/virtualmachine/fields/fields.yml
niraj-elastic
niraj-elastic suggested changes Sep 19, 2024
View reviewed changes
Comment thread packages/vsphere/data_stream/datastore/fields/fields.yml
niraj-elastic
niraj-elastic approved these changes Sep 19, 2024
View reviewed changes

@niraj-elastic niraj-elastic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
ishleenk17
ishleenk17 approved these changes Sep 19, 2024
View reviewed changes

@ishleenk17 ishleenk17 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, once the CI passes!
lucian-ioan
lucian-ioan approved these changes Sep 19, 2024
View reviewed changes

@lucian-ioan lucian-ioan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@elastic-sonarqube

elastic-sonarqube Bot commented Sep 19, 2024

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
@elasticmachine

elasticmachine commented Sep 19, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @kush-elastic
@ishleenk17
ishleenk17 merged commit 925b0b9 into elastic:main Sep 19, 2024
niraj-elastic
niraj-elastic reviewed Sep 19, 2024
View reviewed changes

@niraj-elastic niraj-elastic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package vsphere - 1.15.0 containing this change is available at https://epr.elastic.co/search?package=vsphere
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@kush-elastic
[vSphere][host] Add support for new metrics in Host datastream (elast…
6a7afaa 
…ic#10894)

* initial commit for vSphere Host datastream

* fix unnecessary change

* update changelog entry and manifest to raise incremental PRs

* elastic-package check

* update host data-stream with new intervalID support

* resolve review comments

* revert unnecessary changes

* resolve review comments

* triggerd_alarms -> triggered_alarms

* fix mappings

* kibana version to 8.15.2

* update README.md

* fix pipeline

* remove rename processor

* use group time.

* update metric_type

* update changelog entry

* add tags
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@kush-elastic
[vSphere][host] Add support for new metrics in Host datastream (elast…
549c7b9 
…ic#10894)

* initial commit for vSphere Host datastream

* fix unnecessary change

* update changelog entry and manifest to raise incremental PRs

* elastic-package check

* update host data-stream with new intervalID support

* resolve review comments

* revert unnecessary changes

* resolve review comments

* triggerd_alarms -> triggered_alarms

* fix mappings

* kibana version to 8.15.2

* update README.md

* fix pipeline

* remove rename processor

* use group time.

* update metric_type

* update changelog entry

* add tags
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:vsphere VMware vSphere Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

6 participants

@kush-elastic @elasticmachine @lucian-ioan @ishleenk17 @harnish-crest-data @niraj-elastic