[vSphere][network] Create network datastream by lucian-ioan · Pull Request #10993 · elastic/integrations

lucian-ioan · 2024-09-03T20:08:24Z

Proposed commit message

Following beats code changes in elastic/beats#40559, the following metrics are added to the network datastream:

Metrics Type	Metrics	API	Field Mappings
Network/DistributedVirtualPort	Network.OverallStatus	Summary	status
	Network.Summary.Accessible	Summary	accessible
	Network.ConfigStatus	Summary	config.status
	Network.type	Summary	type
	Network.Vm	ExtraApi	vm.names
		ExtraApi	vm.count
	Network.Host	ExtraApi	host.names
		ExtraApi	host.count

Alerts
triggerd_alarms.name	Name of the alarm.
triggerd_alarms.id	Unique key that identifies the alarm.
triggerd_alarms.status	Overall status of the alarm object.
triggerd_alarms.triggered_time	Time the alarm triggered.
triggerd_alarms.description	Description of the alarm.
triggerd_alarms.entity_name	alarm happened in respective entity

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Create Snapshot of Metricbeat changes.
Use Snapshot to make the changes in the vSphere Integration Host datastream.
Update the system and pipeline changes.
Add versioning to changelog.
Test the Integration for metric collection.
Test Integration from TSDB Perspective.
Code review.

How to test this PR locally

Related issues

Closes https://github.com/elastic/obs-integration-team/issues/40

Screenshots

TSDB testing

Values being used:
        elasticsearch_host = https://localhost:9200
        elasticsearch_ca_path = /home/lucian/.elastic-package/profiles/default/certs/elasticsearch/ca-cert.pem
        elasticsearch_user = elastic
        elasticsearch_pwd = changeme
        data_stream = metrics-vsphere.network-default
        docs_index = First index of the data stream
        settings_mappings_index = Last index of the data stream
        max_docs = All documents
        get_overlapping_files = True
        directory_overlapping_files = overwritten-docs-metrics-vsphere.network-default
        display_docs = 10
        copy_docs_per_dimension = 2

You're testing with version 8.15.2-SNAPSHOT.

Testing data stream metrics-vsphere.network-default.
Index being used for the documents is .ds-metrics-vsphere.network-default-2024.09.23-000001.
Index being used for the settings and mappings is .ds-metrics-vsphere.network-default-2024.09.23-000001.

The time series fields for the TSDB index are: 
        - dimension (9 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name
                - service.address
        - gauge (2 fields):
                - vsphere.network.host.count
                - vsphere.network.vm.count
        - routing_path (9 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name
                - service.address

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-vsphere.network-default-2024.09.23-000001 to tsdb-index-enabled...
All 12 documents taken from index .ds-metrics-vsphere.network-default-2024.09.23-000001 were successfully placed to index tsdb-index-enabled.

ishleenk17 · 2024-09-19T07:38:15Z

README is missing.
update kibana version
Perios has not been taken care in this

ishleenk17 · 2024-09-19T07:36:54Z

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.14.0-next"


change version to 1.15.0-next

ishleenk17 · 2024-09-19T07:37:23Z

+          type: long
+          metric_type: gauge
+          description: >
+            Number of virtual machines connected to this network.


Missing the latest fields

niraj-elastic · 2024-09-21T07:00:31Z

+streams:
+  - input: vsphere/metrics
+    title: vSphere network metrics
+    description: vSphere network metrics


please add period here, along with other configuration parameters.

niraj-elastic · 2024-09-21T07:01:16Z

+username: {{username}}
+password: {{password}}
+# If insecure is true, don't verify the server's certificate chain
+insecure: {{insecure}}


please add tags and processors.

…egration

ishleenk17 · 2024-09-23T04:46:14Z

  changes:
+    - description: Add new network datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11205


Incorrect PR number

ishleenk17 · 2024-09-23T04:46:52Z

+      description: >
+        If the host is a container.
+
+    - name: os.build


These are ecs fields w/o dimensions. Not required here.

ishleenk17 · 2024-09-23T04:49:52Z

@lucian-ioan : Pls share details of TSDB testing

kush-elastic · 2024-09-23T05:29:11Z

@@ -1,10 +1,13 @@
 # newer versions go on top
 - version: "1.15.0-next"


Suggested change

- version: "1.15.0-next"

- version: "1.15.0"

As this PR is the last one we need to remove next tag and update version in manifest to 1.15.0.

I guess so, should we do it with this PR @ishleenk17?

Lets keep the next label as of now.
We will raise a PR when we are ready to release it.

kush-elastic · 2024-09-23T05:29:53Z

+                  "type": "Network"
+                }
+              }
+


Suggested change

Empty Line

kush-elastic · 2024-09-23T05:38:44Z

+    },
+      "metricset": {
+        "name": "network",
+        "period": 20000


This formatting seems wrong, can you check once?

Suggested change

},

"metricset": {

"name": "network",

"period": 20000

},

"metricset": {

"name": "network",

"period": 20000

Should work fine but re-formatted it just in case.

kush-elastic

Added few comments

niraj-elastic · 2024-09-23T05:44:39Z

+        - name: names
+          type: keyword
+          description: >
+            Names of the virtual machines connected to this network.


Suggested change

Names of the virtual machines connected to this network.

List of all the virtual machines connected to this network.

minor nit. Please do it for other similar fields as well.

kush-elastic

LGTM!

niraj-elastic · 2024-09-23T06:06:10Z

+| vsphere.network.vm.count | Number of virtual machines connected to this network. | long | gauge |
+| vsphere.network.vm.names | List of all the virtual machines connected to this network. | keyword |  |
+
+


Suggested change

I don't think this is necessary, it's the default format for all data streams in docs.

elasticmachine · 2024-09-23T06:23:33Z

💚 Build Succeeded

Buildkite Build
Commit: ed60052

History

💚 Build #16273 succeeded 655cf0d
💔 Build #16272 failed ffbe173
💚 Build #16269 succeeded ef389ad
💔 Build #15751 failed c424ec3
💔 Build #15572 failed 4e5d27a
💔 Build #15415 failed 4a84d4b

cc @lucian-ioan

elastic-sonarqube · 2024-09-23T06:23:36Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

niraj-elastic · 2024-09-23T06:38:19Z

LGTM

elastic-vault-github-plugin-prod · 2024-09-26T14:30:32Z

Package vsphere - 1.15.0 containing this change is available at https://epr.elastic.co/search?package=vsphere

initial network integration

4a84d4b

lucian-ioan added enhancement New feature or request Integration:vsphere VMware vSphere Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Sep 3, 2024

lucian-ioan self-assigned this Sep 3, 2024

add sample event

ec88c7c

lucian-ioan marked this pull request as ready for review September 6, 2024 10:08

lucian-ioan requested a review from a team as a code owner September 6, 2024 10:08

Lucian and others added 3 commits September 6, 2024 13:11

add changelog entry

4e5d27a

update sample event

e4f75a4

update fields

c424ec3

ishleenk17 reviewed Sep 19, 2024

View reviewed changes

niraj-elastic suggested changes Sep 21, 2024

View reviewed changes

lucian-ioan closed this Sep 22, 2024

lucian-ioan force-pushed the update_vsphere_network_integration branch from c424ec3 to 8966518 Compare September 22, 2024 19:42

lucian-ioan added 3 commits September 22, 2024 22:48

Merge remote-tracking branch 'origin' into update_vsphere_network_int…

0bd18b3

…egration

add changelog entry

495afab

add period and tags

215473a

lucian-ioan reopened this Sep 22, 2024

update manifest file

743320f

elastic deleted a comment from mergify Bot Sep 22, 2024

lucian-ioan and others added 5 commits September 22, 2024 23:13

add triggered alarms

c94ddaa

update docs

ef389ad

lowercase manifest

873e437

update sample event

e0f7aef

add test case

ffbe173

lucian-ioan requested review from ishleenk17, kush-elastic and niraj-elastic September 22, 2024 21:11

update sample event

655cf0d

ishleenk17 reviewed Sep 23, 2024

View reviewed changes

kush-elastic reviewed Sep 23, 2024

View reviewed changes

Lucian added 2 commits September 23, 2024 08:35

remove fields without dimensions

4465ab1

correct PR number

07a1874

kush-elastic reviewed Sep 23, 2024

View reviewed changes

kush-elastic suggested changes Sep 23, 2024

View reviewed changes

niraj-elastic suggested changes Sep 23, 2024

View reviewed changes

Lucian added 2 commits September 23, 2024 08:51

resolve nits

fe61089

remove empty line

ed60052

lucian-ioan requested review from ishleenk17, kush-elastic and niraj-elastic September 23, 2024 05:54

kush-elastic approved these changes Sep 23, 2024

View reviewed changes

niraj-elastic suggested changes Sep 23, 2024

View reviewed changes

lucian-ioan requested a review from niraj-elastic September 23, 2024 06:15

niraj-elastic approved these changes Sep 23, 2024

View reviewed changes

ishleenk17 approved these changes Sep 23, 2024

View reviewed changes

lucian-ioan merged commit 00ed51b into elastic:main Sep 23, 2024

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025

[vSphere][network] Create network datastream (elastic#10993)

178160b

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025

[vSphere][network] Create network datastream (elastic#10993)

1cb9ce1

lucian-ioan deleted the update_vsphere_network_integration branch February 6, 2025 19:12

Navigation Menu

Uh oh!

[vSphere][network] Create network datastream - #10993

[vSphere][network] Create network datastream#10993
lucian-ioan merged 19 commits into
elastic:mainfrom
lucian-ioan:update_vsphere_network_integration

lucian-ioan commented Sep 3, 2024 •

edited

Loading

ishleenk17 commented Sep 19, 2024

ishleenk17 Sep 19, 2024

ishleenk17 Sep 19, 2024

niraj-elastic Sep 21, 2024

niraj-elastic Sep 21, 2024

ishleenk17 Sep 23, 2024

ishleenk17 Sep 23, 2024

ishleenk17 commented Sep 23, 2024

kush-elastic Sep 23, 2024

lucian-ioan Sep 23, 2024

ishleenk17 Sep 23, 2024

kush-elastic Sep 23, 2024

kush-elastic Sep 23, 2024

lucian-ioan Sep 23, 2024

kush-elastic left a comment

niraj-elastic Sep 23, 2024

kush-elastic left a comment

niraj-elastic Sep 23, 2024

lucian-ioan Sep 23, 2024 •

edited

Loading

elasticmachine commented Sep 23, 2024

elastic-sonarqube Bot commented Sep 23, 2024

niraj-elastic commented Sep 23, 2024

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Labels

5 participants

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.14.0-next"

		@@ -1,10 +1,13 @@
		# newer versions go on top
		- version: "1.15.0-next"

	Names of the virtual machines connected to this network.
	List of all the virtual machines connected to this network.

		\| vsphere.network.vm.count \| Number of virtual machines connected to this network. \| long \| gauge \|
		\| vsphere.network.vm.names \| List of all the virtual machines connected to this network. \| keyword \| \|

Uh oh!

Conversation

lucian-ioan commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

TSDB testing

ishleenk17 commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ishleenk17 commented Sep 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucian-ioan Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

elasticmachine commented Sep 23, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Sep 23, 2024

Quality Gate passed

niraj-elastic commented Sep 23, 2024

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Labels

5 participants

lucian-ioan commented Sep 3, 2024 •

edited

Loading

lucian-ioan Sep 23, 2024 •

edited

Loading