[microsoft_sqlserver] Add performance DataStream by ManojS-shetty · Pull Request #3391 · elastic/integrations

ManojS-shetty · 2022-05-19T16:07:26Z

What does this PR do?

This PR contributes the performance data stream for Microsoft sqlserver which fetches information from what’s commonly known as Performance Counters in MSSQL. Please refer for transaction_log data stream PR here transaction_log

MSSQL server integration developed using generic sql, as per the design doc

Added a data stream (performance) under microsoft_sqlserver integration package
Added data collection logic for the data streams.
Added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added System Testing.
No Documentation added.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Install the MSSQL
Go to Integration UI in Kibana
Add the Microsoft SQL Server Integration
Enroll the new elastic-agent
In the asset -> Performance dashboard you should be able to see the default visualization.

Screenshots

Prelim for System Test-Microsoft_sql Server

…rformance dashboard

…jS-shetty/integrations into sql-generic-datastream-ver1

…jS-shetty/integrations into mssql-generic-datastream-v1

elasticmachine · 2022-05-19T16:37:49Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-06-14T05:22:46.315+0000
Duration: 14 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

ManojS-shetty · 2022-06-03T13:05:45Z

Lets make sure we always use vis by value: #3448

Done . Visualizations are now using by value.

jsoriano

Added some comments about fields definitions.

elasticmachine · 2022-06-03T14:30:01Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

…jS-shetty/integrations into mssql-generic-datastream-v1

lalit-satapathy · 2022-06-08T08:35:21Z

@ManojS-shetty @muthu-mps,
Do we have any changes pending or all the feedbacks are resolved now?

@ruflin @jsoriano @r00tu53r, tar-getting the merge of this PR, need your approval for the same.

ManojS-shetty · 2022-06-08T09:07:48Z

@ManojS-shetty @muthu-mps, Do we have any changes pending or all the feedbacks are resolved now?

@ruflin @jsoriano @r00tu53r, tar-getting the merge of this PR, need your approval for the same.

The changes are Done.

jsoriano

There are a couple of unaddressed comments.

jsoriano

Same comments as in #3395 (review)

It'd be nice to have some pipeline tests, but I am ok with adding them later.

And I would consider removing the query by now, taking into account that it will be easier to add it later, than to remove it.

ManojS-shetty · 2022-06-09T06:45:59Z

Same comments as in #3395 (review)

It'd be nice to have some pipeline tests, but I am ok with adding them later.

And I would consider removing the query by now, taking into account that it will be easier to add it later, than to remove it.

@jsoriano Removed Query field for now.

ManojS-shetty · 2022-06-09T09:41:25Z

Pinging @elastic/security-external-integrations for approval.

efd6

I'd like to see a pipeline test here.

ManojS-shetty · 2022-06-14T05:43:39Z

I'd like to see a pipeline test here.

Thank you for that suggestion , Yes we will be keeping the pipeline test in enhancement.

* Quick & Dirty MSSQL ImportBeats run and verified to be running fine. * System Test for MS-SQL imported package prelim version * System Test for MS-SQL imported package prelim version1 * working prelims- MSSQL Integration System Test * performance datastream working with 8.3.0-SNAPSHOT elastic agent * added mappings to fields.yml file in performance * MSSQL Performance Dashboards with new Mapping * Added the Dynamic Metrics change to handlebar for performance * Added TRIM functionality in the Performance query and remapped the Performance dashboard * Added the change for naming and description in fields.yml * Deleted transaction_log old dashboard * Changes to perf dashboard * Updated changelog.yml * Updated manifest.yml * added changes to manifest.yml in top level for title and description * Updated changelog.yml * updated Changelog.yml and manifest.yml * Updated a comment for hosts * removed the extra line * added transaction dashboard screenshot image * Reverted codeowner value in manifest * deleted system test chnages * Updated the format in the files * Formatted fields yaml file * Added new sql performance query and changed the field yaml mapping accordingly * deleted system test case for now * Added handling of named instance for perf DataStream * Removed the visualization in kibana directory * Maintained consistency in visualization panels * Updated the consistency in naming conventions for perf * Formatting fields yaml file * Added system Integration test for perf datastream * Formatting docker compose file * Formatting test config yaml file * fixed formatting issue in test yaml file * Updated changelog.yml * Adding README and sample event json files * Fixed formatting in sample event json for perf * Updated the kibana version in manifest file * Chnaged the kibana version for compatibality * Added the dynamic counter name change * Perfomance dashboard checked in with naming consistency * Added documentation change to README * Added manifest changes * Added README file changes * Adding build yaml file * Added 1.0.1 version to manifest and changelog entry * Changed the kibana version for backaward compatibality * Changed back the kibana version * Revert "Deleted transaction_log old dashboard" This reverts commit 9bb54347dae66463496dd4cb50bab08ce1099d2f. * Added ingest pipeline for renaming sql to mssql * Deleted the transaction logs visualisation and dashboard to avoid the merge conflict * Removed SNAPSHOT in version from perf dashboard * changed the visualisation from by reference to by value * Adding new image of performance dashboard * Delete microsoft-sqlserver-performance-dashboard.png * Updated fields and manifest yaml file * Removed Snapshot from the dashboard file * Added few changes like ECS import and driver field removal * Added default yaml file in ingest pipeline * Keeping policy template name to previous value * Removed Query * Adding new sample_event file * Updated the comments

lalit-satapathy and others added 21 commits April 25, 2022 10:23

Quick & Dirty MSSQL ImportBeats run and verified to be running fine.

7e04ba3

System Test for MS-SQL imported package prelim version

a2e08ef

System Test for MS-SQL imported package prelim version1

aa07809

Merge pull request elastic#1 from ManojS-shetty/manoj-mssql-systemtest

87f649d

Prelim for System Test-Microsoft_sql Server

working prelims- MSSQL Integration System Test

26c1bf7

performance datastream working with 8.3.0-SNAPSHOT elastic agent

e60bdc2

added mappings to fields.yml file in performance

8839f50

MSSQL Performance Dashboards with new Mapping

8dcbdfa

Added the Dynamic Metrics change to handlebar for performance

fc4cf2b

Added TRIM functionality in the Performance query and remapped the Pe…

1b17375

…rformance dashboard

Added the change for naming and description in fields.yml

d968eaf

Deleted transaction_log old dashboard

e68e631

Changes to perf dashboard

0c7104c

Updated changelog.yml

b020f7c

Updated manifest.yml

6623b25

added changes to manifest.yml in top level for title and description

b3ba674

Merge branch 'sql-generic-datastream-ver1' of https://github.com/Mano…

969ab2c

…jS-shetty/integrations into sql-generic-datastream-ver1

Updated changelog.yml

4fbc88e

updated Changelog.yml and manifest.yml

03a8bcb

Updated a comment for hosts

1b511ac

removed the extra line

b413b66

ManojS-shetty added enhancement New feature or request Team:Integrations Label for the Integrations team labels May 19, 2022

ManojS-shetty requested a review from a team May 19, 2022 16:07

ManojS-shetty self-assigned this May 19, 2022

ManojS-shetty and others added 3 commits May 19, 2022 21:59

added transaction dashboard screenshot image

8267971

Merge branch 'mssql-generic-datastream-v1' of https://github.com/Mano…

67012d4

…jS-shetty/integrations into mssql-generic-datastream-v1

Merge branch 'main' into mssql-generic-datastream-v1

2468ea7

Reverted codeowner value in manifest

b5bb3a4

Delete microsoft-sqlserver-performance-dashboard.png

17e4d24

andrewkroh added the Integration:microsoft_sqlserver Microsoft SQL Server label Jun 3, 2022

jsoriano reviewed Jun 3, 2022

View reviewed changes

andrewkroh added the Team:Security-External Integrations label Jun 3, 2022

jsoriano reviewed Jun 3, 2022

View reviewed changes

Comment thread packages/microsoft_sqlserver/manifest.yml Outdated

Comment thread packages/microsoft_sqlserver/data_stream/performance/manifest.yml

ManojS-shetty added 3 commits June 6, 2022 13:25

Updated fields and manifest yaml file

1d9647d

Merge branch 'mssql-generic-datastream-v1' of https://github.com/Mano…

b04a4b5

…jS-shetty/integrations into mssql-generic-datastream-v1

Removed Snapshot from the dashboard file

4856809

jsoriano suggested changes Jun 8, 2022

View reviewed changes

ManojS-shetty added 3 commits June 8, 2022 13:41

Added few changes like ECS import and driver field removal

b6d860a

Added default yaml file in ingest pipeline

a42c209

Keeping policy template name to previous value

8e0efd8

jsoriano self-requested a review June 8, 2022 15:55

jsoriano reviewed Jun 8, 2022

View reviewed changes

Removed Query

8dff4c7

Adding new sample_event file

18f2a40

jsoriano approved these changes Jun 9, 2022

View reviewed changes

efd6 reviewed Jun 14, 2022

View reviewed changes

Comment thread packages/microsoft_sqlserver/data_stream/performance/agent/stream/stream.yml.hbs Outdated

Updated the comments

e2770c5

efd6 approved these changes Jun 14, 2022

View reviewed changes

ManojS-shetty merged commit 752ffee into elastic:main Jun 14, 2022

andrewkroh mentioned this pull request Jun 22, 2022

[production] Promote many packages from snapshot elastic/package-storage#4671

Merged

Navigation Menu

Uh oh!

[microsoft_sqlserver] Add performance DataStream - #3391

[microsoft_sqlserver] Add performance DataStream#3391
ManojS-shetty merged 73 commits into
elastic:mainfrom
ManojS-shetty:mssql-generic-datastream-v1

ManojS-shetty commented May 19, 2022 •

edited

Loading

elasticmachine commented May 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

ManojS-shetty commented Jun 3, 2022 •

edited

Loading

jsoriano left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jun 3, 2022

Uh oh!

Uh oh!

lalit-satapathy commented Jun 8, 2022

ManojS-shetty commented Jun 8, 2022

jsoriano left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano left a comment

ManojS-shetty commented Jun 9, 2022

ManojS-shetty commented Jun 9, 2022

efd6 left a comment

Uh oh!

ManojS-shetty commented Jun 14, 2022

Labels

13 participants

Uh oh!

Conversation

ManojS-shetty commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

How to test this PR locally

Screenshots

elasticmachine commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

ManojS-shetty commented Jun 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jun 3, 2022

Uh oh!

Uh oh!

lalit-satapathy commented Jun 8, 2022

ManojS-shetty commented Jun 8, 2022

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

ManojS-shetty commented Jun 9, 2022

ManojS-shetty commented Jun 9, 2022

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

ManojS-shetty commented Jun 14, 2022

Labels

13 participants

ManojS-shetty commented May 19, 2022 •

edited

Loading

elasticmachine commented May 19, 2022 •

edited

Loading

ManojS-shetty commented Jun 3, 2022 •

edited

Loading