[microsoft_sqlserver] Add transaction log DataStream by muthu-mps · Pull Request #3395 · elastic/integrations

muthu-mps · 2022-05-20T05:49:17Z

What does this PR do?

Collects the transaction log metrics for Microsoft Sql server.

Please refer this PR for performance datastream.

Added a data stream (transaction_log) under microsoft_sqlserver integration package
Added support to collect metrics from Named Instance.
Added data collection logic for the data streams.
Added fields metadata in the appropriate yaml files.
Added dashboards and visualisations.
Updated documentation for the datastreams

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

MSSQL server integration using generic sql, as per the design doc

How to test this PR locally

Install the MSSQL

Go to Integration UI in Kibana
Add the Microsoft SQL Server Integration
Enroll the new elastic-agent
Test named Instance by providing the named instance port in the Microsoft Sqlserver integration config.
In the asset -> Transaction Log dashboard you should be able to see the default visualisation.

Screenshots

elasticmachine · 2022-05-20T06:03:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-06-14T11:50:47.035+0000
Duration: 15 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	12
Skipped	0
Total	12

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-05-20T06:03:19Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 3.432
Classes	100.0% (`1/1`)	💚 3.432
Methods	90.476% (`19/21`)	👍 1.587
Lines	100.0% (`1198/1198`)	💚 10.104
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2022-05-20T09:16:30Z

Pinging @elastic/integrations (Team:Integrations)

ishleenk17

Add ecs.yml file

This reverts commit e8112f8. ReadMe file changes

This reverts commit bde0d4d.

This reverts commit 655a0aa.

This reverts commit 5936aef.

elasticmachine · 2022-06-03T14:30:08Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

lalit-satapathy · 2022-06-08T08:35:34Z

@ManojS-shetty @muthu-mps,
Do we have any changes pending or all the feedbacks are resolved now?

@ruflin @jsoriano @r00tu53r, tar-getting the merge of this PR, need your approval for the same.

muthu-mps · 2022-06-08T08:43:33Z

The changes are done!

jsoriano

It looks good in general, but I am concerned about some changes in versions.

jsoriano · 2022-06-08T10:06:21Z

+    - name: driver
+      type: keyword
+      description: Driver used to execute the query.
+    - name: query


Actually, do we need the query at all in the final document? This is an implementation detail of the integration, I think that it could be removed.

But, if we are storing metrics coming from different queries in different documents we may need some dimension to distinguish between queries. So we may need to index the query to use it as dimension, or find some other field that could cover this.

We can also wait to have some way to join events (elastic/beats#31806 (comment)), then database_id will be probably enough as dimension.

muthu-mps · 2022-06-08T12:17:57Z

/test

jsoriano

Looks good, thanks for addressing all comments.

The main concern I would have now is about mssql.query. It is not clear to me if we are keeping it at the end or not.

Also I think that this data stream would benefit of elastic/beats#31806, to store all the metrics in the same document. And if we end up storing the results of the queries in the same document, what would we store in mssql.query? Would we remove it? This would be a breaking change.

The more I think about storing the query, the more I think that we shouldn't store it. And it is easier in any case to add it later if I am wrong, than to remove it if needed later.

Main concerns addressed, unblocking PR.

jsoriano

LGTM, please remember to add pipeline tests at some moment

…mps/integrations into mssql_transaction_log_data_stream

* new changes for transaction datastream * codeowner and format changes * Naming changes for dashboard, updated img file * Name changes updated * kibana version upgraded to latest * Updated fields and added documentation * Added Readme file * Added Readme file * Added Readme file * Revert "Added Readme file" This reverts commit 13fbb8f6514e7ecfe533a426a3ed90c5da000095. ReadMe file changes * Revert "Updated fields and added documentation" This reverts commit ea08dd6e66f0b9b6139b68c78c6d187107c8c0c2. * Revert "Added Readme file" This reverts commit 53106e7c470a69562b94372c0576d381e97238b2. * Revert "Added Readme file" This reverts commit 67a2c59241ce976ee866adfbf58e7c7431ff0cda. * Updated the fields and dashboard names * document update * transaction datastream testing * Format changes and ECS version change * doc update * Updated the database name * updated manifest and changelog entreies * renamed sql to mssql & updated 60s * events updated * dashboard changes * Dashboards By Value & updated mb to bytes * Dashboards By Value & updated mb to bytes * visualization by value * visualization by value * updated fields * added dimensions and updated ECS reference * added dimensions and updated ECS reference * Added reference to ECS fields & removed query, driver fields * formating events * pipeline update * Added co-owners * renaming to audit_log * removed query field * updated pipeline * Update stream.yml.hbs * Readme file updated

muthu-mps added 4 commits May 19, 2022 22:02

new changes for transaction datastream

6c544cf

codeowner and format changes

1519e1d

Naming changes for dashboard, updated img file

b3fa097

Name changes updated

669063c

muthu-mps added enhancement New feature or request Team:Integrations Label for the Integrations team labels May 20, 2022

muthu-mps requested a review from a team May 20, 2022 05:49

muthu-mps self-assigned this May 20, 2022

kibana version upgraded to latest

9d4439e

rameshelastic added the Team:Service-Integrations Label for the Observability Service Integrations team label May 20, 2022

muthu-mps marked this pull request as ready for review May 20, 2022 09:16

muthu-mps requested a review from a team as a code owner May 20, 2022 09:16

muthu-mps marked this pull request as draft May 20, 2022 09:19

ishleenk17 reviewed May 23, 2022

View reviewed changes

Comment thread packages/microsoft_sqlserver/data_stream/transaction_log/fields/fields.yml Outdated

ishleenk17 reviewed May 23, 2022

View reviewed changes

Comment thread packages/microsoft_sqlserver/data_stream/transaction_log/fields/fields.yml Outdated

ishleenk17 reviewed May 23, 2022

View reviewed changes

muthu-mps added 11 commits May 24, 2022 10:05

Updated fields and added documentation

bde0d4d

Added Readme file

5936aef

Added Readme file

655a0aa

Added Readme file

e8112f8

Revert "Added Readme file"

4367a9c

This reverts commit e8112f8. ReadMe file changes

Revert "Updated fields and added documentation"

5f39acc

This reverts commit bde0d4d.

Revert "Added Readme file"

d643da6

This reverts commit 655a0aa.

Revert "Added Readme file"

8106660

This reverts commit 5936aef.

Updated the fields and dashboard names

a232927

document update

94ed940

transaction datastream testing

58b0be2

andrewkroh added the Team:Security-External Integrations label Jun 3, 2022

muthu-mps added 3 commits June 3, 2022 20:04

updated fields

2f5d7f6

added dimensions and updated ECS reference

271b171

added dimensions and updated ECS reference

f338a30

jsoriano previously requested changes Jun 8, 2022

View reviewed changes

epixa reviewed Jun 8, 2022

View reviewed changes

Comment thread .github/CODEOWNERS Outdated

muthu-mps added 2 commits June 8, 2022 17:17

Added reference to ECS fields & removed query, driver fields

d44010c

formating events

1ca96a3

muthu-mps added 3 commits June 8, 2022 18:53

pipeline update

9e9079b

Added co-owners

2489810

renaming to audit_log

ab33d10

jsoriano reviewed Jun 8, 2022

View reviewed changes

muthu-mps added 2 commits June 9, 2022 11:47

removed query field

3ebbdff

updated pipeline

e9436dd

jsoriano approved these changes Jun 9, 2022

View reviewed changes

Update stream.yml.hbs

357e6b4

muthu-mps requested a review from efd6 June 14, 2022 05:42

efd6 reviewed Jun 14, 2022

View reviewed changes

muthu-mps added 2 commits June 14, 2022 12:05

Readme file updated

3f9a2a4

Merge branch 'mssql_transaction_log_data_stream' of github.com:muthu-…

4717288

…mps/integrations into mssql_transaction_log_data_stream

efd6 approved these changes Jun 14, 2022

View reviewed changes

performance datastream conflicts resolved

7716acc

muthu-mps merged commit 2ddf051 into elastic:main Jun 14, 2022

andrewkroh mentioned this pull request Jun 22, 2022

[production] Promote many packages from snapshot elastic/package-storage#4671

Merged

Navigation Menu

Uh oh!

[microsoft_sqlserver] Add transaction log DataStream - #3395

[microsoft_sqlserver] Add transaction log DataStream#3395
muthu-mps merged 42 commits into
elastic:mainfrom
muthu-mps:mssql_transaction_log_data_stream

muthu-mps commented May 20, 2022 •

edited

Loading

elasticmachine commented May 20, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 20, 2022 •

edited

Loading

elasticmachine commented May 20, 2022

Uh oh!

Uh oh!

ishleenk17 left a comment

elasticmachine commented Jun 3, 2022

lalit-satapathy commented Jun 8, 2022

muthu-mps commented Jun 8, 2022

jsoriano left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano Jun 8, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Jun 8, 2022

jsoriano left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

11 participants

Uh oh!

Conversation

muthu-mps commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

How to test this PR locally

Screenshots

elasticmachine commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

elasticmachine commented May 20, 2022

Uh oh!

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

elasticmachine commented Jun 3, 2022

lalit-satapathy commented Jun 8, 2022

muthu-mps commented Jun 8, 2022

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano Jun 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Jun 8, 2022

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

11 participants

muthu-mps commented May 20, 2022 •

edited

Loading

elasticmachine commented May 20, 2022 •

edited

Loading

elasticmachine commented May 20, 2022 •

edited

Loading