Skip to content

[MongoDB] Added remove processors in ingest pipeline for log datastream - #4837

Merged
ManojS-shetty merged 3 commits into
elastic:mainfrom
ManojS-shetty:mongo_preserve_original_event_sdh
Dec 24, 2022
Merged

[MongoDB] Added remove processors in ingest pipeline for log datastream#4837
ManojS-shetty merged 3 commits into
elastic:mainfrom
ManojS-shetty:mongo_preserve_original_event_sdh

Conversation

@ManojS-shetty

@ManojS-shetty ManojS-shetty commented Dec 15, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR address the SDH on mongodb log data stream

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone the PR.
  • Run the elastic-package stack up command.
  • Add the mongoDb integration.
  • In Integration UI provide the mongod.log path and enable the preserve original event.
  • Enrol the elastic agent.

Related issues
@ManojS-shetty
Added remove processors in ingest pipeline
377e424 
	modified:   packages/mongodb/changelog.yml
	modified:   packages/mongodb/data_stream/log/elasticsearch/ingest_pipeline/default.yml
	modified:   packages/mongodb/manifest.yml
@ManojS-shetty ManojS-shetty self-assigned this Dec 15, 2022
@ManojS-shetty ManojS-shetty added Team:Service-Integrations Label for the Observability Service Integrations team 8.7 candidate labels Dec 15, 2022
@ManojS-shetty
Added changelog entry
df0cdf2 
	modified:   packages/mongodb/changelog.yml
@ManojS-shetty

ManojS-shetty commented Dec 15, 2022

Copy link
Copy Markdown
Contributor Author

/test
@elasticmachine

elasticmachine commented Dec 15, 2022 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-12-16T08:18:58.149+0000

  • Duration: 19 min 36 sec

Test stats 🧪

Test Results
Failed 0
Passed 34
Skipped 0
Total 34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine

elasticmachine commented Dec 15, 2022 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (3/3) 💚
Classes 100.0% (3/3) 💚
Methods 96.97% (32/33) 👎 -3.03
Lines 67.46% (85/126) 👎 -25.14
Conditionals 100.0% (0/0) 💚
@ManojS-shetty
ManojS-shetty marked this pull request as ready for review December 15, 2022 11:19
@ManojS-shetty
ManojS-shetty requested a review from a team as a code owner December 15, 2022 11:19
ishleenk17
ishleenk17 reviewed Dec 16, 2022
View reviewed changes
Comment thread packages/mongodb/data_stream/log/elasticsearch/ingest_pipeline/default.yml
field:
- first_char
- remove:
field: event.original

@ishleenk17 ishleenk17 Dec 16, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manoj, have we verified in the system tests that the event.original field doesn't appear anymore in the document generated?

@ManojS-shetty ManojS-shetty Dec 16, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes ishleen, verified in document generated in the system test and event.original field is not appearing anymore.

@ishleenk17 ishleenk17 Dec 16, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have the test generated files uploaded in this as well ?

@ManojS-shetty ManojS-shetty Dec 16, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@ManojS-shetty
Updated with sample event json file
729ba19 
	modified:   packages/mongodb/data_stream/log/sample_event.json
	modified:   packages/mongodb/docs/README.md
ishleenk17
ishleenk17 reviewed Dec 19, 2022
View reviewed changes
Comment thread packages/mongodb/data_stream/log/sample_event.json
"@timestamp": "2022-10-20T11:23:04.602Z",
"@timestamp": "2022-12-16T07:30:07.376Z",
"agent": {
"ephemeral_id": "3ec02007-5c86-4a8d-920a-6b1dafdf6f08",

@ishleenk17 ishleenk17 Dec 19, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the event.original removed in the diff.
Was it not present before ?

@ManojS-shetty ManojS-shetty Dec 19, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No It was not present before.

@ishleenk17 ishleenk17 Dec 20, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the original issue was event.original being present in the events. Hence, I was expecting it to be present previously when preserve event original was set to false
@ManojS-shetty
ManojS-shetty merged commit 8abb451 into elastic:main Dec 24, 2022
@elasticmachine

elasticmachine commented Dec 24, 2022

Copy link
Copy Markdown

Package mongodb - 1.5.1 containing this change is available at https://epr.elastic.co/search?package=mongodb
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@ManojS-shetty
[MongoDB] Added remove processors in ingest pipeline for log datastre…
33ea842 
…am (elastic#4837)

* Added remove processors in ingest pipeline
	modified:   packages/mongodb/changelog.yml
	modified:   packages/mongodb/data_stream/log/elasticsearch/ingest_pipeline/default.yml
	modified:   packages/mongodb/manifest.yml

* Added changelog entry
	modified:   packages/mongodb/changelog.yml

* Updated with sample event json file
	modified:   packages/mongodb/data_stream/log/sample_event.json
	modified:   packages/mongodb/docs/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

8.7 candidate Team:Service-Integrations Label for the Observability Service Integrations team

3 participants

@ManojS-shetty @elasticmachine @ishleenk17