Skip to content

Add null checks and ignore_missing checks to the rename processor - #7845

Merged
ishleenk17 merged 9 commits into
elastic:mainfrom
ishleenk17:event_original_ifcheck
Sep 21, 2023
Merged

Add null checks and ignore_missing checks to the rename processor#7845
ishleenk17 merged 9 commits into
elastic:mainfrom
ishleenk17:event_original_ifcheck

Conversation

@ishleenk17

@ishleenk17 ishleenk17 commented Sep 18, 2023 •
edited
Loading

Copy link
Copy Markdown
Member
  • Bug

What does this PR do?

This PR adds the if check for event.original and the ignore_missing check for the renaming of message to event.original

This PR does changes for below Packages:

  • apache
  • apache_tomcat
  • cassandra
  • ceph
  • citrix_adc
  • golang
  • hadoop
  • haproxy
  • iis
  • kafka
  • microsoft_sqlserver
  • nagios_xi
  • nginx
  • oracle
  • oracle_weblogic
  • php_fpm
  • postgresql
  • rabbitmq
  • redis
  • traefik

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues
@elasticmachine

elasticmachine commented Sep 18, 2023 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-09-21T12:01:57.740+0000

  • Duration: 39 min 57 sec

Test stats 🧪

Test Results
Failed 0
Passed 547
Skipped 0
Total 547

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine

elasticmachine commented Sep 18, 2023 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (53/53) 💚
Files 93.939% (62/66) 👎 -6.061
Classes 93.939% (62/66) 👎 -6.061
Methods 91.304% (609/667) 👎 -3.757
Lines 92.538% (7974/8617) 👍 12.692
Conditionals 100.0% (0/0) 💚
@ishleenk17 ishleenk17 self-assigned this Sep 18, 2023
@ishleenk17
ishleenk17 marked this pull request as ready for review September 18, 2023 09:37
@ishleenk17
ishleenk17 requested a review from a team as a code owner September 18, 2023 09:37
@ishleenk17 ishleenk17 changed the title Initial commit for event.original if check Sep 18, 2023
@muthu-mps

muthu-mps commented Sep 21, 2023

Copy link
Copy Markdown
Contributor

@ishleenk17,
As this PR closes the #7822 , Are we consider addressing all the cloud native and cloud monitoring integration changes as well with this PR ?

Observations

As the event.original is renamed and the grok processor is applied without validating the preserve_original_event flag. Will this be an extra overhead when we remove the event.original field at the end after performing rename and applying grok?

Remove processor Sample

- remove:
      field: event.original
      if: "ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))"
      ignore_failure: true
      ignore_missing: true
@shmsr

shmsr commented Sep 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Can we also note down the names of the packages touched in this PR (in the PR description itself) so that it is trackable?

Also, in the PR description:

Closes #7822

^ This issue will be closed once this PR gets merged and not all packages are taken care of, right? So can we do s/Closes/Relates
@shmsr

shmsr commented Sep 21, 2023

Copy link
Copy Markdown
Member

Looks good to me.
shmsr
shmsr reviewed Sep 21, 2023
View reviewed changes

@shmsr shmsr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment in PR comments.
@ishleenk17

ishleenk17 commented Sep 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member Author

Can we also note down the names of the packages touched in this PR (in the PR description itself) so that it is trackable?

Also, in the PR description:

Closes #7822

^ This issue will be closed once this PR gets merged and not all packages are taken care of, right? So can we do s/Closes/Relates

Makes sense. Updated!
@ishleenk17

ishleenk17 commented Sep 21, 2023

Copy link
Copy Markdown
Member Author

As the event.original is renamed and the grok processor is applied without validating the preserve_original_event flag. Will this be an extra overhead when we remove the event.original field at the end after performing rename and applying grok?

@muthu-mps , the renaming of message to event.original was already happening. We are mainly adding if checks.
Also since we are having check for preserve original in the end after all the groking has been done, so it should be ok.
For tracking preserve original issue I have raised this github ticket
@muthu-mps

muthu-mps commented Sep 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

@ishleenk17 - As part of the new validation rules here we should also remove the message field. Can we consider removing message field as a part of this change ?
@ishleenk17

ishleenk17 commented Sep 21, 2023

Copy link
Copy Markdown
Member Author

@ishleenk17 - As part of the new validation rules here we should also remove the message field. Can we consider removing message field as a part of this change ?

Muthu, removing of fields that have been renamed should be tackled at a generic level, not specific to event.original. Some examples, in haproxy, iis for eg. there exists many renamed fields which are not removed. So, I don't think we should keep this removal as a validation check in. I will comment same in this ticket as well.
@muthu-mps
muthu-mps self-requested a review September 21, 2023 09:27
muthu-mps
muthu-mps approved these changes Sep 21, 2023
View reviewed changes

@muthu-mps muthu-mps left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@ishleenk17
ishleenk17 merged commit 9d22b59 into elastic:main Sep 21, 2023
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package apache - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=apache
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package apache_tomcat - 0.12.3 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package cassandra - 1.9.1 containing this change is available at https://epr.elastic.co/search?package=cassandra
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package ceph - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=ceph
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package citrix_adc - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=citrix_adc
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package golang - 0.4.3 containing this change is available at https://epr.elastic.co/search?package=golang
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package hadoop - 1.1.8 containing this change is available at https://epr.elastic.co/search?package=hadoop
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package haproxy - 1.8.3 containing this change is available at https://epr.elastic.co/search?package=haproxy
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package iis - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=iis
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package kafka - 1.9.2 containing this change is available at https://epr.elastic.co/search?package=kafka
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package microsoft_sqlserver - 2.2.1 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package nagios_xi - 1.1.1 containing this change is available at https://epr.elastic.co/search?package=nagios_xi
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package nginx - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=nginx
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package oracle - 1.20.2 containing this change is available at https://epr.elastic.co/search?package=oracle
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package oracle_weblogic - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=oracle_weblogic
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package php_fpm - 0.6.2 containing this change is available at https://epr.elastic.co/search?package=php_fpm
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package postgresql - 1.16.1 containing this change is available at https://epr.elastic.co/search?package=postgresql
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package rabbitmq - 1.10.1 containing this change is available at https://epr.elastic.co/search?package=rabbitmq
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package redis - 1.11.1 containing this change is available at https://epr.elastic.co/search?package=redis
@elasticmachine

elasticmachine commented Sep 21, 2023

Copy link
Copy Markdown

Package traefik - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=traefik
@ishleenk17 ishleenk17 mentioned this pull request Sep 22, 2023
Closed
35 tasks
@ishleenk17
ishleenk17 deleted the event_original_ifcheck branch February 7, 2025 03:44
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@ishleenk17
Add null checks and ignore_missing checks to the rename processor (el…
9b4453e 
…astic#7845)

* Initial commit for event.original if check

* Bulk checkin for if and ignore_missing checks

* update manifest file for apache

* Update manifest for mssql

* update php_fpm manifest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants

@ishleenk17 @elasticmachine @muthu-mps @shmsr