Skip to content

[tenable_sc] Add drop to pipeline if event is empty - #5380

Merged
P1llus merged 10 commits into
elastic:mainfrom
LaZyDK:patch-2
Feb 27, 2023
Merged

[tenable_sc] Add drop to pipeline if event is empty#5380
P1llus merged 10 commits into
elastic:mainfrom
LaZyDK:patch-2

Conversation

@LaZyDK

@LaZyDK LaZyDK commented Feb 24, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Drop event in Vulnerability pipeline if there is no returned results.

What does this PR do?

Drops empty events.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
@LaZyDK
Add ignore_missing
e669c33 
Add ignore_missing to json.seeAlso field
@LaZyDK
LaZyDK requested a review from a team as a code owner February 24, 2023 10:07
@LaZyDK LaZyDK changed the title Add ignore_missing Feb 24, 2023
@LaZyDK

LaZyDK commented Feb 24, 2023

Copy link
Copy Markdown
Contributor Author

@P1llus Test :)
@elasticmachine

elasticmachine commented Feb 24, 2023 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-02-27T09:36:53.064+0000

  • Duration: 17 min 32 sec

Test stats 🧪

Test Results
Failed 0
Passed 19
Skipped 0
Total 19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine

elasticmachine commented Feb 24, 2023

Copy link
Copy Markdown

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
@jamiehynds jamiehynds added the Integration:tenable_sc Tenable Security Center label Feb 24, 2023
@P1llus

P1llus commented Feb 24, 2023

Copy link
Copy Markdown
Member

/test
@P1llus

P1llus commented Feb 24, 2023

Copy link
Copy Markdown
Member

@LaZyDK any idea why we would want to ignore this? If its missing then there is no data.
@LaZyDK

LaZyDK commented Feb 24, 2023

Copy link
Copy Markdown
Contributor Author

@P1llus I see that you are right! So we should actually drop the events instead.
@vinit-chauhan

vinit-chauhan commented Feb 25, 2023

Copy link
Copy Markdown
Contributor

Hey @LaZyDK @P1llus - These are the extra external references for the Vulnerability. I don't see a point in dropping the entire event as the field is expected to be empty for many vulnerabilities. With that being said, this is a new behavior for the API though previously the response would have an empty string in cases where there were no external references for the vulnerability.

By the way, thanks for the PR @LaZyDK, Really appreciate it. :)
@LaZyDK

LaZyDK commented Feb 25, 2023

Copy link
Copy Markdown
Contributor Author

After testing with the ignore_missing set we get a new error, which should also be fixed if not everything should be dropped.
I will look into this on Monday.
@LaZyDK

LaZyDK commented Feb 27, 2023

Copy link
Copy Markdown
Contributor Author

I don't find any data in the events that are not having the seeAlso field.
Here is event.original of one:

{"error_code":0,"error_msg":"","response":{"endOffset":"118000","matchingDataElementCount":"-1","results":[],"returnedRecords":0,"startOffset":"117000","totalRecords":"116095"},"timestamp":1677232486,"type":"regular","warnings":[]}

Data is all neat after implementing a Drop action as a failure event handler for that particular split.
@LaZyDK LaZyDK changed the title [tenable_sc] Add ignore_missing to pipeline Feb 27, 2023
@LaZyDK

LaZyDK commented Feb 27, 2023

Copy link
Copy Markdown
Contributor Author

I removed the failure processor and included a conditional drop processor instead, to be inline with other integrations like the one for MISP.

Also included the empty original event in the test document.
@efd6

efd6 commented Feb 27, 2023

Copy link
Copy Markdown
Contributor

/test
@P1llus

P1llus commented Feb 27, 2023

Copy link
Copy Markdown
Member

It needs to generate pipeline tests, adding it now
@P1llus

P1llus commented Feb 27, 2023

Copy link
Copy Markdown
Member

/test
@elasticmachine

elasticmachine commented Feb 27, 2023 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (3/3) 💚
Classes 100.0% (3/3) 💚
Methods 100.0% (45/45) 💚
Lines 96.964% (1118/1153) 👍 8.186
Conditionals 100.0% (0/0) 💚
@P1llus
P1llus merged commit ec5cbb7 into elastic:main Feb 27, 2023
@LaZyDK
LaZyDK deleted the patch-2 branch February 27, 2023 10:19
@elasticmachine

elasticmachine commented Feb 27, 2023

Copy link
Copy Markdown

Package tenable_sc - 1.7.1 containing this change is available at https://epr.elastic.co/search?package=tenable_sc
agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 20, 2023
@LaZyDK @P1llus @agithomas
[tenable_sc] Add drop to pipeline if event is empty (elastic#5380)
fd8429b 
* Add ignore_missing

Add ignore_missing to json.seeAlso field

* Update changelog.yml

* Update manifest.yml

* Drop event if no results

* Update changelog.yml

* Add Drop processor with conditions

And remove Drop Failure Processor

* Update changelog.yml

* Add test for empty event

* adding pipeline test generation

---------

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 21, 2023
@LaZyDK @P1llus @agithomas
[tenable_sc] Add drop to pipeline if event is empty (elastic#5380)
80888de 
* Add ignore_missing

Add ignore_missing to json.seeAlso field

* Update changelog.yml

* Update manifest.yml

* Drop event if no results

* Update changelog.yml

* Add Drop processor with conditions

And remove Drop Failure Processor

* Update changelog.yml

* Add test for empty event

* adding pipeline test generation

---------

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@LaZyDK @P1llus
[tenable_sc] Add drop to pipeline if event is empty (elastic#5380)
4e161bf 
* Add ignore_missing

Add ignore_missing to json.seeAlso field

* Update changelog.yml

* Update manifest.yml

* Drop event if no results

* Update changelog.yml

* Add Drop processor with conditions

And remove Drop Failure Processor

* Update changelog.yml

* Add test for empty event

* adding pipeline test generation

---------

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:tenable_sc Tenable Security Center

6 participants

@LaZyDK @elasticmachine @P1llus @vinit-chauhan @efd6 @jamiehynds