Skip to content

zscaler_zia: Remove department field and fix parsing errors in web logs - #10874

Merged
kcreddy merged 7 commits into
elastic:mainfrom
kcreddy:zscaler_hexfields
Aug 28, 2024
Merged

zscaler_zia: Remove department field and fix parsing errors in web logs#10874
kcreddy merged 7 commits into
elastic:mainfrom
kcreddy:zscaler_hexfields

Conversation

@kcreddy

@kcreddy kcreddy commented Aug 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Remove department field and fix parsing errors in web logs.

The department value is already being ingested from dept field. This PR removes the
hex-encoded department field in favour of dept due to noticed improper hex-encoding in
fields prefixed with e here.

Remove urldecode processor on parsing urls causing parsing errors. The b64url
field is already being decoded by script processor.

Also adds on_failure clause on urldecode processors to continue pipeline execution
to prevent parsing errors on hex-encoded values to stop running rest of the pipeline.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Ran pipeline tests removing department field in web logs. Since there is already a dept field, the tests are updated with values from dept.
Added new test events with failing b64url values and verifying the new change doesn't lead to parsing error, and populates url ECS field.

Run pipeline tests:
eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=web

--- Test results for package: zscaler_zia - START ---
╭─────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME                                             │ RESULT │ TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────┼────────┼──────────────┤
│ zscaler_zia │ web         │ pipeline  │ (ingest pipeline warnings test-web-http-endpoint.log) │ PASS   │ 513.032208ms │
│ zscaler_zia │ web         │ pipeline  │ (ingest pipeline warnings test-web.log)               │ PASS   │ 478.037458ms │
│ zscaler_zia │ web         │ pipeline  │ test-web-http-endpoint.log                            │ PASS   │ 219.403208ms │
│ zscaler_zia │ web         │ pipeline  │ test-web.log                                          │ PASS   │ 247.017667ms │
╰─────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zscaler_zia - END   ---
Done
@kcreddy
kcreddy marked this pull request as ready for review August 26, 2024 11:20
@kcreddy
kcreddy requested a review from a team as a code owner August 26, 2024 11:20
@kcreddy kcreddy self-assigned this Aug 26, 2024
@kcreddy kcreddy added bugfix Pull request that fixes a bug issue Integration:zscaler_zia Zscaler Internet Access Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 26, 2024
@elasticmachine

elasticmachine commented Aug 26, 2024

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@kcreddy kcreddy changed the title zscaler_zia: Remove department field and add on_failure clauses in web Aug 26, 2024
@chrisberkhout
chrisberkhout self-requested a review August 26, 2024 14:08
chrisberkhout
chrisberkhout requested changes Aug 26, 2024
View reviewed changes
Comment thread packages/zscaler_zia/_dev/build/docs/README.md Outdated
Comment thread packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml
@elasticmachine

elasticmachine commented Aug 27, 2024 •
edited
Loading

Copy link
Copy Markdown

🚀 Benchmarks report

Package zscaler_zia 👍(3) 💚(2) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alerts 4255.32 3571.43 -683.89 (-16.07%) 💔
endpoint_dlp 2475.25 2008.03 -467.22 (-18.88%) 💔
tunnel 4566.21 3773.58 -792.63 (-17.36%) 💔

To see the full report comment with /test benchmark fullreport
@kcreddy
kcreddy requested a review from chrisberkhout August 27, 2024 07:15
@kcreddy kcreddy changed the title zscaler_zia: Remove department field and add on_failure clause in web logs Aug 27, 2024
@kcreddy kcreddy changed the title zscaler_zia: Remove department field and parsing errors in web logs Aug 27, 2024
@elastic-sonarqube

elastic-sonarqube Bot commented Aug 27, 2024

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
@elasticmachine

elasticmachine commented Aug 27, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @kcreddy
@kcreddy
kcreddy merged commit 1fb5746 into elastic:main Aug 28, 2024
@elasticmachine

elasticmachine commented Aug 28, 2024

Copy link
Copy Markdown

Package zscaler_zia - 3.2.3 containing this change is available at https://epr.elastic.co/search?package=zscaler_zia
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@kcreddy
zscaler_zia: Remove department field and fix parsing errors in web lo…
67cd27e 
…gs (elastic#10874)

Remove `department` field and fix parsing errors in web logs.

The `department` value is already being ingested from `dept` field. This PR removes the 
hex-encoded `department` field in favour of `dept` due to noticed improper hex-encoding in 
fields prefixed with `e` [here](elastic#10855 (comment)). 

Remove `urldecode` processor on parsing urls causing parsing errors. The `b64url` 
field is already being decoded by `script` processor. 

Also adds `on_failure` clause on `urldecode` processors to continue pipeline execution 
to prevent parsing errors on hex-encoded values to stop running rest of the pipeline.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@kcreddy
zscaler_zia: Remove department field and fix parsing errors in web lo…
ac2932b 
…gs (elastic#10874)

Remove `department` field and fix parsing errors in web logs.

The `department` value is already being ingested from `dept` field. This PR removes the 
hex-encoded `department` field in favour of `dept` due to noticed improper hex-encoding in 
fields prefixed with `e` [here](elastic#10855 (comment)). 

Remove `urldecode` processor on parsing urls causing parsing errors. The `b64url` 
field is already being decoded by `script` processor. 

Also adds `on_failure` clause on `urldecode` processors to continue pipeline execution 
to prevent parsing errors on hex-encoded values to stop running rest of the pipeline.
@kcreddy
kcreddy deleted the zscaler_hexfields branch February 7, 2025 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:zscaler_zia Zscaler Internet Access Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

3 participants

@kcreddy @elasticmachine @chrisberkhout