zscaler_zia: Remove url field from web logs by kcreddy · Pull Request #10776 · elastic/integrations

kcreddy · 2024-08-13T04:30:05Z

Proposed commit message

Remove url field from web logs.

The alternate field named eurl is added in 3.0.3 version which is
primarily used for url parsing. The url field is not used inside the pipeline
other than storing in a custom variable. It is also causing json parsing
errors when it contains non-escaped characters such as ".

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=web

--- Test results for package: zscaler_zia - START ---
╭─────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME                                             │ RESULT │ TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────┼────────┼──────────────┤
│ zscaler_zia │ web         │ pipeline  │ (ingest pipeline warnings test-web-http-endpoint.log) │ PASS   │ 510.314875ms │
│ zscaler_zia │ web         │ pipeline  │ (ingest pipeline warnings test-web.log)               │ PASS   │ 436.758708ms │
│ zscaler_zia │ web         │ pipeline  │ test-web-http-endpoint.log                            │ PASS   │    126.314ms │
│ zscaler_zia │ web         │ pipeline  │ test-web.log                                          │ PASS   │  216.41375ms │
╰─────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zscaler_zia - END   ---
Done

kcreddy · 2024-08-13T04:31:05Z

+Zscaler Web Log response format (v6):
 ```
-\{"sourcetype":"zscalernss-web","event":\{"time":"%s{time}","cloudname":"%s{cloudname}","host":"%s{host}","serverip":"%s{sip}","external_devid":"%s{external_devid}","devicemodel":"%s{devicemodel}","action":"%s{action}","recordid":"%d{recordid}","reason":"%s{reason}","threatseverity":"%s{threatseverity}","tz":"%s{tz}","filesubtype":"%s{filesubtype}","upload_filesubtype":"%s{upload_filesubtype}","sha256":"%s{sha256}","bamd5":"%s{bamd5}","filename":"%s{filename}","upload_filename":"%s{upload_filename}","filetype":"%s{filetype}","devicename":"%s{devicename}","devicehostname":"%s{devicehostname}","deviceostype":"%s{deviceostype}","deviceosversion":"%s{deviceosversion}","devicetype":"%s{devicetype}","reqsize":"%d{reqsize}","reqmethod":"%s{reqmethod}","refererurl":"%s{referer}","respsize":"%d{respsize}","respcode":"%s{respcode}","reqversion":"%s{reqversion}","respversion":"%s{respversion}","proto":"%s{proto}","company":"%s{company}","dlpmd5":"%s{dlpmd5}","apprulelabel":"%s{apprulelabel}","dlprulename":"%s{dlprulename}","rulelabel":"%s{rulelabel}","urlfilterrulelabel":"%s{urlfilterrulelabel}","cltip":"%s{cip}","cltintip":"%s{cintip}","cltsourceport":"%d{clt_sport}","threatname":"%s{threatname}","cltsslcipher":"%s{clientsslcipher}","clttlsversion":"%s{clienttlsversion}","eurl":"%s{eurl}","url":"%s{url}","useragent":"%s{ua}","login":"%s{login}","applayerprotocol":"%s{alpnprotocol}","appclass":"%s{appclass}","appname":"%s{appname}","appriskscore":"%s{app_risk_score}","bandwidthclassname":"%s{bwclassname}","bandwidthrulename":"%s{bwrulename}","bwthrottle":"%s{bwthrottle}","bypassedtime":"%s{bypassed_etime}","bypassedtraffic":"%d{bypassed_traffic}","cltsslsessreuse":"%s{clientsslsessreuse}","cltpubip":"%s{cpubip}","cltsslfailcount":"%d{cltsslfailcount}","cltsslfailreason":"%s{cltsslfailreason}","contenttype":"%s{contenttype}","datacentercity":"%s{datacentercity}","datacentercountry":"%s{datacentercountry}","datacenter":"%s{datacenter}","day":"%s{day}","day_of_month":"%02d{dd}","dept":"%s{dept}","deviceappversion":"%s{deviceappversion}","deviceowner":"%s{deviceowner}","df_hosthead":"%s{df_hosthead}","df_hostname":"%s{df_hostname}","dlpdicthitcount":"%s{dlpdicthitcount}","dlpdict":"%s{dlpdict}","dlpeng":"%s{dlpeng}","dlpidentifier":"%d{dlpidentifier}","eedone":"%s{eedone}","epochtime":"%d{epochtime}","fileclass":"%s{fileclass}","flow_type":"%s{flow_type}","forward_gateway_ip":"%s{fwd_gw_ip}","forward_gateway_name":"%s{fwd_gw_name}","forward_type":"%s{fwd_type}","hour":"%02d{hh}","is_sslexpiredca":"%s{is_sslexpiredca}","is_sslselfsigned":"%s{is_sslselfsigned}","is_ssluntrustedca":"%s{is_ssluntrustedca}","keyprotectiontype":"%s{keyprotectiontype}","location":"%s{location}","malwarecategory":"%s{malwarecat}","malwareclass":"%s{malwareclass}","minute":"%02d{mm}","mobappcategory":"%s{mobappcat}","mobappname":"%s{mobappname}","mobdevtype":"%s{mobdevtype}","module":"%s{module}","month":"%s{mon}","month_of_year":"%02d{mth}","nssserviceip":"%s{nsssvcip}","oapprulelabel":"%s{oapprulelabel}","obwclassname":"%s{obwclassname}","ocip":"%d{ocip}","ocpubip":"%d{ocpubip}","odevicehostname":"%s{odevicehostname}","odevicename":"%s{odevicename}","odeviceowner":"%s{odeviceowner}","odlpdict":"%s{odlpdict}","odlpeng":"%s{odlpeng}","odlprulename":"%s{odlprulename}","ofwd_gw_name":"%s{ofwd_gw_name}","ologin":"%s{ologin}","ordr_rulename":"%s{ordr_rulename}","ourlcat":"%s{ourlcat}","ourlfilterrulelabel":"%s{ourlfilterrulelabel}","ozpa_app_seg_name":"%s{ozpa_app_seg_name}","externalsslpolicyreason":"%s{externalspr}","productversion":"%s{productversion}","rdr_rulename":"%s{rdr_rulename}","refererhost":"%s{refererhost}","reqheadersize":"%d{reqhdrsize}","reqdatasize":"%d{reqdatasize}","respheadersize":"%d{resphdrsize}","respdatasize":"%d{respdatasize}","riskscore":"%d{riskscore}","ruletype":"%s{ruletype}","second":"%02d{ss}","srvcertchainvalpass":"%s{srvcertchainvalpass}","srvcertvalidationtype":"%s{srvcertvalidationtype}","srvcertvalidityperiod":"%s{srvcertvalidityperiod}","srvsslcipher":"%s{srvsslcipher}","serversslsessreuse":"%s{serversslsessreuse}","srvocspresult":"%s{srvocspresult}","srvtlsversion":"%s{srvtlsversion}","srvwildcardcert":"%s{srvwildcardcert}","ssldecrypted":"%s{ssldecrypted}","throttlereqsize":"%d{throttlereqsize}","throttlerespsize":"%d{throttlerespsize}","totalsize":"%d{totalsize}","trafficredirectmethod":"%s{trafficredirectmethod}","unscannabletype":"%s{unscannabletype}","upload_doctypename":"%s{upload_doctypename}","upload_fileclass":"%s{upload_fileclass}","upload_filetype":"%s{upload_filetype}","urlcatmethod":"%s{urlcatmethod}","urlsubcat":"%s{urlcat}","urlsupercat":"%s{urlsupercat}","urlclass":"%s{urlclass}","useragentclass":"%s{uaclass}","useragenttoken":"%s{ua_token}","userlocationname":"%s{userlocationname}","year":"%04d{yyyy}","ztunnelversion":"%s{ztunnelversion}","zpa_app_seg_name":"%s{zpa_app_seg_name}"\}\}
+\{"sourcetype":"zscalernss-web","event":\{"time":"%s{time}","cloudname":"%s{cloudname}","host":"%s{host}","serverip":"%s{sip}","external_devid":"%s{external_devid}","devicemodel":"%s{devicemodel}","action":"%s{action}","recordid":"%d{recordid}","reason":"%s{reason}","threatseverity":"%s{threatseverity}","tz":"%s{tz}","filesubtype":"%s{filesubtype}","upload_filesubtype":"%s{upload_filesubtype}","sha256":"%s{sha256}","bamd5":"%s{bamd5}","filename":"%s{filename}","upload_filename":"%s{upload_filename}","filetype":"%s{filetype}","devicename":"%s{devicename}","devicehostname":"%s{devicehostname}","deviceostype":"%s{deviceostype}","deviceosversion":"%s{deviceosversion}","devicetype":"%s{devicetype}","reqsize":"%d{reqsize}","reqmethod":"%s{reqmethod}","refererurl":"%s{referer}","respsize":"%d{respsize}","respcode":"%s{respcode}","reqversion":"%s{reqversion}","respversion":"%s{respversion}","proto":"%s{proto}","company":"%s{company}","dlpmd5":"%s{dlpmd5}","apprulelabel":"%s{apprulelabel}","dlprulename":"%s{dlprulename}","rulelabel":"%s{rulelabel}","urlfilterrulelabel":"%s{urlfilterrulelabel}","cltip":"%s{cip}","cltintip":"%s{cintip}","cltsourceport":"%d{clt_sport}","threatname":"%s{threatname}","cltsslcipher":"%s{clientsslcipher}","clttlsversion":"%s{clienttlsversion}","eurl":"%s{eurl}","useragent":"%s{ua}","login":"%s{login}","applayerprotocol":"%s{alpnprotocol}","appclass":"%s{appclass}","appname":"%s{appname}","appriskscore":"%s{app_risk_score}","bandwidthclassname":"%s{bwclassname}","bandwidthrulename":"%s{bwrulename}","bwthrottle":"%s{bwthrottle}","bypassedtime":"%s{bypassed_etime}","bypassedtraffic":"%d{bypassed_traffic}","cltsslsessreuse":"%s{clientsslsessreuse}","cltpubip":"%s{cpubip}","cltsslfailcount":"%d{cltsslfailcount}","cltsslfailreason":"%s{cltsslfailreason}","contenttype":"%s{contenttype}","datacentercity":"%s{datacentercity}","datacentercountry":"%s{datacentercountry}","datacenter":"%s{datacenter}","day":"%s{day}","day_of_month":"%02d{dd}","dept":"%s{dept}","deviceappversion":"%s{deviceappversion}","deviceowner":"%s{deviceowner}","df_hosthead":"%s{df_hosthead}","df_hostname":"%s{df_hostname}","dlpdicthitcount":"%s{dlpdicthitcount}","dlpdict":"%s{dlpdict}","dlpeng":"%s{dlpeng}","dlpidentifier":"%d{dlpidentifier}","eedone":"%s{eedone}","epochtime":"%d{epochtime}","fileclass":"%s{fileclass}","flow_type":"%s{flow_type}","forward_gateway_ip":"%s{fwd_gw_ip}","forward_gateway_name":"%s{fwd_gw_name}","forward_type":"%s{fwd_type}","hour":"%02d{hh}","is_sslexpiredca":"%s{is_sslexpiredca}","is_sslselfsigned":"%s{is_sslselfsigned}","is_ssluntrustedca":"%s{is_ssluntrustedca}","keyprotectiontype":"%s{keyprotectiontype}","location":"%s{location}","malwarecategory":"%s{malwarecat}","malwareclass":"%s{malwareclass}","minute":"%02d{mm}","mobappcategory":"%s{mobappcat}","mobappname":"%s{mobappname}","mobdevtype":"%s{mobdevtype}","module":"%s{module}","month":"%s{mon}","month_of_year":"%02d{mth}","nssserviceip":"%s{nsssvcip}","oapprulelabel":"%s{oapprulelabel}","obwclassname":"%s{obwclassname}","ocip":"%d{ocip}","ocpubip":"%d{ocpubip}","odevicehostname":"%s{odevicehostname}","odevicename":"%s{odevicename}","odeviceowner":"%s{odeviceowner}","odlpdict":"%s{odlpdict}","odlpeng":"%s{odlpeng}","odlprulename":"%s{odlprulename}","ofwd_gw_name":"%s{ofwd_gw_name}","ologin":"%s{ologin}","ordr_rulename":"%s{ordr_rulename}","ourlcat":"%s{ourlcat}","ourlfilterrulelabel":"%s{ourlfilterrulelabel}","ozpa_app_seg_name":"%s{ozpa_app_seg_name}","externalsslpolicyreason":"%s{externalspr}","productversion":"%s{productversion}","rdr_rulename":"%s{rdr_rulename}","refererhost":"%s{refererhost}","reqheadersize":"%d{reqhdrsize}","reqdatasize":"%d{reqdatasize}","respheadersize":"%d{resphdrsize}","respdatasize":"%d{respdatasize}","riskscore":"%d{riskscore}","ruletype":"%s{ruletype}","second":"%02d{ss}","srvcertchainvalpass":"%s{srvcertchainvalpass}","srvcertvalidationtype":"%s{srvcertvalidationtype}","srvcertvalidityperiod":"%s{srvcertvalidityperiod}","srvsslcipher":"%s{srvsslcipher}","serversslsessreuse":"%s{serversslsessreuse}","srvocspresult":"%s{srvocspresult}","srvtlsversion":"%s{srvtlsversion}","srvwildcardcert":"%s{srvwildcardcert}","ssldecrypted":"%s{ssldecrypted}","throttlereqsize":"%d{throttlereqsize}","throttlerespsize":"%d{throttlerespsize}","totalsize":"%d{totalsize}","trafficredirectmethod":"%s{trafficredirectmethod}","unscannabletype":"%s{unscannabletype}","upload_doctypename":"%s{upload_doctypename}","upload_fileclass":"%s{upload_fileclass}","upload_filetype":"%s{upload_filetype}","urlcatmethod":"%s{urlcatmethod}","urlsubcat":"%s{urlcat}","urlsupercat":"%s{urlsupercat}","urlclass":"%s{urlclass}","useragentclass":"%s{uaclass}","useragenttoken":"%s{ua_token}","userlocationname":"%s{userlocationname}","year":"%04d{yyyy}","ztunnelversion":"%s{ztunnelversion}","zpa_app_seg_name":"%s{zpa_app_seg_name}"\}\}


Change here is the removal of url field. Rest of the template is the same.

elasticmachine · 2024-08-13T05:18:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-08-13T05:19:15Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

Just a query about the removal of the custom field.

efd6 · 2024-08-13T05:24:21Z

-            - name: name
-              type: keyword
-              description: The destination URL.


Will this break people?

This does have potential for breaking, if they are already using this custom field for dashboards. So, I added it back, but the value is derived from decoded eurl field in this commit.

Perfect. Thanks.

efd6 · 2024-08-13T06:59:24Z

Needs a rebuild. Then good.

elasticmachine · 2024-08-13T07:43:05Z

💚 Build Succeeded

Buildkite Build
Commit: 5203117

History

💔 Build #14616 failed e60109a
💚 Build #14614 succeeded b4eb58b
💔 Build #14613 failed 62a9305

cc @kcreddy

elastic-sonarqube · 2024-08-13T07:43:09Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-08-13T08:16:47Z

Package zscaler_zia - 3.1.0 containing this change is available at https://epr.elastic.co/search?package=zscaler_zia

Remove url field from web logs. The alternate field named `eurl` is added in `3.0.3` version which is primarily used for url parsing. The `url` field is not used inside the pipeline other than storing in a custom variable. It is also causing json parsing errors when it contains non-escaped characters such as `"`.

Remove url field from web logs

62a9305

kcreddy commented Aug 13, 2024

View reviewed changes

update changelog

b4eb58b

kcreddy self-assigned this Aug 13, 2024

kcreddy added enhancement New feature or request Integration:zscaler_zia Zscaler Internet Access Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 13, 2024

kcreddy marked this pull request as ready for review August 13, 2024 05:19

kcreddy requested a review from a team as a code owner August 13, 2024 05:19

efd6 approved these changes Aug 13, 2024

View reviewed changes

Add web.url.name from decoded eurl

e60109a

update readme

5203117

kcreddy merged commit a235a80 into elastic:main Aug 13, 2024

kcreddy deleted the zscaler_zia_remove-url branch February 7, 2025 09:12

Navigation Menu

Uh oh!

zscaler_zia: Remove url field from web logs - #10776

zscaler_zia: Remove url field from web logs#10776
kcreddy merged 4 commits into
elastic:mainfrom
kcreddy:zscaler_zia_remove-url

kcreddy commented Aug 13, 2024 •

edited

Loading

kcreddy Aug 13, 2024

elasticmachine commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

efd6 left a comment

efd6 Aug 13, 2024

kcreddy Aug 13, 2024

efd6 Aug 13, 2024

efd6 commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

elastic-sonarqube Bot commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

Labels

3 participants

Uh oh!

Conversation

kcreddy commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

kcreddy Aug 13, 2024

Choose a reason for hiding this comment

elasticmachine commented Aug 13, 2024

🚀 Benchmarks report

elasticmachine commented Aug 13, 2024

efd6 left a comment

Choose a reason for hiding this comment

efd6 Aug 13, 2024

Choose a reason for hiding this comment

kcreddy Aug 13, 2024

Choose a reason for hiding this comment

efd6 Aug 13, 2024

Choose a reason for hiding this comment

efd6 commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Aug 13, 2024

Quality Gate passed

elasticmachine commented Aug 13, 2024

Labels

3 participants

kcreddy commented Aug 13, 2024 •

edited

Loading