[zscaler_zia] In web v8, use b64 fields to avoid encoding issues

[chrisberkhout]

Proposed commit message

[zscaler_zia] In web v8, use b64 fields to avoid encoding issues (#)

In the web data stream, the v7 response format requested the
hex-encoded fields `eurl`, `eurlpath`, `ereferer`, and `erefererpath`.
Although those fields have non-printable characters encoded, they
aren't fully URL-encoded (e.g. they don't encode `%`), and they may
still include characters that can break JSON (including `"`).

URLs and their paths sometimes include characters that break the v7
processing, so the following changes have been made in the v8 response
format and pipeline.

We switch from `eurl` to `b64url` and from `ereferer` to `b64referer`.

We no longer request the `eurlpath` and `erefererpath` fields. They
have no base64-encoded equivalents. Paths and query strings are
present in the URL fields.

From the output we remove the `zscaler_zia.web.url.path` and
`zscaler_zia.web.referer.path` fields. However, use of the `uri_parts`
processor to populate `url.path` and other `url.*` fields remains
intact.

The ingest pipeline will tolerate absent fields, so the updated
package will accept any v7 data that isn't broken by problematic
characters and simply skip population of url/path fields.

A new `"version":"v8"` field is inserted at the start of the response
format to aid future debugging. No validation or use of that field has
been added to the ingest pipeline.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Review tips

The following diff compares pretty-printed versions of the v7 and v8 formats:

1a2
>   "version": "v8",
29,30c30
<     "refererurl": "%s{ereferer}",
<     "refererpath": "%s{erefererpath}",
---
>     "b64referer": "%s{b64referer}",
48,49c48
<     "eurl": "%s{eurl}",
<     "urlpath": "%s{eurlpath}",
---
>     "b64url": "%s{b64url}",

You can generate diffs of the the pipeline test files in-pretty printed form as follows, from the package directory:

diff <(git show main:packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http-endpoint.log | jq .) \
     <(jq . data_stream/web/_dev/test/pipeline/test-web-http-endpoint.log)

diff <(git show main:packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log | jq .) \
     <(jq . data_stream/web/_dev/test/pipeline/test-web.log)

Note that in one example, I combined a URL value and path in the b64 version, since it seems that URL fields do have the full URL, not only the base.

I did a non-strict URL-decode and Base64 encode on the old values using this CyberChef recipe.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 57523b7

History

• 💔 Build #14926 failed 232ec3c

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package zscaler_zia - 3.2.1 containing this change is available at https://epr.elastic.co/search?package=zscaler_zia