imperva_cloud_waf: step over non-UTF-8 data work items

[efd6]

Proposed commit message

imperva_cloud_waf: step over non-UTF-8 data work items

It is possible for users to incorrectly configure their Imperva Cloud
WAF logging to send compressed data (this is the default). This results
in a CEL evaluation failure since we depend on the data being string
CEF. When the non-UTF-8 data is converted to a string, CEL refuses to
peform the conversion by design. When a user has done this, it appears
that logs in the compressed form persist, so the agent is unable to move
past the integration-invalid data. This change tries to perform the
conversion, falling back to an error message being sent to the index
when it is not possible. This helps identify cases where the
configuration is incorrect, and allows the collection to step over the
bodies that are not consumable.

Unfortunately we cannot make use of fleet health notifications since
sending object errors prevents cursor updates, and so would result in
continuing to be stuck.

Warning

The CEL input in this integration is not tested in system tests. Please review with extra care.

PoC:

mito src.cel
! stderr .
cmp stdout want.txt

-- src.cel --
[
	b"\xc3\x28",
	b"good message",
].map(body,
	{
		"events": try(string(body), "error").as(body, type(body) == type("") ?
			[{"message": body}]
		:
			[{"error": {"message": body.error}}]
		),
	}
)
-- want.txt --
[
	{
		"events": [
			{
				"error": {
					"message": "invalid UTF-8 in bytes, cannot convert to string"
				}
			}
		]
	},
	{
		"events": [
			{
				"message": "good message"
			}
		]
	}
]

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

Very nice.

The commit description is good and clear.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 975b717

History

• 💚 Build #23492 succeeded 8f5776e

cc @efd6

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package imperva_cloud_waf - 1.11.0 containing this change is available at https://epr.elastic.co/package/imperva_cloud_waf/1.11.0/