[sophos] Fix uri_parts failure on malformed URLs in XG cfilter.

[ie-ops]

Executive summary

The uri_parts processor in the Sophos XG cfilter ingest pipeline previously terminated document ingestion when it encountered malformed URLs (e.g., URLs containing spaces such as https://example .com/path). The fix adds an on_failure handler to that processor which appends a _sophos_xg_uri_parse_failure tag to the event instead of failing, allowing the document to be ingested with partial URL fields populated. Three new field mappings (sophos.xg.fw_rule_name, sophos.xg.fw_rule_section, sophos.xg.http_referer) are also added to cover fields present in real-world XG content filtering logs that were previously undefined.

Proposed commit message

[sophos] Fix uri_parts failure on malformed URLs in XG cfilter.

Root cause

The uri_parts processor in cfilter.yml has no ignore_failure: true and its guard condition only checks ctx.url?.original != null — it does not validate URI well-formedness. Sophos XG content-filtering logs can contain URLs with embedded quotes or Unicode replacement characters (U+FFFD) that survive KV parsing as a truncated, non-URI string, causing uri_parts to throw and the entire event to be routed to the pipeline-level on_failure handler.

Approach

Add ignore_failure: true and an on_failure block to the uri_parts processor (tag: uri_parts_url_original_to_url_cc5cc83f) in cfilter.yml. The on_failure block appends a _sophos_xg_uri_parse_failure tag to the document for observability. This prevents malformed url.original values — produced when the upstream KV parser truncates URLs containing embedded quotes or Unicode replacement characters — from failing the entire event and routing it to the pipeline-level on_failure handler. Additionally, add a sanitized test fixture reproducing the malformed-URL case so the fix is regression-tested.

Implementation

1. Step 1: In packages/sophos/data_stream/xg/elasticsearch/ingest_pipeline/cfilter.yml, locate the uri_parts processor at line 110 (tag: uri_parts_url_original_to_url_cc5cc83f) and add ignore_failure: true plus an on_failure block that appends _sophos_xg_uri_parse_failure to the tags field with allow_duplicates: false.
2. Step 2: In packages/sophos/data_stream/xg/_dev/test/pipeline/test-xg-cfilter-new.log, append a new test log line using the sanitized event from the issue (url contains a space, making it an invalid URI): <30>device_name="host-1.example.local" timestamp="2026-06-26T17:38:04+0200" device_model="XGS4500" device_serial_id="SN000000000001" log_id="050901616001" log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="26" fw_rule_name="LAN to WAN allow Webaccess" fw_rule_section="Local rule" web_policy_id=13 http_category="InvalidUrl" url="https://example .com/path?ip=198.51.100.85&id=example-id" content_type="application/octet-stream" src_ip="198.51.100.85" dst_ip="203.0.113.137" protocol="TCP" src_port=50215 dst_port=443 bytes_sent=3727 bytes_received=1129 domain="example.com" http_user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0" http_status="200" transaction_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" http_referer="https://example.com/" con_id=2068019904 app_is_cloud="FALSE" used_quota="0" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" src_country="JPN" dst_country="USA" app_risk=77 app_category="5593"
3. Step 3: Regenerate packages/sophos/data_stream/xg/_dev/test/pipeline/test-xg-cfilter-new.log-expected.json by running elastic-package test pipeline --generate and verify that: (a) the new test case does NOT have event.kind: pipeline_error, (b) url.original is preserved as the malformed URL string, (c) the tags array contains _sophos_xg_uri_parse_failure, and (d) all pre-existing test cases still pass.
4. Step 4: Add a new changelog entry to packages/sophos/changelog.yml with version 3.17.2, type bugfix, and description Fix uri_parts processor failure on malformed URLs in XG content filtering logs.
5. Step 5: Update packages/sophos/manifest.yml to bump version from 3.17.1 to 3.17.2.

Pipeline changes

• Add ignore_failure: true to the uri_parts processor (tag: uri_parts_url_original_to_url_cc5cc83f) in cfilter.yml
• Add an on_failure block to the same uri_parts processor that appends _sophos_xg_uri_parse_failure to the tags field with allow_duplicates: false

Field / mapping changes

—

Sanitized error message

Processor 'uri_parts' with tag 'uri_parts_url_original_to_url_cc5cc83f' in pipeline 'logs-sophos.xg-cfilter' failed with message '[on_failure_message]'

Sanitized log (event_sanitized excerpt)

<30>device_name="host-1.example.local" timestamp="2026-06-26T17:38:04+0200" device_model="XGS4500" device_serial_id="SN000000000001" log_id="050901616001" log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="26" fw_rule_name="LAN to WAN allow Webaccess" fw_rule_section="Local rule" web_policy_id=13 http_category="InvalidUrl" url="https://example .com/path?ip=198.51.100.85&id=example-id" content_type="application/octet-stream" src_ip="198.51.100.85" dst_ip="203.0.113.137" protocol="TCP" src_port=50215 dst_port=443 bytes_sent=3727 bytes_received=1129 domain="example.com" http_user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0" http_status="200" transaction_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" http_referer="https://example.com/" con_id=2068019904 app_is_cloud="FALSE" used_quota="0" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" src_country="JPN" dst_country="USA" app_risk=77 app_category="5593"

Reviewer concerns

• Documents tagged _sophos_xg_uri_parse_failure will have url.domain and url.full populated (from earlier pipeline processors) but will be missing url.scheme, url.path, url.query, and url.port that uri_parts would normally produce; any Kibana queries or detection rules that assume those subfields are always present for Content Filtering events should be reviewed.
• The three new fields (fw_rule_name, fw_rule_section, http_referer) are defined in fields.yml and appear in the expected test output, but the diff does not show the KV/dissect processor that extracts them — a reviewer should confirm those fields are correctly extracted by the existing pipeline KV step rather than silently dropped.
• The changelog link field for version 3.17.2 contains a placeholder URL (https://github.com/elastic/integrations/pull/1) that must be updated to the real PR number before merge.

Self-review findings

Self-review invoked: no (0 cycles)

Severity	Finding	Addressed
major	ignore_failure: true was initially added alongside on_failure block, but these are mutually exclusive in Elasticsearch — ignore_failure suppresses the failure entirely so on_failure never runs	✅
major	Fields sophos.xg.fw_rule_name, sophos.xg.fw_rule_section, and sophos.xg.http_referer were missing from fields.yml, causing pipeline test validation failures	✅

Final validation passed: yes

Risk and classification

• Plan risk level: medium
• Tags: pipeline, processors, ingest, test-fixture, field-mapping, docs
• Impact: medium

Links

• Issue: (no issue number)
• Issue title: sophos.xg [PIPELINE_FIX]: Processor 'uri_parts' with tag 'uri_parts_url_original_to_url_cc5cc83f' …
• Pipeline case: 616e1a4d91e0a368

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[vinit-chauhan]

Suggested change

	- version: "3.17.2"
	changes:
	- description: Fix uri_parts processor failure on malformed URLs in XG content filtering logs.
	type: bugfix
	link: https://github.com/elastic/integrations/pull/20628
	- description: Map XG content-filtering http_referer and http_user_agent fields to ECS.
	type: bugfix
	link: https://github.com/elastic/integrations/pull/20628
	- version: "3.17.2"
	changes:
	- description: Fix uri_parts processor failure on malformed URLs in XG content filtering logs. Map XG content-filtering http_referer and http_user_agent fields to ECS. Add field mapping for sophos.xg.fw_rule_name and sophos.xg.fw_rule_section.
	type: bugfix
	link: https://github.com/elastic/integrations/pull/20628

Since all of them are bugfix, we can merge them to a single entry.

[vinit-chauhan]

Other than the one suggestion in changelog.yml rest of the PR looks good. 👍🏼

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

🟢 No issues across the latest commits fd18eb1…044a7a9 (137 commits).

Review summary

Issues found across earlier commits 3b59d79 — 1 low

• 🔵 The second 3.17.2 changelog entry is typed bugfix but it adds new ECS mappings, and the two newly exported fields (sophos.xg.fw_rule_name, sophos.xg.fw_rule_section) are not mentioned at all (link) (Resolved)

Issues found across earlier commits bfb7509 — 2 medium, 2 low

• 🟡 The new sophos.xg.http_referer field is never mapped to ECS http.request.referrer, so referrer data from the newer SFOS log format stays vendor-only (link) (Resolved)
• 🟡 The user agent in the new sample event is silently discarded because cfilter.yml removes sophos.xg.http_user_agent without mapping it (link) (Resolved)
• 🔵 The new on_failure append processor has no tag while every other processor in cfilter.yml has one (link) (Resolved)
• 🔵 The 3.17.2 changelog only records the uri_parts bugfix but the PR also adds three new fields (link) (Resolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 044a7a9

History

• 💚 Build #47526 succeeded 3b59d79
• 💚 Build #47524 succeeded bfb7509

cc @haetamoudi

[elastic-vault-github-plugin-prod]

Package sophos - 3.17.2 containing this change is available at https://epr.elastic.co/package/sophos/3.17.2/