[qnap_nas] Add RFC 5424 QuLog@Access syslog format support.

[ie-ops]

Executive summary

The ingest pipeline for qnap_nas.log was failing with MISSING_CASE errors because QTS 5.x devices emit logs in RFC 5424 format (ISO 8601 timestamps, structured-data blocks) while the existing grok processor only covered the legacy BSD syslog format. This fix adds a new grok pattern for RFC 5424 messages before the existing BSD pattern, a kv processor plus supplementary grok steps to extract QuLog@Access structured-data key-value pairs (including multi-word quoted values like client_app and client_agent), rename processors to map parsed fields to ECS, a painless script to translate numeric action codes to event.action vocabulary, and new ECS and custom field declarations for source.mac, user_agent.original, qnap.nas.client_app, qnap.nas.client_id, and qnap.nas.action_code.

Proposed commit message

[qnap_nas] Add RFC 5424 QuLog@Access syslog format support.

The ingest pipeline for qnap_nas.log was failing with MISSING_CASE
errors because QTS 5.x devices emit logs in RFC 5424 format (ISO 8601
timestamps, structured-data blocks) while the existing grok processor
only covered the legacy BSD syslog format. This fix adds a new grok
pattern for RFC 5424 messages before the existing BSD pattern, a kv
processor plus supplementary grok steps to extract QuLog@Access
structured-data key-value pairs (including multi-word quoted values like
client_app and client_agent), rename processors to map parsed fields to
ECS, a painless script to translate numeric action codes to event.action
vocabulary, and new ECS and custom field declarations for source.mac,
user_agent.original, qnap.nas.client_app, qnap.nas.client_id, and
qnap.nas.action_code.

  - Add RFC 5424 grok pattern as first alternative in
    grok_event_original_cad2ef7a: '^(%{ECS_SYSLOG_PRI})?1
    %{TIMESTAMP_ISO8601:_tmp.timestamp}
    (?:%{IP:host.ip}|%{HOSTNAME:host.name}) qulogd:?
    %{POSINT:process.pid:int} - \[QuLog@Access
    %{DATA:_tmp.structured_data}\] %{GREEDYDATA:_tmp.message}'

  - Add ISO8601 to formats list in
    date__tmp_timestamp_to_@timestamp_e440143c (with timezone branch)

  - Add ISO8601 to formats list in
    date__tmp_timestamp_to_@timestamp_baf3310e (without timezone branch)

  - Add condition 'if: ctx._tmp?.structured_data == null' to
    grok__tmp_message_c75b80dc to skip RFC 5424 events

  - Add kv processor: field=_tmp.structured_data, field_split=' ',
    value_split='=', prefix='_tmp.sd.', trim_value='"', if:
    ctx._tmp?.structured_data != null

  - Add rename processor: _tmp.sd.ip → source.address (if RFC 5424 path)

  - Add rename processor: _tmp.sd.user → user.name (if RFC 5424 path)

  - Add set processor: source.mac from _tmp.sd.mac (if RFC 5424 path)

  - Add rename processor: _tmp.sd.computer → source.domain (if RFC 5424
    path and not '---')

  - Add rename processor: _tmp.sd.application → qnap.nas.application (if
    RFC 5424 path and not '---')

  - Add rename processor: _tmp.sd.client_agent → user_agent.original (if
    RFC 5424 path)

  - Add rename processor: _tmp.sd.client_app → qnap.nas.client_app (if
    RFC 5424 path)

  - Add rename processor: _tmp.sd.client_id → qnap.nas.client_id (if RFC
    5424 path)

  - Add set processor: event.provider='conn-log' if RFC 5424 path and
    event.provider not yet set

  - Add Painless script processor before script_1d15b5b0: map
    _tmp.sd.action numeric codes → event.action string; map
    _tmp.sd.action_result=0 → event.outcome=success, else failure;
    unmapped codes → qnap.nas.action_code

  - Add 'external: ecs / name: source.mac' to
    data_stream/log/fields/ecs.yml

  - Add 'external: ecs / name: user_agent.original' to
    data_stream/log/fields/ecs.yml

  - Add field 'qnap.nas.client_app' (keyword) to
    data_stream/log/fields/fields.yml under qnap.nas group

  - Add field 'qnap.nas.client_id' (keyword) to
    data_stream/log/fields/fields.yml under qnap.nas group

  - Add field 'qnap.nas.action_code' (keyword) to
    data_stream/log/fields/fields.yml under qnap.nas group for unmapped
    numeric action codes

Root cause

The grok_event_original_cad2ef7a processor contains only an RFC 3164 pattern using SYSLOGTIMESTAMP (MMM dd HH:mm:ss), but QTS 5.x devices emit RFC 5424 syslog with a version byte '1', an ISO 8601 timestamp, and a structured-data block [QuLog@Access key="val" ...] whose format never matches the existing pattern, causing every QTS 5.x event to reach the on_failure handler with pipeline_error.

Approach

Add an RFC 5424 grok pattern as the first alternative in grok_event_original_cad2ef7a to match '1 qulogd:? - [QuLog@Access ] ', then add a kv processor to parse the structured-data key-value pairs into _tmp.sd.* fields, followed by rename/set processors mapping ip→source.address, user→user.name, mac→source.mac, client_agent→user_agent.original. Add ISO8601 to the existing date processor formats, condition the second grok on RFC 3164 path only (if: _tmp.structured_data == null), and add a Painless script to map numeric action codes (512→login) combined with action_result=0→success into the existing event.action/event.outcome vocabulary consumed by the ECS categorization script.

Implementation

1. Step 1: In packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml, add an RFC 5424 grok pattern as the FIRST entry in the grok_event_original_cad2ef7a patterns list: '^(%{ECS_SYSLOG_PRI})?1 %{TIMESTAMP_ISO8601:_tmp.timestamp} (?:%{IP:host.ip}|%{HOSTNAME:host.name}) qulogd:? %{POSINT:process.pid:int} - [QuLog@Access %{DATA:_tmp.structured_data}] %{GREEDYDATA:_tmp.message}'; TIMESTAMP_ISO8601 is a built-in Elasticsearch grok pattern so no custom definition is needed.
2. Step 2: Add 'ISO8601' to the formats list of both existing date processors (date__tmp_timestamp_to_@timestamp_e440143c and date__tmp_timestamp_to_@timestamp_baf3310e) so they correctly parse the RFC 5424 ISO 8601 timestamp including the embedded +HH:MM timezone offset.
3. Step 3: Add condition 'if: ctx._tmp?.structured_data == null' to the grok__tmp_message_c75b80dc processor so it is skipped for RFC 5424 events (all data already in structured-data; running it on MSG='Administration' would cause a grok failure and pipeline_error).
4. Step 4: Add a kv processor immediately after grok__tmp_message_c75b80dc to parse _tmp.structured_data into _tmp.sd.* sub-fields: field=_tmp.structured_data, field_split=' ', value_split='=', prefix='_tmp.sd.', trim_value='"', ignore_missing=true, if: ctx._tmp?.structured_data != null.
5. Step 5: Add rename processors (all with ignore_missing=true and conditional on ctx._tmp?.structured_data != null) mapping: _tmp.sd.ip→source.address, _tmp.sd.user→user.name, _tmp.sd.mac→source.mac, _tmp.sd.computer→source.domain, _tmp.sd.application→qnap.nas.application, _tmp.sd.client_agent→user_agent.original, _tmp.sd.client_app→qnap.nas.client_app, _tmp.sd.client_id→qnap.nas.client_id.
6. Step 6: Add a set processor to assign event.provider='conn-log' when the event came from RFC 5424 and event.provider was not set by the RFC 3164 grok (if: ctx._tmp?.structured_data != null && ctx.event?.provider == null), ensuring downstream gsub and categorization processors receive a valid event.provider.
7. Step 7: Add a Painless script processor placed BEFORE the existing ECS categorization script (script_1d15b5b0) to map numeric action codes to the human-readable event.action vocabulary: 512+action_result=0→'login-success', 512+action_result!=0→'login-fail'; for unmapped codes preserve the raw numeric value in qnap.nas.action_code. Also set event.outcome='success' when action_result='0', 'failure' otherwise (only when _tmp.sd.action_result is present).
8. Step 8: Update packages/qnap_nas/data_stream/log/fields/ecs.yml: add 'external: ecs / name: source.mac' and 'external: ecs / name: user_agent.original'.
9. Step 9: Update packages/qnap_nas/data_stream/log/fields/fields.yml: add 'name: qnap.nas.client_app, type: keyword, description: Client application name from RFC 5424 QuLog@Access structured data' and 'name: qnap.nas.client_id, type: keyword, description: Client session/device ID from RFC 5424 QuLog@Access structured data' and 'name: qnap.nas.action_code, type: keyword, description: Numeric action code from RFC 5424 QuLog@Access when no mapping is defined' under the qnap.nas group.
10. Step 10: Add the sanitized RFC 5424 event to packages/qnap_nas/data_stream/log/_dev/test/pipeline/test-access.log and add its expected-output object to test-access.log-expected.json, verifying: @timestamp parses to UTC from +01:00, source.ip=192.0.2.10, user.name=alice.johnson, source.mac=00-00-5E-00-53-23, user_agent.original set to the full UA string, event.action=login-success, event.outcome=success, event.category=[authentication], event.type=[start], event.provider=conn-log, related.ip=[192.0.2.10], related.user=[alice.johnson], process.pid=19683.
11. Step 11: Update packages/qnap_nas/changelog.yml with a new enhancement entry at the top, bump version in changelog and manifest.yml from 1.25.3 to 1.26.0.

Pipeline changes

• Add RFC 5424 grok pattern as first alternative in grok_event_original_cad2ef7a: '^(%{ECS_SYSLOG_PRI})?1 %{TIMESTAMP_ISO8601:_tmp.timestamp} (?:%{IP:host.ip}|%{HOSTNAME:host.name}) qulogd:? %{POSINT:process.pid:int} - [QuLog@Access %{DATA:_tmp.structured_data}] %{GREEDYDATA:_tmp.message}'
• Add ISO8601 to formats list in date__tmp_timestamp_to_@timestamp_e440143c (with timezone branch)
• Add ISO8601 to formats list in date__tmp_timestamp_to_@timestamp_baf3310e (without timezone branch)
• Add condition 'if: ctx._tmp?.structured_data == null' to grok__tmp_message_c75b80dc to skip RFC 5424 events
• Add kv processor: field=_tmp.structured_data, field_split=' ', value_split='=', prefix='_tmp.sd.', trim_value='"', if: ctx._tmp?.structured_data != null
• Add rename processor: _tmp.sd.ip → source.address (if RFC 5424 path)
• Add rename processor: _tmp.sd.user → user.name (if RFC 5424 path)
• Add set processor: source.mac from _tmp.sd.mac (if RFC 5424 path)
• Add rename processor: _tmp.sd.computer → source.domain (if RFC 5424 path and not '---')
• Add rename processor: _tmp.sd.application → qnap.nas.application (if RFC 5424 path and not '---')
• Add rename processor: _tmp.sd.client_agent → user_agent.original (if RFC 5424 path)
• Add rename processor: _tmp.sd.client_app → qnap.nas.client_app (if RFC 5424 path)
• Add rename processor: _tmp.sd.client_id → qnap.nas.client_id (if RFC 5424 path)
• Add set processor: event.provider='conn-log' if RFC 5424 path and event.provider not yet set
• Add Painless script processor before script_1d15b5b0: map _tmp.sd.action numeric codes → event.action string; map _tmp.sd.action_result=0 → event.outcome=success, else failure; unmapped codes → qnap.nas.action_code

Field / mapping changes

• Add 'external: ecs / name: source.mac' to data_stream/log/fields/ecs.yml
• Add 'external: ecs / name: user_agent.original' to data_stream/log/fields/ecs.yml
• Add field 'qnap.nas.client_app' (keyword) to data_stream/log/fields/fields.yml under qnap.nas group
• Add field 'qnap.nas.client_id' (keyword) to data_stream/log/fields/fields.yml under qnap.nas group
• Add field 'qnap.nas.action_code' (keyword) to data_stream/log/fields/fields.yml under qnap.nas group for unmapped numeric action codes

Sanitized error message

Processor 'grok' with tag 'grok_event_original_cad2ef7a' in pipeline 'logs-qnap_nas.log-default' failed with message '[on_failure_message]'

Sanitized log (event_sanitized excerpt)

<30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration

Reviewer concerns

• The kv processor (field_split=' ') splits on spaces and will mangle quoted multi-word values such as client_app="Web Desktop" and client_agent="Mozilla/5.0 …". The fix handles this with dedicated grok processors that run after kv and overwrite the truncated kv values. This is correct for the known fields but could silently drop or truncate any future QuLog@Access keys whose values contain spaces unless a corresponding grok override is added.
• The action-code Painless script maps only code 512 (login). All other numeric codes fall through to qnap.nas.action_code. If QNAP publishes additional codes, reviewers may want a follow-up issue to extend the mapping table.
• The source key in QuLog@Access structured data (e.g., source="example-source") is extracted by kv but never renamed to a target field and is silently discarded. It is unclear what this field represents; a comment or intentional ignore would clarify intent.
• Date processor behaviour: when event.timezone is set by agent config AND the ISO 8601 timestamp already embeds a timezone offset, Elasticsearch converts to the configured timezone rather than preserving the original offset. The test fixture confirms this produces correct UTC-equivalent timestamps, but operators should be aware that the displayed timezone in event.created/@timestamp will reflect the agent's configured timezone, not the device's local offset.

Self-review findings

Self-review invoked: yes (1 cycle)

Severity	Finding	Addressed
major	RFC 5424 MSG field ('Administration') is not mapped to qnap.nas.application, creating an inconsistency with the RFC 3164 path Out of scope per the approved plan: no detailed_step maps the RFC 5424 MSG token. The RFC 5424 structured data carries application="---" which is correctly discarded by the null-cleanup script; mapping MSG would require a separate enhancement to define the semantics.	⚠️
major	changelog link is placeholder pull/1 rather than a real PR URL Per the task instructions the placeholder #1 is required and will be updated to the real PR URL after the PR is created.	⚠️
major	kv ignore_failure silently swallows atomic failure; ip/user/mac could be lost if kv fails before parsing them Design tradeoff: the two supplementary grok processors (grok_rfc5424_client_app, grok_rfc5424_client_agent) run after kv and correctly restore multi-word values. The kv failure point is deterministic (always on the 'Desktop"' token that follows client_id, after all simple fields are already written). Mitigated by test fixture that validates ip/user/mac are present in output.	⚠️
nit	RFC 5424 grok hardcodes 'qulogd' literal instead of capturing process.name, inconsistent with RFC 3164 path Minor inconsistency; process.name is not required by the plan and the RFC 5424 format does not use the bracketed SYSLOGPROG syntax. Can be addressed in a follow-up.	⚠️
nit	ignore_missing: true on supplementary grok processors is redundant given the if: guard Harmless; keeping it for defensive clarity.	⚠️

Final validation passed: yes

Risk and classification

• Plan risk level: medium
• Tags: pipeline, processors, ecs, ingest, test-fixture, field-mapping, docs
• Impact: medium

Links

• Issue: (no issue number)
• Issue title: qnap_nas.log [MISSING_CASE]: Processor 'grok' with tag 'grok_event_original_cad2ef7a' in pipeline 'lo…
• Pipeline case: b57b0e02f567c44d

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: b67e7d5