[syslog_router] Rename message to event.original

[ilyannn]

Context

The syslog_router integration (original PR: #11727) never had the rename processor that would create event.original. This is suboptimal and also means that the fix applied in #15806 never really worked.

This PR

We add such a processor, copying it – together with a hash in the tag – from another pipeline. As required by the pipeline spec we also add a remove processor for the message field.

Note on this solution

Apparently doing a global rename of message→ event.original for logs flowing through this integration might break the workflow of people who currently use this integration and have set up a custom pipeline to process message. We have therefore several choices how to proceed:

0. Keep everything as-is: the customers can add a custom pipeline with the rename if they desire.
1. Keep the PR as-is. Especially if there are few customers who have this custom processing.
2. Copy to the event.original instead of renaming, iff there is a preserve_original_event tag. We'll need to make sure that the integrations we route to don't break because of this change. Also this might make customers unhappy by duplicating the data (but only in the case when this tag is set, so they have control over it).
3. Have a separate toggle for the rename – let the customers make the decision based on their tradeoffs.
4. Something else, e.g. a full syslog processing?

Additionally we're getting -72% on the benchmark speed with this PR. This is probably concerning.

Anyway I've set the PR back to draft mode until we decide.

Proposed commit message

[syslog_router] Rename message to event.original

Adds the standard rename of message to event.original to the the syslog router integration.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• [ ] I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• [ ] I have verified that Kibana version constraints are current according to guidelines.
• [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

cd packages/syslog_router
elastic-package test

Warning

Due to limitations in elastic-package, automated tests cannot verify routing behavior to data streams external to this package. Verification will have to be performed manually. To verify, follow the instructions in the integration on installing assets for another integration (Cisco ASA, for example), configure the integration, and send a relevant log to the Agent.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 39bdf00

History

• 💚 Build #34810 succeeded 70f435b
• 💔 Build #34809 failed aff9394
• 💔 Build #34808 failed 4396138

cc @ilyannn

[taylor-swanson]

My thoughts on this:

0. Keep everything as-is: the customers can add a custom pipeline with the rename if they desire.
1. Keep the PR as-is. Especially if there are few customers who have this custom processing.
2. Copy to the event.original instead of renaming, iff there is a preserve_original_event tag. We'll need to make sure that the integrations we route to don't break because of this change. Also this might make customers unhappy by duplicating the data (but only in the case when this tag is set, so they have control over it).

If an event lands in the syslog_router log data stream, that means it was never routed in the first place and ended up there as a last resort. The pipeline for this data stream was never meant to do any sort of processing other than the bare minimum (which may include the creation of the event.original field). For logs that do get routed, the event should contain the preserve_original_event tag and it is the responsibility of the destination integration pipeline to handle it correctly.

At least for the syslog_router log data stream, the message field should NOT be removed. I'm not entirely sure how reindexing works with routing rules, but if for example the user fixed an issue with the routing logic and reindexed events in the log data stream, it would in theory allow the routing logic to try to reroute events again. The routing logic depends entirely on the message field being present. Most integrations tolerate the presence of event.original, so this shouldn't be an issue.

3. Have a separate toggle for the rename – let the customers make the decision based on their tradeoffs.

I think based on how the syslog_router data stream works (it's a last resort catch all), I don't want to give extra functionality to it.

4. Something else, e.g. a full syslog processing?

See my first point, syslog_router should not do extra processing.

Additionally we're getting -72% on the benchmark speed with this PR. This is probably concerning.

This is not surprising, since the pipeline does so little in the first place. Any change is going to cause a "big" speed change in the benchmark. I've also noticed that benchmarks can be wildly inaccurate and it takes multiple runs to get a stable number (if it ever gets stable).

So to summarize, I could see us creating the event.original field, but NOT removing the message field.

[ilyannn]

After the consultation with the team – let's keep the pipeline as-is.