-
Notifications
You must be signed in to change notification settings - Fork 25.8k
Closed
Labels
:Distributed/Ingest NodeExecution or management of Ingest PipelinesExecution or management of Ingest Pipelines>enhancementTeam:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.
Description
Description
It's common for the message in a log event to actually be a JSON itself. Specifically, there are shippers that produce ECS-JSON.
We would like to handle this automatically through the normalize_for_stream ingest processor. The idea is to add an additional step, so that it now does the following:
- If it's OTel data: just use as is
- If it's not OTel:
- apply a cheap check whether
messageis a JSON-encoded string (e.g. if it starts with{and ends with}) - If yes:
- Parse the
messageas JSON* - Apply a cheap check whether the resulting object is ECS (e.g. if contains a
@timestampkey) - If it's ECS: merge the resulting object back into the root of the document*
- If not: add the resulting object as is as the value of the
body.structuredfield
- Parse the
- Proceed with namespacing/normalization as before
- apply a cheap check whether
* JSON parsing and merge should behave exactly exactly as defined in logs@json-pipeline
@dakrone is it a problem to have dependencies between one processor and another? More specifically, is it possible for the NormalizeForStreamProcessor to use JsonProcessor#apply?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
:Distributed/Ingest NodeExecution or management of Ingest PipelinesExecution or management of Ingest Pipelines>enhancementTeam:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.