Skip to content

[lambda-promtail] Add support for log line parsing/transforming/filtering feature #17986

Open
@pyo-counting

Description

@pyo-counting

Is your feature request related to a problem? Please describe.
Currently, lambda-promtail lacks the ability to parse, transform, and filter log lines within the Lambda function itself before sending them to Loki. Users who need this functionality are forced to deploy additional agents like Promtail or Grafana Alloy as intermediaries. This architecture introduces several problems.

  1. Increased operational overhead: Managing and maintaining additional EC2 instances or containerized deployments solely for log processing.
  2. Higher infrastructure costs: Running extra compute resources for intermediate agents and incurring additional network transfer costs for logs that will eventually be dropped.
  3. Complexity: Adds an extra layer of configuration and potential points of failure to the logging pipeline.

Describe the solution you'd like
I propose adding a configuration mechanism to lambda-promtail that allows users to define Promtail or Grafana Alloy-style pipeline stages directly via environment variables.

Specifically, I envision a new environment variable (e.g., PIPELINE_STAGES_CONFIG) where users can provide a JSON or YAML string representing an array of pipeline stage definitions, similar to how pipeline_stages/loki.process is configured in Promtail/Alloy.

Describe alternatives you've considered

Additional context
lambda-promtail already supports Prometheus-style relabeling, demonstrating its capability to process labels. Extending this to full pipeline stage support would be a natural and highly valuable evolution. It would empower users to build more efficient and cost-effective serverless logging pipelines to Loki directly

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions