Skip to content

processor_labels: add support for record accessor in values - #9841

Merged
edsiper merged 3 commits into
masterfrom
labels-record-accessor
Mar 28, 2025
Merged

processor_labels: add support for record accessor in values#9841
edsiper merged 3 commits into
masterfrom
labels-record-accessor

Conversation

@edsiper

@edsiper edsiper commented Jan 16, 2025

Copy link
Copy Markdown
Member

Label processors allow to manipulate the labels from metrics, this PR extends the actions insert, update and upsert to support a record accessor pattern that helps to use the $TAG variable that can be populated dinamically, e.g:

service:
  flush: 1

pipeline:
  inputs:
    - name: event_type
      type: metrics
      tag: my_test

      processors:
        metrics:
          - name: labels
            upsert: tag $TAG

  outputs:
    - name: stdout
      match: '*'

Fluent Bit output:

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/01/15 22:16:42] [ info] [fluent bit] version=4.0.0, commit=27734ee3e6, pid=1586419
[2025/01/15 22:16:42] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/15 22:16:42] [ info] [simd    ] disabled
[2025/01/15 22:16:42] [ info] [cmetrics] version=0.9.9
[2025/01/15 22:16:42] [ info] [ctraces ] version=0.5.7
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] initializing
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] storage_strategy='memory' (memory only)
[2025/01/15 22:16:42] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/15 22:16:42] [ info] [input:event_type:event_type.0] thread instance initialized
[2025/01/15 22:16:42] [ info] [sp] stream processor started
[2025/01/15 22:16:43] [ info] [input:event_type:event_type.0] [OK] collector_time
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test"} = 3
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test",hostname="localhost",app="cmetrics"} = 1
2025-01-16T04:16:43.837152867Z kubernetes_network_load_counter{tag="my_test",hostname="localhost",app="test"} = 12.15
2025-01-16T04:16:43.837152867Z kubernetes_network_load_gauge{tag="my_test"} = 1
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test"} = { quantiles = { 0.1=1.1, 0.2=2.2, 0.3=3.3, 0.4=4.4, 0.5=5.5 }, sum=51.6129, count=10 }
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test",my_label="my_label"} = { quantiles = { 0.1=0.1, 0.2=0.2, 0.3=0.3, 0.4=0.4, 0.5=0.5 }, sum=51.6129, count=5 }
2025-01-16T04:16:43.837152867Z k8s_disk_load_summary{tag="my_test",my_label="my_val"} = { quantiles = { 0.1=11.11, 0.2=0, 0.3=33.33, 0.4=44.44, 0.5=55.55 }, sum=51.6129, count=10 }
2025-01-16T04:16:43.837152867Z k8s_network_load_histogram{tag="my_test"} = { buckets = { 0.05=2, 5=3, 10=4, +Inf=0 }, sum=1013.02, count=5 }
2025-01-16T04:16:43.837152867Z k8s_network_load_histogram{tag="my_test",my_label="my_val"} = { buckets = { 0.05=2, 5=3, 10=4, +Inf=0 }, sum=1013.02, count=5 }

This PR also adds a new unit test for the processor.


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
…anges

Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
}

TEST_LIST = {
#ifdef FLB_HAVE_METRICS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just my curiosity but do we need to use this if def for test cases? This is because when disabling FLB_HAVE_METRICS, the defined test cases are not effective under this condition.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm maybe we don't need it... I don't think metrics are optional anymore

@edsiper
edsiper merged commit b0f3a63 into master Mar 28, 2025
@edsiper
edsiper deleted the labels-record-accessor branch March 28, 2025 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-required ok-package-test Run PR packaging tests

3 participants