[AWSFirehose] New ingest pipeline for firehose

[gizas]

• Enhancement

Proposed commit message

• WHAT: Updates the Firehose ingest pipeline to be less cpu intensive
• WHY: the rationale/motivation for the changes was the specific clusotmer enviroment that showcased high CPU when trying to load high throughput. With the script we dont force all checks as thec current pipeline. The if/else logic will stop consecutive checks when it finds the match.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Follow instructions https://www.elastic.co/docs/extend/integrations/pipeline-testing

cd integrations/packages/awsfirehose

elastic-package test pipeline --data-streams logs

The final results:

elastic-package test pipeline --data-streams logs
2025/08/05 14:57:25  INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline tests for the package
--- Test results for package: awsfirehose - START ---
╭─────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME                                                 │ RESULT │ TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────┼────────┼──────────────┤
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-apigateway-log.json)       │ PASS   │ 408.229584ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-cloudfront-log.json)       │ PASS   │   304.9685ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-cloudtrail-log.json)       │ PASS   │ 306.929042ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-elb-log.json)              │ PASS   │ 272.348334ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-firewall-log.json)         │ PASS   │ 255.452083ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-route53-public-log.json)   │ PASS   │ 269.798834ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-route53-resolver-log.json) │ PASS   │ 234.283209ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-s3access-log.json)         │ PASS   │ 271.973584ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-vpcflow-log.json)          │ PASS   │ 259.089917ms │
│ awsfirehose │ logs        │ pipeline  │ (ingest pipeline warnings test-waf-log.json)              │ PASS   │ 258.731458ms │
│ awsfirehose │ logs        │ pipeline  │ test-apigateway-log.json                                  │ PASS   │  46.023167ms │
│ awsfirehose │ logs        │ pipeline  │ test-cloudfront-log.json                                  │ PASS   │  39.086583ms │
│ awsfirehose │ logs        │ pipeline  │ test-cloudtrail-log.json                                  │ PASS   │  38.824416ms │
│ awsfirehose │ logs        │ pipeline  │ test-elb-log.json                                         │ PASS   │  38.959458ms │
│ awsfirehose │ logs        │ pipeline  │ test-firewall-log.json                                    │ PASS   │  37.704916ms │
│ awsfirehose │ logs        │ pipeline  │ test-route53-public-log.json                              │ PASS   │   36.51825ms │
│ awsfirehose │ logs        │ pipeline  │ test-route53-resolver-log.json                            │ PASS   │  36.912417ms │
│ awsfirehose │ logs        │ pipeline  │ test-s3access-log.json                                    │ PASS   │  37.841708ms │
│ awsfirehose │ logs        │ pipeline  │ test-vpcflow-log.json                                     │ PASS   │     37.507ms │
│ awsfirehose │ logs        │ pipeline  │ test-waf-log.json                                         │ PASS   │  40.844709ms │
╰─────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: awsfirehose - END   ---
Done

[gizas]

Some tests to verify improvement:

For each pipeline I recreated the environment with:

elastic-package stack down
elastic-package stack up -d --services=elasticsearch
$(elastic-package stack shellinit)
elastic-package benchmark pipeline --data-streams logs (repeated the command 4 times)

With current pipeline from main

❯ elastic-package benchmark pipeline --data-streams logs 2025/08/05 15:04:14 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0 Run pipeline benchmarks for the package --- Benchmark results for package: awsfirehose - START --- ╭─────────────────────────╮ │ parameters │ ├──────────────────┬──────┤ │ source_doc_count │ 16 │ │ doc_count │ 1000 │ ╰──────────────────┴──────╯ ╭───────────────────────────╮ │ pipeline_performance │ ├─────────────────┬─────────┤ │ processing_time │ 0.22s │ │ eps │ 4608.29 │ ╰─────────────────┴─────────╯ ╭───────────────────────────────────╮ │ procs_by_total_time │ ├───────────────────────────┬───────┤ │ reroute @ default.yml:175 │ 0.46% │ │ reroute @ default.yml:167 │ 0.46% │ │ reroute @ default.yml:151 │ 0.46% │ │ reroute @ default.yml:143 │ 0.46% │ │ set @ default.yml:7 │ 0.46% │ ╰───────────────────────────┴───────╯ ╭─────────────────────────────────────╮ │ procs_by_avg_time_per_doc │ ├───────────────────────────┬─────────┤ │ reroute @ default.yml:175 │ 8.064µs │ │ reroute @ default.yml:143 │ 8.064µs │ │ reroute @ default.yml:167 │ 5.291µs │ │ reroute @ default.yml:151 │ 5.291µs │ │ set @ default.yml:7 │ 1µs │ ╰───────────────────────────┴─────────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:04:52 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.07s │
│ eps │ 15151.52 │
╰─────────────────┴──────────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:05:02 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.08s │
│ eps │ 12500.00 │
╰─────────────────┴──────────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:05:11 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.06s │
│ eps │ 15625.00 │
╰─────────────────┴──────────╯

With pipeline from this pr

❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:12:55 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭───────────────────────────╮
│ pipeline_performance │
├─────────────────┬─────────┤
│ processing_time │ 0.14s │
│ eps │ 7407.41 │
╰─────────────────┴─────────╯
╭───────────────────────────────────╮
│ procs_by_total_time │
├──────────────────────────┬────────┤
│ script @ default.yml:10 │ 79.26% │
│ reroute @ default.yml:78 │ 1.48% │
│ reroute @ default.yml:62 │ 1.48% │
│ set @ default.yml:7 │ 1.48% │
│ reroute @ default.yml:86 │ 0.74% │
│ reroute @ default.yml:54 │ 0.74% │
╰──────────────────────────┴────────╯
╭─────────────────────────────────────╮
│ procs_by_avg_time_per_doc │
├──────────────────────────┬──────────┤
│ script @ default.yml:10 │ 107µs │
│ reroute @ default.yml:78 │ 10.582µs │
│ reroute @ default.yml:62 │ 10.582µs │
│ reroute @ default.yml:86 │ 8.064µs │
│ reroute @ default.yml:54 │ 8.064µs │
│ set @ default.yml:7 │ 2µs │
╰──────────────────────────┴──────────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:13:04 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.04s │
│ eps │ 24390.24 │
╰─────────────────┴──────────╯
╭───────────────────────────────────╮
│ procs_by_total_time │
├──────────────────────────┬────────┤
│ script @ default.yml:10 │ 73.17% │
│ reroute @ default.yml:78 │ 2.44% │
│ set @ default.yml:7 │ 2.44% │
╰──────────────────────────┴────────╯
╭────────────────────────────────────╮
│ procs_by_avg_time_per_doc │
├──────────────────────────┬─────────┤
│ script @ default.yml:10 │ 30µs │
│ reroute @ default.yml:78 │ 5.291µs │
│ set @ default.yml:7 │ 1µs │
╰──────────────────────────┴─────────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:13:07 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.04s │
│ eps │ 26315.79 │
╰─────────────────┴──────────╯
╭──────────────────────────────────╮
│ procs_by_total_time │
├─────────────────────────┬────────┤
│ script @ default.yml:10 │ 78.95% │
╰─────────────────────────┴────────╯
╭────────────────────────────────╮
│ procs_by_avg_time_per_doc │
├─────────────────────────┬──────┤
│ script @ default.yml:10 │ 30µs │
╰─────────────────────────┴──────╯

--- Benchmark results for package: awsfirehose - END ---
Done
❯ elastic-package benchmark pipeline --data-streams logs
2025/08/05 15:13:11 INFO New version is available - v0.113.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.113.0
Run pipeline benchmarks for the package
--- Benchmark results for package: awsfirehose - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭────────────────────────────╮
│ pipeline_performance │
├─────────────────┬──────────┤
│ processing_time │ 0.03s │
│ eps │ 29411.76 │
╰─────────────────┴──────────╯
╭──────────────────────────────────╮
│ procs_by_total_time │
├─────────────────────────┬────────┤
│ script @ default.yml:10 │ 79.41% │
╰─────────────────────────┴────────╯
╭────────────────────────────────╮
│ procs_by_avg_time_per_doc │
├─────────────────────────┬──────┤
│ script @ default.yml:10 │ 27µs │
╰─────────────────────────┴──────╯

--- Benchmark results for package: awsfirehose - END ---
Done

The new pipeline with script has more eps in general but executes quicker (by 2ms small diff with the amount of docs we have). I am still trying to digest the results

[gizas]

@jsoriano do you have any ideas/insights for the above numbers?
More eps (execution per second is better from what I understand?)

Basically I would like to measure the CPU consumption of the pipeline, do we have any other way to do so?

[jsoriano]

do you have any ideas/insights for the above numbers?
More eps (execution per second is better from what I understand?)

Not sure if I understand the question. eps are events per second, so faster executions (lower processing times) are having more eps, this would be expected.

According to this data it looks like the new pipeline is faster.

It may be surprising to find an script that is faster than its equivalent pipeline, but I guess this is caused by the many complex conditions the original pipeline has.

Basically I would like to measure the CPU consumption of the pipeline, do we have any other way to do so?

You could try to run other benchmarks, for example I think that rally benchmarks give more information. But for the pipeline itself the current data seems good enough.

[gizas]

Thank you this is what I wanted to confirm: translating the increase of eps positively is a good sign or not or I was biased with the fix :)

The script is actually faster because the if/else logic will stop consecutive checks when it finds the match.

For rally I would review with the team and let you know

[Kavindu-Dodan]

LGTM. And I agree with this remark https://github.com/elastic/integrations/pull/14691/files#r2254677562

[gizas]

@Kavindu-Dodan , @kaiyan-sheng one more review please as I added the doc update in order also to close #14676

I think it is overkill to open another pr for the above

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f322484

History

• 💚 Build #29608 succeeded 0786566
• 💔 Build #29604 failed b607aa3
• 💔 Build #29591 failed 290b781
• 💔 Build #29079 failed 1ee05ad
• 💔 Build #29076 failed e31cb0d
• 💔 Build #29073 failed f42ba7a

cc @gizas

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[Kavindu-Dodan]

LGTM

[kaiyan-sheng]

Looks good! Thank you!

[kaiyan-sheng]

Second round for the enhancement: #14914