feat(outputs.arc): Add plugin by xe-nvdk · Pull Request #17875 · influxdata/telegraf

xe-nvdk · 2025-10-22T17:45:27Z

Summary

Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol.

Features

MessagePack binary protocol (3-5x faster than line protocol)
Columnar data format for direct Arrow/Parquet writes
Multi-database support for environment/tenant isolation
Gzip compression (reduces bandwidth by ~10x)
Batch processing with configurable sizes
Comprehensive error handling and logging

Testing

All tests passing (9/9):

Connection handling
Write operations (single/batch)
API key authentication
Error scenarios
MessagePack encoding
Configuration validation

Documentation

Comprehensive README with examples
Configuration guide for basic/high-performance/multi-database setups
Troubleshooting section
Performance tuning guide

Checklist

No AI generated code was used in this PR

sync with main

… writes Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol. - MessagePack binary protocol (3-5x faster than line protocol) - Columnar data format for direct Arrow/Parquet writes - Multi-database support for environment/tenant isolation - Gzip compression (reduces bandwidth by ~10x) - Batch processing with configurable sizes - Comprehensive error handling and logging - Columnar format: 2.66x faster than row format - Throughput: 2.42M records/sec - Optimized for high-volume IoT and observability workloads All tests passing (9/9): - Connection handling - Write operations (single/batch) - API key authentication - Error scenarios - MessagePack encoding - Configuration validation - Comprehensive README with examples - Configuration guide for basic/high-performance/multi-database setups - Troubleshooting section - Performance tuning guide

telegraf-tiger · 2025-10-22T17:45:34Z

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

xe-nvdk · 2025-10-22T17:47:55Z

Hey team. I hope that everybody is doing good over there. Happy to contribute with an output plugin. Open to make modifications in the case that I missed something. By the way: !signed-cla

Have a great days, folks.
Ignacio

xe-nvdk · 2025-10-22T17:48:24Z

!signed-cla

Added github.com/vmihailenco/tagparser with BSD 2-Clause License Entries added in alphabetical order between vjeantet/grok and vmware/govmomi Verification ✅ make check-deps now passes successfully across all 18 target platforms (darwin, freebsd, linux, windows on various architectures) All Telegraf CI checks should now pass: ✅ Markdown linting (README.md) ✅ Go code linting (arc.go, arc_test.go) ✅ Dependency license documentation (LICENSE_OF_DEPENDENCIES.md) Modified files ready for commit: plugins/outputs/arc/README.md - All markdown linting issues fixed docs/LICENSE_OF_DEPENDENCIES.md - Added msgpack and tagparser licenses

…oid shadowing the imported config package

Hipska · 2025-10-24T13:49:03Z

Maybe I'm missing something, but can't you already use outputs.http with data_format = "msgpack"?

xe-nvdk · 2025-10-24T14:07:00Z

Maybe I'm missing something, but can't you already use outputs.http with data_format = "msgpack"?

Yes, it can be used, but the best performance comes from our columnar format. The implementation you're referring to sends row-based data, which introduces conversion overhead when we transform rows to columns.

The output send directly the data in columnar format.

Let me know if you have additional questions.

Thank you
Ignacio

srebhan

Thanks for your contribution @xe-nvdk! I do have some comments in the code...

xe-nvdk · 2025-10-27T12:36:31Z

Awesome stuff! thank you for taking the time, we are going to review all. Again, thank you!

@srebhan

Apply code review comments from @srebhan to improve code quality and fix bugs: - Make arcColumnarData struct private (only used internally) - Preallocate columns map based on first metric's field/tag count for performance - Remove redundant inline comments throughout codebase - Add default timeout of 5s in Init() to fix context deadline exceeded errors - Correct API endpoint to /api/v1/write/msgpack (verified against OpenAPI spec) Bug fix: Previously, when timeout was not configured, a.Timeout was 0, causing context.WithTimeout() to create an immediately-expired context. Now defaults to 5s. Tests: All 10 tests passing Verified: Successfully writing metrics to production Arc instance

Remove vmihailenco/msgpack and vmihailenco/tagparser from LICENSE_OF_DEPENDENCIES.md as they are no longer used after migrating Arc output plugin to tinylib/msgp. The Arc plugin now uses tinylib/msgp (already in Telegraf dependencies) instead of vmihailenco/msgpack, so these license entries are no longer needed. Fixes: make check-deps

…est file

srebhan · 2025-11-06T09:45:47Z

@xe-nvdk maybe it was an oversight but I missed the "AI" checkbox but I can see it now. Maybe I need better glasses. ;-)

SORRY for the confusion!

srebhan

Thanks for the update @xe-nvdk! Two tiny last things...

srebhan · 2025-11-06T09:50:12Z

+	payload := buf.Bytes()
+
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(a.Timeout))
+	defer cancel()
+
+	req, err := http.NewRequestWithContext(ctx, "POST", a.URL, bytes.NewReader(payload))


How about

Suggested change

payload := buf.Bytes()

ctx, cancel := context.WithTimeout(context.Background(), time.Duration(a.Timeout))

defer cancel()

req, err := http.NewRequestWithContext(ctx, "POST", a.URL, bytes.NewReader(payload))

ctx, cancel := context.WithTimeout(context.Background(), time.Duration(a.Timeout))

defer cancel()

req, err := http.NewRequestWithContext(ctx, "POST", a.URL, &buf)

as the buffer can server as a reader... This avoids constructing another instance... Maybe you want to rename buf to payload but not a must.

srebhan · 2025-11-06T09:52:57Z

+  ## Arc MessagePack API URL
+  url = "http://localhost:8000/api/v1/write/msgpack"


As you are setting this as a default in Init you should comment the url here to denote this setting as optional. Otherwise, keep it uncommented here and error-out on Init if the URL is empty...

xe-nvdk · 2025-11-06T11:55:52Z

@xe-nvdk maybe it was an oversight but I missed the "AI" checkbox but I can see it now. Maybe I need better glasses. ;-)

SORRY for the confusion!

About the checkbox, I rembered having that the first time, but I dont remember taking that out at some point, so I modified that yesterday to include it! Anyways. Thank you. Let me address the last comments.

srebhan · 2025-11-12T19:11:29Z

@xe-nvdk any chance you address the last two comments from my side? I would love to integrate this into v1.37...

xe-nvdk · 2025-11-12T19:33:20Z

Yes. I was expecting to do this today. Let me accept your suggestion and regarding the init, in the config should be uncommented and in the init declare the error.

Address PR feedback: keep URL uncommented in sample.conf and validate as required in Init(). Users will now get clear error "url is required" if URL is not configured. Updated TestInit to verify both success case (with URL) and failure case (without URL).

xe-nvdk · 2025-11-12T19:49:38Z

@xe-nvdk any chance you address the last two comments from my side? I would love to integrate this into v1.37...

Ok, we should be good now. Let me know if you need something else. Thank you, this couldn't happened with you continuos support.

srebhan

@xe-nvdk only one last suggestion on directly using the buffer (buf) as payload instead of copying the content and creating a new reader...

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

srebhan

@xe-nvdk I proposed a fix for the linter issue. Please check if that fixes the issue by running make lint!

One thing for the future: If there is a docker-based image of the database I would appreciate an integration-test at some point to make sure the plugin works with the actual thing...

xe-nvdk · 2025-11-14T10:25:10Z

@xe-nvdk I proposed a fix for the linter issue. Please check if that fixes the issue by running make lint!

One thing for the future: If there is a docker-based image of the database I would appreciate an integration-test at some point to make sure the plugin works with the actual thing...

Gotcha!

I can work on that today if that makes sense, I have been tested on my side and works pretty well, in fact, Im using it these builds right now and works like a charm!

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

telegraf-tiger · 2025-11-14T11:53:36Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB	. RPM	. TAR . GZ	. ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

srebhan

Thanks a lot @xe-nvdk!

skartikey

Excellent work @xe-nvdk!

xe-nvdk · 2025-11-17T13:21:13Z

Thank you folks. As you saw, I'm not the strongest on Go, and for that reason I appreciate the support across this PR.

Thank you

xe-nvdk and others added 9 commits October 6, 2025 18:42

Merge pull request #1 from influxdata/master

fdfa547

sync with main

Merge branch 'influxdata:master' into master

ebf9e89

Merge branch 'influxdata:master' into master

c0193b2

Merge branch 'influxdata:master' into master

7de540f

Merge branch 'influxdata:master' into master

72c1324

Merge branch 'influxdata:master' into master

b296d8a

Merge branch 'influxdata:master' into master

5befa6a

Merge branch 'master' of github.com:Basekick-Labs/telegraf

f55d597

xe-nvdk changed the title ~~Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol.~~ Oct 22, 2025

telegraf-tiger Bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Oct 22, 2025

Ignacio Van Droogenbroeck added 5 commits October 22, 2025 14:54

small fixes on readme for arc

03ebdd7

go mod tidy Issue

949adaf

fixing errors with fmt reported by CI/CD checks

6b4aa8f

revive error in arc_test.go:394: Renamed config variable to cfg to av…

7ea1614

…oid shadowing the imported config package

srebhan reviewed Oct 27, 2025

View reviewed changes

srebhan changed the title ~~feat(outputs): add arc output plugin for high-performance time-series writes~~ Oct 27, 2025

telegraf-tiger Bot added the plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins label Oct 27, 2025

srebhan added the new plugin label Oct 27, 2025

srebhan self-assigned this Oct 27, 2025

Ignacio Van Droogenbroeck added 3 commits October 27, 2025 10:40

fixing the lint errors in telegraf repo

e21b755

fixing the ci test generated with the removal of the http server in t…

179e808

…est file

srebhan reviewed Nov 6, 2025

View reviewed changes

srebhan added the waiting for response waiting for response from contributor label Nov 12, 2025

telegraf-tiger Bot removed the waiting for response waiting for response from contributor label Nov 12, 2025

xe-nvdk and others added 3 commits November 12, 2025 16:37

Merge branch 'influxdata:master' into master

e452e87

Merge branch 'master' of github.com:Basekick-Labs/telegraf

071464c

Hipska approved these changes Nov 13, 2025

View reviewed changes

srebhan reviewed Nov 13, 2025

View reviewed changes

Comment thread plugins/outputs/arc/arc.go Outdated

xe-nvdk and others added 2 commits November 13, 2025 11:26

Update plugins/outputs/arc/arc.go

31eceeb

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

Merge branch 'influxdata:master' into master

4b2e937

srebhan reviewed Nov 14, 2025

View reviewed changes

Comment thread plugins/outputs/arc/arc.go

Update plugins/outputs/arc/arc.go

444dbd7

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

srebhan approved these changes Nov 17, 2025

View reviewed changes

srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Nov 17, 2025

srebhan assigned skartikey and mstrandboge and unassigned srebhan Nov 17, 2025

skartikey unassigned mstrandboge Nov 17, 2025

skartikey approved these changes Nov 17, 2025

View reviewed changes

skartikey merged commit aa3681d into influxdata:master Nov 17, 2025
27 checks passed

github-actions Bot added this to the v1.37.0 milestone Nov 17, 2025

		## Arc MessagePack API URL
		url = "http://localhost:8000/api/v1/write/msgpack"

Uh oh!

Conversation

xe-nvdk commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Features

Testing

Documentation

Checklist

telegraf-tiger Bot commented Oct 22, 2025

xe-nvdk commented Oct 22, 2025

xe-nvdk commented Oct 22, 2025

Hipska commented Oct 24, 2025

xe-nvdk commented Oct 24, 2025

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xe-nvdk commented Oct 27, 2025

srebhan commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

srebhan Nov 6, 2025

Choose a reason for hiding this comment

srebhan Nov 6, 2025

Choose a reason for hiding this comment

xe-nvdk commented Nov 6, 2025

srebhan commented Nov 12, 2025

xe-nvdk commented Nov 12, 2025

xe-nvdk commented Nov 12, 2025

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

xe-nvdk commented Nov 14, 2025

telegraf-tiger Bot commented Nov 14, 2025

Artifact URLs

srebhan left a comment

Choose a reason for hiding this comment

skartikey left a comment

Choose a reason for hiding this comment

Uh oh!

xe-nvdk commented Nov 17, 2025

Labels

5 participants

xe-nvdk commented Oct 22, 2025 •

edited

Loading

srebhan commented Nov 6, 2025 •

edited

Loading