Skip to content

feat(kafka): Support multiple kafka clients in component - #17757

Merged
periklis merged 5 commits into
mainfrom
kafka-multi-client-config
May 20, 2025
Merged

feat(kafka): Support multiple kafka clients in component#17757
periklis merged 5 commits into
mainfrom
kafka-multi-client-config

Conversation

@periklis

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
This pull request enables Loki components to use separate reader and writer clients in parallel. The separation of reader/writer address and client ID is required for kafka-compatible implementations that provide different endpoints for writes and reads.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR
@periklis periklis self-assigned this May 20, 2025
@periklis
periklis requested a review from a team as a code owner May 20, 2025 08:37
Comment thread pkg/kafka/config.go Outdated
}

func (cfg *ClientConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
f.StringVar(&cfg.Address, prefix+".address", "localhost:9092", "The Kafka backend address.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember to fix duplicate defaults.

Comment thread pkg/kafka/config.go
Comment thread pkg/kafka/client/reader_client.go Outdated

@grobinson-grafana grobinson-grafana left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Please see comments but otherwise I don't need to see this again, thanks!

As discussed on Slack, please make sure to update this configuration in all of our environments and then come back and remove the deprecated fields. May I suggest creating an issue for this so you don't forget about it?

@periklis
periklis merged commit 6ef715f into main May 20, 2025
@periklis
periklis deleted the kafka-multi-client-config branch May 20, 2025 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants