By default, the client does not linger. As soon as a message is produced, the client kicks off a goroutine that builds batches as fast as possible and sends them to the broker. If you produce at a high enough rate, you'll eventually hit network back-pressure, at which point larger batches will be buffered while the client waits to send more produce requests.
I defaulted to no linger based on a comment I read from Jay Kreps on a Confluence page ~two links deep from a KIP when I started implementing the library in 2019. The gist of the comment was effectively what I wrote above: No linger means you'll create small batches at the start, but at high enough throughput, back-pressure forces larger batches. Eventually the client reaches a steady state at effectively the best throughput it can have.
I can't find that comment anymore. I've tried a few times over the years, at this point I almost wonder if I'm making it up.
#1064 showed that introducing a large linger when producing messages at a _very_high volume actually increases throughput significantly. That is: the original theory is bunk; lingering does help throughput.
I'm not the biggest fan of no linger anymore: it's always bothered me that the first few batches you produce in a high-volume client are always single-message sized, since the concurrent goroutine that cuts batches starts immediately after the first Produce and is fast enough that it cuts a batch before your second Produce. This, in tandem with seeing for years how a little bit of linger helps librdkafka / helps throughput for high volume users has me thinking to change the default linger to either 10ms, 50ms, or 100ms.
I could do this poll in a discussion, but those have less visibility in this project since they've never been used. Given that:
- React 👎 if you prefer to keep the current default of 0ms
- React 🎉 if you're open to the idea and don't care about the value
- React ❤️ if you prefer 10ms
- React 🚀 if you prefer 50ms
- React 👀 if you prefer 100ms
I'm pinning this issue and will do something in a few months (three or more).
By default, the client does not linger. As soon as a message is produced, the client kicks off a goroutine that builds batches as fast as possible and sends them to the broker. If you produce at a high enough rate, you'll eventually hit network back-pressure, at which point larger batches will be buffered while the client waits to send more produce requests.
I defaulted to no linger based on a comment I read from Jay Kreps on a Confluence page ~two links deep from a KIP when I started implementing the library in 2019. The gist of the comment was effectively what I wrote above: No linger means you'll create small batches at the start, but at high enough throughput, back-pressure forces larger batches. Eventually the client reaches a steady state at effectively the best throughput it can have.
I can't find that comment anymore. I've tried a few times over the years, at this point I almost wonder if I'm making it up.
#1064 showed that introducing a large linger when producing messages at a _very_high volume actually increases throughput significantly. That is: the original theory is bunk; lingering does help throughput.
I'm not the biggest fan of no linger anymore: it's always bothered me that the first few batches you produce in a high-volume client are always single-message sized, since the concurrent goroutine that cuts batches starts immediately after the first
Produceand is fast enough that it cuts a batch before your secondProduce. This, in tandem with seeing for years how a little bit of linger helps librdkafka / helps throughput for high volume users has me thinking to change the default linger to either 10ms, 50ms, or 100ms.I could do this poll in a discussion, but those have less visibility in this project since they've never been used. Given that:
I'm pinning this issue and will do something in a few months (three or more).