After message compression, we want to control that batch. By default, Kafka wants to minimize latency. And so it’ll try to send the records as soon as possible. It will have up to, five requests in flight, which means that up to five messages individually can be sent at the same time. Then, once there are more messages to send, Kafka will start batching them together while they wait, so that when one acknowledgment comes back, Kafka can send the whole batch at once, and minimize latency overall.
Now smart batching is basically what allows the Kafka producer to have super low latency and high throughputs. And that comes out of the box and we don’t have to do anything, no settings to be changed, if you throw a lot of .send it to Kafka, it won’t do 1,000 requests, it will batch them at some point, automatically. Basically, they have a higher compression ratio, so better efficiency. And also, when a producer sends a batch to Kafka, it’s just one request. So, there is no overhead. Now how can we control that batching mechanism? Because it is something we definitely want. So we have two settings for this, It’s linger.ms and batch.size.
Prerequisite: Apache Kafka – Message Compression
linger.ms in Apache Kafka
linger.ms, is the number of milliseconds a producer is willing to wait before sending a batch out. And by default, it’s zero. That means that the producer should send data to Kafka right away. Now if you introduce a little bit of lag, for example, it’s okay if we don’t get the data right away, we can wait maybe five or ten milliseconds more. So you set linger.ms=5. So, basically, we increase the chances of messages being sent together in a batch. And at the expense of this small delay, we can increase the throughput, compression, and efficiency of the producer. So overall, adding a small delay may actually increase the efficiency. Now, if your batch is full, which is the batch.size setting, before the end of the linger.ms period, then we’ll be sent to Kafka right away. Now, look at the below diagram.
Here, the producer produces messages to Kafka. And the first message is produced with .send and then we have a producer record. And the producer will say it will wait up to linger.ms; so maybe five or ten milliseconds, to see if the code is going to more .send. So maybe while the producer waits to send the batch, there is 10 more producer.send that are being created. So we have 10 messages because we have waited up to linger.ms. Now, after 10 milliseconds, the producer is going to batch them together and that’s going to be one request. Now, you have to make sure that the batch size, you know, isn’t going to be too small. But if it’s a big enough batch size, then all these messages will go along together in one request. Now, the producer will send this whole batch at once, to Kafka, and obviously, that batch will be compressed if you enable compression. And so what we’ve done here is by introducing a small delay, we’ve basically enabled Kafka to get fewer requests and more throughputs. Now, what about the batch.size?
batch.size in Apache Kafka
batch.size is the maximum number of bytes that will be included in one batch. By default, it is 16 KB. But you can increase that batch size to something like 32 KB or 64 KB because that helps to have a bigger batch, so that helps the compression, the throughput, and that places fewer requests on to Kafka. And so, overall, this is a good setting. Now you should know that any message that is bigger than the batch.size, will not be batched. So if you have a message that’s 100KB, then it won’t be batched. The batch.size, is allocated per partition. So when you set the batch size to a very high number, then you may just completely outrun your producer memory or waste memory, so don’t set it to too high a number. The defaults are fine, if you double it to 32KB, or quadruple it to 64KB, that’s good, but don’t set it to a super high number.
