Kafka: Handling poison pills

Kafka is a distributed event log and messages come and get processed in the order they get added. A poison pill is a message that our application is unable to process. This can lead to our consumer failing and if more messages are getting added, lead to the lag increasing which in turn means the amazing throughput we expect is now affected.

There are cases where a corrupt message stopping the consumption of more messages may be a good thing, like if the order of the messages is important to the consumer. In that case we can tolerate a disruption.

On the other hand, if ordering does not matter, then there’s no reason for one bad message that our app can’t process to hold up other messages that we can process, which is likely why you chose Kafka the first place.

n this article we’ll go over how we can achieve this by making use of retry and dead letter topics to prevent congestions. We’ll make use of spring-Kafka as it provides support for such features.

Use case

Say we have an app that receives messages in topic: customer-ratings. We have two services that consume from this one topic. One that updates the rating for the given product and another that updates the customer loyalty where we keep track of their reviews so we reward them.

@Bean
 public NewTopic topic() {
  return TopicBuilder.name("customer-ratings")
    .partitions(10)
    .replicas(1)
    .build();
 }

Hands on example