Kafka: Handling poison pills

Kafka is a distributed event log and messages come and get processed in the order they get added. A poison pill is a message that our application is unable to process. This can lead to our consumer failing and if more messages are getting added, lead to the lag increasing which in turn means the amazing throughput we expect is now affected.

There are cases where a corrupt message stopping the consumption of more messages may be a good thing, like if the order of the messages is important to the consumer. In that case we can tolerate a disruption.