Empty a Kafka topic

Empty a topic in Kafka

Kafka is a distributed streaming platform designed to be a highly scalable. It’s publish-and-subscribe model is very easy to follow and is endorsed by top companies and experts in the field.

There are some cases when you might need to delete all data in Kafka topic. There is no straightforward way mentioned in the documentation to achieve this. However, we can leverage the topic config to do that.

Kafka keeps data for a configurable amount of time. The Kafka config responsible for this is “rentention.ms”. This can be set on the topic level. The idea is to set the retention time to 1 millisecond and wait a while for the cleanup process to execute. Once that completes, all Kafka messages are discarded and you have an empty topic to write to. We have to make sure to switch back the config to previous value before using it.

Command to modify retention time

kafka-topics.sh --zookeeper <zk_url> --topic <topic_name> --alter --config retention.ms=1

*Note that this works for only non-campacted topics. You might have to alter the ‘cleanup_policy’ to ‘delete’ to remove compaction and add back again. If the compaction is enabled, Kafka keeps the data for all unique keys overwriting previous message with the same key