Kafka command cheatsheet

Startup

Windows

bin\windows\zookeeper-server-start config\zookeeper.properties
bin\windows\kafka-server-start config\server.properties

Linux

nohup ./zookeeper-server-start.sh ../config/zookeeper.properties 2> zookeeper_err.log 1> /dev/null &
nohup ./kafka-server-start.sh ../config/server.properties 2> kafka_err.log 1> /dev/null &

List topics

./kafka-topics.sh --list --zookeeper localhost:2181
./kafka-topics.sh --list --bootstrap-server localhost:9092

Create Topic

./kafka-topics.sh --create --topic topic-name --bootstrap-server localhost:9092

## create topic of 2 partitions
./kafka-topics.sh --create --topic topic-name --bootstrap-server localhost:9092 --partitions 2

Produce to Topic

./kafka-console-producer.sh --topic topic-name --bootstrap-server localhost:9092

## produce message with format of <key>:<message>
./kafka-console-producer.sh --topic topic-name --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Consume Topic

## consume from beginning
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --from-beginning 
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --from-beginning --max-messages 1

## with consumer group ID and sticky rebalancing strategy
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --group yourGroupName --consumer-property partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

## consume a particular partition
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --partition 0 

## read 10 messages from beginning with displying of key, offset, timestamp
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --from-beginning  --property print.key=true --property key.separator=":" --property print.offset=true --property print.timestamp=true --max-messages 10

## read from offset 18153855
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --offset 18153855 --property print.key=true --property key.separator="|" --partition 0 --max-messages 10

multiple approaches of consume message (related to commit): https://dzone.com/articles/kafka-internals-consumer
for rebalancing: https://kafka.apache.org/documentation.html#consumerconfigs_partition.assignment.strategy
for custom rebalancing : https://medium.com/streamthoughts/understanding-kafka-partition-assignment-strategies-and-how-to-write-your-own-custom-assignor-ebeda1fc06f3
Kafka further reading: https://dev.to/confluentinc/5-things-every-apache-kafka-developer-should-know-4nb

Delete Topic

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic sample-csv-02

Reset Applcation

> bin/kafka-streams-application-reset.sh --bootstrap-servers localhost:9092 --application-id your-app-id --input-topics topic-name --to-earliest

Set Retension to 24 hours:

./kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic-name --config retention.ms=86400000 --config cleanup.policy=compact,delete

Kafka command cheatsheet

Startup

Windows

Linux

List topics

Create Topic

Produce to Topic

Consume Topic

Delete Topic

Reset Applcation

Set Retension to 24 hours:

References & Further Reading: