Apache Kafka Command-Line Operations: A Step-by-Step comprehensive Guide 2024
Apache Kafka is a distributed streaming platform that provides powerful command-line utilities to manage Kafka clusters, topics, producers, and consumers. These commands help users perform administrative tasks, debugging, and real-time testing.
In this guide, we will cover: ✅ Starting and stopping Kafka and ZooKeeper
✅ Creating and managing Kafka topics
✅ Producing and consuming messages
✅ Deleting Kafka topics
1. Understanding Kafka and ZooKeeper
Kafka operates as a distributed event-streaming platform that requires ZooKeeper for cluster coordination.
🔹 Kafka is used for:
- Streaming data pipelines (moving data between applications).
- Real-time event processing (logs, analytics, IoT data).
🔹 Kafka Components:
| Component | Description |
|---|---|
| ZooKeeper | Manages broker metadata and leader election. |
| Kafka Broker | Stores and serves messages to consumers. |
| Producer | Publishes messages to Kafka topics. |
| Consumer | Reads messages from Kafka topics. |
| Topic | A logical name where messages are stored. |
✅ Kafka stores messages in topics, divided into partitions, ensuring scalability and fault tolerance.
2. Starting Kafka and ZooKeeper
Kafka requires ZooKeeper to run before starting the Kafka broker.
A. Checking Kafka Installation
🔹 Open a terminal and check if Kafka is installed:
bashCopyEditpwd # Shows the current directory
ls # Lists files in the directory
You should see a Kafka installation directory.
B. Starting ZooKeeper
Navigate to the Kafka bin directory and start ZooKeeper:
bashCopyEditcd /path/to/kafka/bin
./zookeeper-server-start.sh ../config/zookeeper.properties
✅ ZooKeeper should now be running on localhost:2181.
C. Starting the Kafka Server
In another terminal, navigate to the bin directory and start Kafka:
bashCopyEditcd /path/to/kafka/bin
./kafka-server-start.sh ../config/server.properties
✅ Kafka is now running on localhost:9092.
🚨 Troubleshooting:
- Ensure ZooKeeper is running before starting Kafka.
- Check server.properties for any incorrect configurations.
3. Creating and Managing Kafka Topics
Kafka organizes messages into topics, which act as channels.
A. Checking Kafka Version
Before creating a topic, check the Kafka version:
bashCopyEdit./kafka-topics.sh --version
B. Listing Existing Topics
To list all Kafka topics:
bashCopyEdit./kafka-topics.sh --list --zookeeper localhost:2181
🚨 If you are using Kafka 2.8+ (KRaft mode), use:
bashCopyEdit./kafka-topics.sh --list --bootstrap-server localhost:9092
C. Creating a Kafka Topic
Create a topic named first_topic with 3 partitions:
bashCopyEdit./kafka-topics.sh --zookeeper localhost:2181 --create --topic first_topic --replication-factor 1 --partitions 3
🔹 Explanation:
--replication-factor 1: Only one copy of the data.--partitions 3: The topic is divided into 3 partitions.
✅ Run --list again to confirm the topic was created.
4. Producing and Consuming Messages
Now, let’s send messages to Kafka and read them in real-time.
A. Producing Messages
Kafka provides a built-in console producer to send messages:
bashCopyEdit./kafka-console-producer.sh --topic first_topic --broker-list localhost:9092
🔹 Type messages in the terminal and press Enter after each:
csharpCopyEditHello Kafka!
This is my first Kafka message.
✅ Messages are now stored in Kafka.
B. Consuming Messages
To read messages from first_topic, use the console consumer:
bashCopyEdit./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --from-beginning
✅ You should see the messages printed in real-time.
5. Deleting a Kafka Topic
To delete first_topic, use:
bashCopyEdit./kafka-topics.sh --zookeeper localhost:2181 --delete --topic first_topic
✅ Run --list again to confirm deletion.
🚨 Kafka does not immediately delete topics. Ensure delete.topic.enable=true is set in server.properties.
6. Summary of Kafka Commands
| Operation | Command |
|---|---|
| Start ZooKeeper | ./zookeeper-server-start.sh ../config/zookeeper.properties |
| Start Kafka | ./kafka-server-start.sh ../config/server.properties |
| List topics | ./kafka-topics.sh --list --zookeeper localhost:2181 |
| Create topic | ./kafka-topics.sh --create --topic first_topic --replication-factor 1 --partitions 3 --zookeeper localhost:2181 |
| Send messages | ./kafka-console-producer.sh --topic first_topic --broker-list localhost:9092 |
| Consume messages | ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --from-beginning |
| Delete topic | ./kafka-topics.sh --delete --topic first_topic --zookeeper localhost:2181 |
7. Best Practices for Kafka CLI Operations
✅ Always start ZooKeeper before Kafka.
✅ Use --list to confirm topic creation/deletion.
✅ Check logs in logs/ for troubleshooting errors.
✅ Use --from-beginning to replay messages in a topic.
✅ Monitor Kafka with kafka-topics.sh --describe --topic <topic_name>.
8. Final Thoughts
Kafka command-line operations provide a quick and effective way to manage topics, messages, and brokers. Whether you’re debugging, testing, or managing Kafka, these commands simplify administration.
💡 What Kafka operations do you use most frequently? Let us know in the comments! 🚀