In Apache Kafka, a topic is a category or stream of messages that the Kafka message broker (or cluster) stores. Producers write data to topics and consumers read from topics. A Topic in Kafka is similar to a table in a database or a stream in a stream processing system. Each topic is divided into a number of partitions, which allows for parallel processing of the data by consumers. Messages are assigned to partitions based on a partitioning key, which can be any field in the message. Topics are used to store and transmit data in Kafka. They can be used for a wide range of applications, such as real-time data processing, event-driven architectures, and log aggregation. A topic in Kafka can be created, deleted, described, or changed using the CLI. Make sure Kafka is open before you begin. The necessary parameters must be specified in order to describe a Kafka topic and obtain partition information.
- Use the hostname and port for Kafka v2.2+, for example, localhost:9092.
- Use localhost:2181 as the Zookeeper URL and port if you’re using a previous version of Kafka.
Create a Kafka Topic
- The name of the topic, number of partitions, and replication factor are required parameters.
- Use the “–create” option in the CLI kafka-topics.sh.
To create a Kafka topic called “topic1” with 3 partitions and a replication factor of 1, connect to the Kafka broker running at localhost:9092.
kafka-topics.sh --bootstrap-server localhost:9092 --topic topic1 --create --partitions 3 --replication-factor 1
Output:
Creating kafka topic
Additional important Parameters You Can Set (Advanced)
--config
You may configure topics at the topic level, using a command like –config max.message.bytes=64000.
--disable-rack-aware
Disable rack-aware replica assignment (not advised; only set if you are confident in your ability to use it)
List Kafka Topic
Use the “–list” option in the CLI for kafka-topics.sh. Listing topics when my Kafka broker is running at localhost:9092
kafka-topics.sh --bootstrap-server localhost:9092 --list
Output:
Listing kafka topics
Describe a Kafka Topic
In the CLI kafka-topics.sh, use the “–describe“ option. Describing topics when my Kafka broker is running at localhost:9092
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic1
Output:
Describing a kafka topic
These are some of the advanced parameters that can be used with “–list” and “–describe.”
--exclude-internal
When using the list or describe commands, ignore internal subjects (they are by default listed).
Using replication status as a filter for topics:
--at-min-isr-partitions
If enabled, only display partitions whose isr count is equal to the given minimum while explaining subjects.
--unavailable-partitions
Only display divisions with an unavailable leader.
--under-min-isr-partitions
A partition will only be shown if its isr count is below the defined minimum.
--under-replicated-partitions
only display the under-replicated partitions.
