Below are the steps to increase the Replication Factor for a topic
To list all the topics in your cluster
$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — list
To view all the default and custom configurations for a topic
$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — topic <topic_name> — describe
The first line of the output describes the configs. For example:
Topic: <topic_name> PartitionCount: 8 ReplicationFactor: 2 Configs:
The above output tells us that this topic has a Partition Count as 8 with a Replication Factor of 2.
Say we would like to increase the Replication Factor to 3 for this topic. In which case, we need to construct the following increase-replication-factor.json file that elaborates the increase in Replication Factor from 2 to 3.
cat increase-replication-factor.json
{“version”:1,
“partitions”:[
{“topic”:”<topic_name>”,”partition”:0,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:1,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:2,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:3,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:4,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:5,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:6,”replicas”:[1,2,3]},
{“topic”:”<topic_name>”,”partition”:7,”replicas”:[1,2,3]}
]}
To execute the replication execute the following command:
$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — execute
To view the progress of the process the following command can be used
$KAFKA/kafka-reassign-partitions.sh — list — bootstrap-server $BROKER — command-config $CONFIG/client.properties
You can verify the progress of the process via the following command
$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — verify
Once the replication factor is increased the above command will display the following output
Status of partition reassignment:
Reassignment of partition <topic_name>-0 is complete.
Reassignment of partition <topic_name>-1 is complete.
Reassignment of partition <topic_name>-2 is complete.
Reassignment of partition <topic_name>-3 is complete.
Reassignment of partition <topic_name>-4 is complete.
Reassignment of partition <topic_name>-5 is complete.
Reassignment of partition <topic_name>-6 is complete.
Reassignment of partition <topic_name>-7 is complete.
The following environment variables have been considered
$KAFKA — location of the bin directory /root/kafka/bin
$CONFIG — location of config directory /root/kafka/config
$ZOOKEEPER — zookeeper string in the localhost:port format
$BROKER — broker string in the localhost:port format
