Multi Node Apache Kafka 0.9 in Linux

Manually Deploying Apache Kafka

Please Install Java version 8 before proceeding the Kafka installation.
1) Edit the /etc/hostname in each node.

        $ vi /etc/hostname

                   kafka1 or kafka2 or kafka3

2) Check hostname.

        $ hostname

3)  Confirm everything went right.

     $ sudo hostname -F /etc/hostname

4) Now edit /etc/hosts to point one node to other nodes and do the same in each node.

     $ nano /etc/hosts

         127.0.0.1       localhost

127.0.1.1       ubuntu

<IP of node1 >   kafka1

<IP of node2>    kafka2

<IP of node3>    kafka3

# The following lines are desirable for IPv6 capable hosts

::1     localhost ip6-localhost ip6-loopback

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

5) Reboot each node.

6) Download Scala and untar it in all nodes.

    Add following below command in .bashrc

$ export SCALA_HOME=/root/scala-2.11.7

$ export PATH=$PATH:$SCALA_HOME/bin

7) Download Apache Kafka and untar it in all nodes and do the following commands.

$ cd kafka/config/

$ vi zookeeper.properties

dataDir=/root/zookeeper

# the port at which the clients will connect

clientPort=2181

# disable the per-ip limit on the number of connections since this is a non-production config

maxClientCnxns=0

initLimit=5

syncLimit=2

server.1=kafka1:2888:3888

server.2=kafka2:2888:3888

server.3=kafka3:2888:3888

$ vi server.properties in all nodes

               broker.id=<1 or 2 or 3>

               listeners=PLAINTEXT://:9092

               port=9092

               host.name=<kafka1 or kafka2 or kafka3>

               advertised.host.name=<IP of node>

               advertised.port=9092 

               log.retention.ms=86400000

               log.roll.ms=86400000

               log.cleaner.delete.retention.ms=10000

               log.flush.interval.messages=1000000

               log.flush.scheduler.interval.ms=60000

               log.cleaner.threads=4

               log.segment.bytes=1073741824

               log.retention.check.interval.ms=3000

               log.cleaner.enable=true

               zookeeper.connect=kafka1:2181,kafka2:2181,kafka3:2181

               zookeeper.connection.timeout.ms=30000

               delete.topic.enable = true

               auto.create.topics.enable = true

               default.replication.factor=2

               auto.leader.rebalance.enable=true

               controlled.shutdown.enable=true

               controller.socket.timeout.ms=120000

Create a path directory that you have defined in zookeeper.properties file in key dataDir and then within dataDir directory touch a file myid. Now edit myid file and write same value as you have defined in the key broker.id of server.properties.

8) Automatically start Kafka on Reboot (in all nodes) 

      $ cd /etc/init
      $ vi kafka.conf

kill timeout 300

start on runlevel [2345]

stop on runlevel [06]

script

    KAFKA_HOME=/root/kafka_2.10-0.9.0.0

    nohup $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties > /root/zookeeper.log 2>&1 &

    sleep 30

    nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties > /root/kafka.log 2>&1 &

end script