Sunday, March 5, 2017

Configuring and Running Apache Kafka in IBM BigInsights

This blog describes on Configuring and running the Kafka from IBM BigInsights.

Apache Kafka is an open source that provides a publish-subscribe model for messaging system. Refer : https://kafka.apache.org/

I assume that you were aware of  terminologies like Producer, Subscriber, Kafka Brokers, Topic and Partitions. Here, I will be focusing on creating multiple Brokers in BigInsights then create a topic and publish the messages from command line and consumer getting it from the Broker.


Environment: BigInsights 4.2

 Step 1: Creating Kafka Brokers from Ambari

By default, Ambari will have one Kafka Broker configured.  Based on your usecase, you may need to create multiple brokers.

Login to Ambari UI --> Click on Host and add the Kafka Broker to the node where you need to install Broker.


 You can see multiple brokers running in Kafka UI.




















 
Step 2: Create a Topic

Login to one of the node where broker is running.  Then create a topic.

cd /usr/iop/4.2.0.0/kafka/bin

su kafka -c "./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 -partitions 1 --topic CustomerOrder"









You can get the details of the topic using the below describe command.

su kafka -c "./kafka-topics.sh --describe --zookeeper localhost:2181 --topic CustomerOrder"






 
Step 3: Start the Producer

In the argument --broker-list, pass all the brokers that are running.

su kafka -c "./kafka-console-producer.sh --broker-list bi1.test.com:6667,bi2.test.com:6667 --topic CustomerOrder"

When you run the above command, it will be waiting for user input. You can pass a sample message

{"ID":99, "CUSTOMID":234,"ADDRESS":"12,5-7,westmead", "ORDERID":99, "ITEM":"iphone6", "COST":980}









Step 4: Start the Consumer

Open an other Linux terminal and start the consumer. It will display all the messages send to producer.

su kafka -c "./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic CustomerOrder"

 

 Thus, We are able to configure and perfom a sample pub-sub system using Kafka.


25 comments:

Eva Joseph said...

Thanks for your interesting ideas.the information's in this blog is very much useful for me to improve my knowledge. Big Data Analytics Services Companies
Web Design and Development Company
Ecommerce Service Provider
Data Extraction Services
Payment Gateway Providers in India
Online Appointment Scheduling Software
Digital Marketing Services

nisharoshan said...

I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..

Web Designing Course Training in Chennai | Web Designing Course Training in annanagar | Web Designing Course Training in omr | Web Designing Course Training in porur | Web Designing Course Training in tambaram | Web Designing Course Training in velachery

tejaswini said...

I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…data science course

DVS Technologies said...

Thank you For Sharing Information . We are giving all software Courses such as
DVS Technologies
AWS Training in Bangalore
AWS Training institute in Bangalore
AWS Training institutes Best Data Science Training in Bangalore
Data Science Training institute in Bangalore
Data Analytics Training in Bangalore
Python Training in Bangalore
Python Training institute in Bangalore
Big Data training in Bangalore
Best Hadoop Training institute in Bangalore
Hadoop Training institute in Bangalore
Data Science Training institute in Bangalore
Best Data Science Training in Bangalore
Spark Scala Training in Bangalore
Best Spark Training institutes in Bangalore
Devops Training Institute In Bangalore Marathahalli
SNOW FLAKE Training in Bangalore

Digital Marketing Training in Bangalore


Payal Sathe said...
This comment has been removed by the author.
Python said...

Informative blog. Thank you for sharing with us..
Data Science Online Training

Unknown said...

Great blog
microstrategy courses
microstrategy training

360digitmgdelhi said...

This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..
data science course in noida

Devi said...

Thank you for posting informative insights, I think we have got some more information to share with! Do check out Big data training in chennai and let us know your thoughts. Let’s have great learning!

Myclasstraining said...

Deep Learning Projects assist final year students with improving your applied Deep Learning skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include Deep Learning projects for final year into your portfolio, making it simpler to get a vocation, discover cool profession openings,
IBM Training in Bangalore

360digitmgdelhi said...

Nice and very informative blog, glad to learn something through you.
data analytics courses delhi

360digitmgdelhi said...

Amazing blog.Thanks for sharing such excellent information with us. keep sharing...
data analytics courses delhi

Kathy said...

Well written articles like yours renews my faith in today's writers. The article is very informative. Thanks for sharing such beautiful information.
Best Data Migration tools
Penetration testing companies USA
What is Data Lake
Artificial Intelligence in Banking
What is Data analytics

360DigiTMGAurangabad said...

This Blog is very useful and informative.
best machine learning course in aurangabad

Hemapriya said...

Phenomenal Blog!!! thanks for your post and awaiting for your new updates...
Digital Marketing Course in Chennai
Digital Marketing Online Course
Digital Marketing Course in Coimbatore

Hemapriya said...

Wonderful Blog.... Thanks for sharing with us...
Hadoop Training in Chennai
Hadoop Training in Bangalore
Big Data Onlie Course
Big Data Training in Coimbatore

360DigiTMG-Pune said...

Good information you shared. keep posting.
data science course in pune

nayar said...




Amazing Post. keep update more information.
Selenium Training in Bangalore
Selenium Training in Pune
Selenium Taining in Hyderabad
Selenium Training in Gurgaon
Selenium Training in Delhi

chitra said...

Intersting Information... keep sharing your blog
RPA Course in Chennai
Robotic Process Automation Training in Bangalore

Devi said...

Great post. keep sharing such a worthy information

cyber security course in bangalore
cyber security training in chennai

ramyapranauv said...

Amazing Blog...keep sharing like this
Python Training in Bangalore
Python Course in Bangalore
Best Python Training in Bangalore
Python Training in Chennai
Python Training Institute in Chennai
Best Python Training in Chennai


360DigiTMGAurangabad said...

thanks for share
artificial intelligence course aurangabad

360DigiTMGAurangabad said...

This is my first time visit here. From the tremendous measures of comments on your articles.I deduce I am not only one having all the fulfillment legitimately here!
artificial intelligence training aurangabad

Reshma said...


Wonderful post and more informative!keep sharing Like this!
salesforce institute in bangalore
Salesforce institute in bangalore

data science marketplace said...

Thanks for the great information
data science marketplace