Sunday, March 5, 2017

Configuring and Running Apache Kafka in IBM BigInsights

This blog describes on Configuring and running the Kafka from IBM BigInsights.

Apache Kafka is an open source that provides a publish-subscribe model for messaging system. Refer :

I assume that you were aware of  terminologies like Producer, Subscriber, Kafka Brokers, Topic and Partitions. Here, I will be focusing on creating multiple Brokers in BigInsights then create a topic and publish the messages from command line and consumer getting it from the Broker.

Environment: BigInsights 4.2

 Step 1: Creating Kafka Brokers from Ambari

By default, Ambari will have one Kafka Broker configured.  Based on your usecase, you may need to create multiple brokers.

Login to Ambari UI --> Click on Host and add the Kafka Broker to the node where you need to install Broker.

 You can see multiple brokers running in Kafka UI.

Step 2: Create a Topic

Login to one of the node where broker is running.  Then create a topic.

cd /usr/iop/

su kafka -c "./ --create --zookeeper localhost:2181 --replication-factor 2 -partitions 1 --topic CustomerOrder"

You can get the details of the topic using the below describe command.

su kafka -c "./ --describe --zookeeper localhost:2181 --topic CustomerOrder"

Step 3: Start the Producer

In the argument --broker-list, pass all the brokers that are running.

su kafka -c "./ --broker-list, --topic CustomerOrder"

When you run the above command, it will be waiting for user input. You can pass a sample message

{"ID":99, "CUSTOMID":234,"ADDRESS":"12,5-7,westmead", "ORDERID":99, "ITEM":"iphone6", "COST":980}

Step 4: Start the Consumer

Open an other Linux terminal and start the consumer. It will display all the messages send to producer.

su kafka -c "./ --zookeeper localhost:2181 --from-beginning --topic CustomerOrder"


 Thus, We are able to configure and perfom a sample pub-sub system using Kafka.


Eva Joseph said...

Thanks for your interesting ideas.the information's in this blog is very much useful for me to improve my knowledge. Big Data Analytics Services Companies
Web Design and Development Company
Ecommerce Service Provider
Data Extraction Services
Payment Gateway Providers in India
Online Appointment Scheduling Software
Digital Marketing Services

nisharoshan said...

I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..

Web Designing Course Training in Chennai | Web Designing Course Training in annanagar | Web Designing Course Training in omr | Web Designing Course Training in porur | Web Designing Course Training in tambaram | Web Designing Course Training in velachery

tejaswini said...

I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…data science course

DVS Technologies said...

Thank you For Sharing Information . We are giving all software Courses such as
DVS Technologies
AWS Training in Bangalore
AWS Training institute in Bangalore
AWS Training institutes Best Data Science Training in Bangalore
Data Science Training institute in Bangalore
Data Analytics Training in Bangalore
Python Training in Bangalore
Python Training institute in Bangalore
Big Data training in Bangalore
Best Hadoop Training institute in Bangalore
Hadoop Training institute in Bangalore
Data Science Training institute in Bangalore
Best Data Science Training in Bangalore
Spark Scala Training in Bangalore
Best Spark Training institutes in Bangalore
Devops Training Institute In Bangalore Marathahalli
SNOW FLAKE Training in Bangalore

Digital Marketing Training in Bangalore

Payal Sathe said...
This comment has been removed by the author.
Python said...

Informative blog. Thank you for sharing with us..
Data Science Online Training

Unknown said...

Great blog
microstrategy courses
microstrategy training

360digitmgdelhi said...

This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..
data science course in noida

Devi said...

Thank you for posting informative insights, I think we have got some more information to share with! Do check out Big data training in chennai and let us know your thoughts. Let’s have great learning!

Realtime Experts said...

Deep Learning Projects assist final year students with improving your applied Deep Learning skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include Deep Learning projects for final year into your portfolio, making it simpler to get a vocation, discover cool profession openings,
IBM Training in Bangalore

360digitmgdelhi said...

Nice and very informative blog, glad to learn something through you.
data analytics courses delhi

360digitmgdelhi said...

Amazing blog.Thanks for sharing such excellent information with us. keep sharing...
data analytics courses delhi

Kathy said...

Well written articles like yours renews my faith in today's writers. The article is very informative. Thanks for sharing such beautiful information.
Best Data Migration tools
Penetration testing companies USA
What is Data Lake
Artificial Intelligence in Banking
What is Data analytics

360DigiTMGAurangabad said...

This Blog is very useful and informative.
best machine learning course in aurangabad

Hemapriya said...

Phenomenal Blog!!! thanks for your post and awaiting for your new updates...
Digital Marketing Course in Chennai
Digital Marketing Online Course
Digital Marketing Course in Coimbatore

Hemapriya said...

Wonderful Blog.... Thanks for sharing with us...
Hadoop Training in Chennai
Hadoop Training in Bangalore
Big Data Onlie Course
Big Data Training in Coimbatore

360DigiTMG-Pune said...

Good information you shared. keep posting.
data science course in pune

nayar said...

Amazing Post. keep update more information.
Selenium Training in Bangalore
Selenium Training in Pune
Selenium Taining in Hyderabad
Selenium Training in Gurgaon
Selenium Training in Delhi

chitra said...

Intersting Information... keep sharing your blog
RPA Course in Chennai
Robotic Process Automation Training in Bangalore

Devi said...

Great post. keep sharing such a worthy information

cyber security course in bangalore
cyber security training in chennai

ramyapranauv said...

Amazing Blog...keep sharing like this
Python Training in Bangalore
Python Course in Bangalore
Best Python Training in Bangalore
Python Training in Chennai
Python Training Institute in Chennai
Best Python Training in Chennai

360DigiTMGAurangabad said...

thanks for share
artificial intelligence course aurangabad

360DigiTMGAurangabad said...

This is my first time visit here. From the tremendous measures of comments on your articles.I deduce I am not only one having all the fulfillment legitimately here!
artificial intelligence training aurangabad

Reshma said...

Wonderful post and more informative!keep sharing Like this!
salesforce institute in bangalore
Salesforce institute in bangalore

data science marketplace said...

Thanks for the great information
data science marketplace

lakshmibhucynix said...

Fantastic article post.Really thank you! Awesome.
data science course in hyderabad
data science training in hyderabad

bruce wayne said...

Excellent blog.thanks for sharing such a worthy information....
Best Selenium Training Institute in Bangalore

Dhilshan said...

Happy to read the informative blog. Thanks for sharing
best digital marketing course in chennai
best digital marketing training in chennai

manasha said...

Great post. keep sharing such a worthy information.
Google Analytics Training In Chennai
Google Analytics Online Course

data science marketplace said...

Thanks for the post.
data science marketplace

Pavithra Devi said...

Such a good post .thanks for sharing
Software testing training in Porur
Software testing training in chennai

manasha said...

Great post. keep sharing such a worthy information.
Web Designing Course In Chennai

360DigiTMGAurangabad said...

Thanks for sharing nice information....
data science courses in aurangabad

traininginstitute said...

Excellent effort to make this blog more wonderful and attractive.
full stack web development course

Matt Reeves said...

Mindblowing blog very useful thanks
Tally Course in Velachery
Tally course in Chennai