Real Time Application (Twitter) - Apache Kafka

Explain the Real Time Application (Twitter) of Apache Kafka

The real time application is analyzed to obtain the twitter feeds and its hashtags. In the integration of Storm and Spark with Kafka, a Kafka Producer (using cli) is created to send message to the Kafka ecosystem. Using the Kafka consumer the messages are read by storm and spark integration, and is injected into storm and spark ecosystem respectively. Hence a Kafta producer is created which -

  • Read the twitter feeds using “Twitter Streaming API”,
  • Process the feeds,
  • Extract the HashTags and
  • Send it to Kafka.

Once the HashTags are received by Kafka, the Storm / Spark integration receive the information and send it to Storm / Spark ecosystem.

Twitter Streaming API

Any programming language can access the “Twitter Streaming API”. To easily access the “Twitter Streaming API”, a Java based module is provided by an open source unofficial Java library, “twitter4j”. To access the tweets, a listener based framework is provided by “twitter4j”. To access the “Twitter Streaming API”, sign in for Twitter developer account and should get the following OAuth authentication details.

  • Customerkey
  • CustomerSecret
  • AccessToken
  • AccessTookenSecret

On the creation of developer account, “twitter4j” jar files are downloaded ad are placed in Java class path.

The Complete Twitter Kafka producer coding ( is listed below −


The application is compiled by the following command:


Open two consoles and in one console, run the above compiled application as:

In a separate window, run any one of the Spark / Storm application as explained in previous chapter. The topic used should be the same in both cases. The topic name used is “my-first-topic”.


The output of this application depends on the keywords and the current feed of the twitter. A sample output is specified below (storm integration).

All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd Protection Status

Apache Kafka Topics