From the labs of LinkedIn, Kafka, a publish-subscribe messaging system, became a new entrant into the long list of open source project governed by the Apache group. Now, Apache Kafka has been comprehensively integrated into the Cloudera’s Big Data platforms empowering Hadoop to deliver production-ready, real-time analytics.

About Apache Kafka
LinkedIn addressed its technology challenges through Kafka to quickly write a separate data pipeline for each source and destination pairing. LinkedIn was able to standardize the data pipeline and enabled it to get data out of each system once and into each system once thus reducing cost and complexity. Kafka performs well both as a traditional message broker and a method of ingesting events into Hadoop.

Features of Kafka include: scalability; data partitioning; low latency; ability to handle large number of diverse consumer; support for batch processing with no latency; fault-tolerant.

Use cases of Kafka include: website activity tracking; operational metrics; log aggregation; stream processing.

“Now that Kafka is fully integrated into Cloudera’s platform, users can build complete end-to-end workloads, such as real-time streaming, together with components like Apache Spark Streaming and Apache HBase - all within a single system. With this integration, users have greater flexibility and performance for ingesting new and varied streams of data and exploring new use cases, for both faster processing and faster insights,” said Charles Zedlewski, VP, Products at Cloudera.