You Are Here: Home » Amazon Kinesis

Exactly-once Semantics is Possible: Here’s How Apache Kafka Does it

I’m thrilled that we have hit an exciting milestone the Kafka community has long been waiting for: we have introduced exactly-once semantics in Apache Kafka in the 0.11 release. In this post, I’d like to tell you what exactly-once semantics mean in Apache Kafka, why it is a hard problem, and how the new idempotence and transactions features in Kafka enable correct exactly-once stream processing using Kafka’ ...

Read more

Amazon Kinesis vs. Apache Kafka For Big Data Analysis – Dataconomy

Data processing today is done in form of pipelines which include various steps like aggregation, sanitization, filtering and finally generating insights by applying various statistical models. Amazon Kinesis is a platform to build pipelines for streaming data at the scale of terabytes per hour. Parts of the Kinesis platform are a direct competitor to the Apache Kafka project for Big Data Analysis. The platf ...

Read more

Serverless Scaling for Ingesting, Aggregating, and Visualizing Apache Logs with Amazon Kinesis Firehose, AWS Lambda, and Amazon Elasticsearch Service | AWS Database Blog

In 2016, AWS introduced the EKK stack (Amazon Elasticsearch Service, Amazon Kinesis, and Kibana, an open source plugin from Elastic) as an alternative to ELK (Amazon Elasticsearch Service, the open source tool Logstash, and Kibana) for ingesting and visualizing Apache logs. One of the main features of the EKK stack is that the data transformation is handled via the Amazon Kinesis Firehose agent. In this pos ...

Read more

Send Apache Web Logs to Amazon Elasticsearch Service with Kinesis Firehose | AWS Database Blog

We have many customers who own and operate Elasticsearch, Logstash, and Kibana (ELK) stacks to load and visualize Apache web logs, among other log types. Amazon Elasticsearch Service provides Elasticsearch and Kibana in the AWS Cloud in a way that’s easy to set up and operate. Amazon Kinesis Firehose provides reliable, serverless delivery of Apache web logs (or other log data) to Amazon Elasticsearch Servic ...

Read more

Implement Serverless Log Analytics Using Amazon Kinesis Analytics | AWS Big Data Blog

Applications log a large amount of data that—when analyzed in real time—provides significant insight into your applications. Real-time log analysis can be used to ensure security compliance, troubleshoot operation events, identify application usage patterns, and much more. Ingesting and analyzing this data in real time can be accomplished by using a variety of open source tools on Amazon EC2. Alternatively, ...

Read more

Writing SQL on Streaming Data with Amazon Kinesis Analytics – Part 2 – AWS Big Data Blog

Amazon Kinesis Analytics allows you to easily write SQL ­­­on streaming data, providing a powerful way to build a stream processing application in minutes. The service allows you to connect to streaming data sources, process the data with sub-second latencies, and continuously emit results to downstream destinations for use in real-time alerts, dashboards, or further analysis. This post introduces you to th ...

Read more

Analyze Realtime Data from Amazon Kinesis Streams Using Zeppelin and Spark Streaming – AWS Big Data Blog

There is streaming data everywhere. This includes clickstream data, data from sensors, data emitted from billions of IoT devices, and more. Not suprisingly, data scientists want to analyze and explore these data streams in real time. This post shows you how you can use Spark Streaming to process data coming fromAmazon Kinesis streams, build some graphs using Zeppelin, and then store the Zeppelin notebook in ...

Read more

Optimize Spark-Streaming to Efficiently Process Amazon Kinesis Streams – AWS Big Data Blog

Do you use real-time analytics on AWS to quickly extract value from large volumes of data streams? For example, have you built a recommendation engine on clickstream data to personalize content suggestions in real time using  Amazon Kinesis Streams and Apache Spark? These frameworks make it easy to implement real-time analytics, but understanding how they work together helps you optimize performance. In thi ...

Read more

Real-world applications with Amazon Machine Learning

How to apply Machine Learning to social media to make your customers happyAt his AWS re:Invent presentation, Alex Ingerman – technical product manager at AWS – went through the design and implementation of a real-world end-to-end application to transform a high-volume social stream into actionable support requests for your customer support team.The idea is that social networks like Twitter generate a huge a ...

Read more

Amazon Kinesis Update – Simplified Capture of Streaming Data | AWS Official Blog

Amazon Kinesis is a managed service designed to handle real-time streaming of big data. It can accept any amount of data, from any number of sources, scaling up and down as needed (see my introductory post for more information on Kinesis). Developers can use the Kinesis Client Library (KCL) to simplify the implementation of apps that consume and process streamed data. Today we are making the capture of stre ...

Read more

2015 © Big Data Cloud Inc. All Rights Reserved.

Hadoop and the Hadoop elephant logo, Sprark are trademarks of the Apache Software Foundation.

Scroll to top