EMC Corporation and Cloudera have announced the formation of an alliance so that Hadoop-based services that Cloudera offers will be integrated with EMC's Greenplum technology. This move will help businesses better manage and analyze the challenge of ever-growing big data - including log files, sensor data, emails, images, receipts, research data and so on. The integration between Cloudera's Distribution for ...

Flume is a flexible, scalable, and reliable system for collecting streaming data. The Flume User Guide describes how to configure Flume, and the new Flume Cookbook contains instructions (called recipes) for common Flume use cases. In this post, we present a recipe that describes the common use case of using a Flume node collect Apache 2 web servers logs in order to deliver them to HDFS. Follow this posting ...

Storage, MapReduce and Query are ushering in data-driven products and service "Big data" is data that becomes large enough that it cannot be processed using conventional methods. Creators of web search engines were among the first to confront this problem. Today, social networks, mobile phones, sensors and science contribute to petabytes of data created daily. To meet the challenge of processing such large ...

Hadoop and other tools can unlock critical insights from unfathomable volumes of corporate and external data In the never-ending quest for a competitive advantage, organizations are turning to large repositories of corporate and external data to uncover trends, statistics, and other actionable information to help decide on their next move. Those data sets, along with their associated tools, platforms, and a ...

When Amazon announced the release of Amazon Micro Instances, I was excited about how useful it will be for SMBs. Amazon Micro Instances + Open Source software solves one of the problems faced by SMBs. Some pundits outright dismissed the possibility of using Micro Instances for web hosting. Even though I agree that Micro Instances are not good candidates to replace traditional web hosting, what I was highlig ...

To provide a small taste of what the event second annual Hadoop World Conference next month in New York will offer, I corresponded with Hadoop World speaker Linden Hillenbrand, product manager of Hadoop Technologies at General Electric, to get an idea of how GE leverages Hadoop and the use case he'll be presenting at the show. Hillenbrand has been using Hadoop for six months, starting with distribution 18 f ...

Analysis of online activity to make predictions about future behavior The task I was charged with at my most recent position was to build the backend database system and application to handle modeling processing on our VLD (Very Large Data sets). What the business needed was a way to analyze past online activity in the hopes of making predictions about future online shopping behavior (propensity modeling). ...