Innovation around the management of large data sets is coming from the cloud, such as through MapReduce and Hadoop InfoWorld's own Pete Babb provided some good coverage around the "analytics cloud" recently debuted by IBM, called Blue Insight. You can think of Blue Insight as a system that gathers data from those who use it and externalizes the data to those who need it, doing so on a cloud -- a private clo ...

In Part 1 of this post the focus was on finding the correlation between items, based on rating data available in individual items. The MR job output was the correlation coefficient matrix, with correlation coefficient values between 0 and 1 for any item pair. Next step Armed with the item correlation data and items rating data for any visitor, we will find the new items correlated with the current items of ...

Personalized recommendations are ubiquitous in social network and shopping sites these days. How do they do it? Al long as enough user interaction data is available for items e.g., products in shopping sites, a kind of recommendation engine based on what’s know as Collaborative Filtering is not that difficult to build. My approach I will follow a technique called Item Based Collaborative Filtering. The basi ...

The Hadoop hoopla is generating increasing numbers of announcements from more and more vendors. From startups to large established players, new products and partnerships are emerging which confirm the emergence of a vibrant Big Data ecosystem evolving around Apache Hadoop. However, there’s frequent misunderstanding of the layers at which companies are operating, which leads to misconceptions over which coll ...

EMC Corporation and Cloudera have announced the formation of an alliance so that Hadoop-based services that Cloudera offers will be integrated with EMC's Greenplum technology. This move will help businesses better manage and analyze the challenge of ever-growing big data - including log files, sensor data, emails, images, receipts, research data and so on. The integration between Cloudera's Distribution for ...

Hadoop and other tools can unlock critical insights from unfathomable volumes of corporate and external data In the never-ending quest for a competitive advantage, organizations are turning to large repositories of corporate and external data to uncover trends, statistics, and other actionable information to help decide on their next move. Those data sets, along with their associated tools, platforms, and a ...

To provide a small taste of what the event second annual Hadoop World Conference next month in New York will offer, I corresponded with Hadoop World speaker Linden Hillenbrand, product manager of Hadoop Technologies at General Electric, to get an idea of how GE leverages Hadoop and the use case he'll be presenting at the show. Hillenbrand has been using Hadoop for six months, starting with distribution 18 f ...