Hadoop & Startups: Where Open Source Meets Business Data

Posted by: From TechCrunch Posted date: July 17, 2011 In: Hadoop, MapReduce, Open Sources | comment : 0

A decade ago, the open-source LAMP (Linux, Apache, MySQL, PHP/Python) stack began to transform web startup economics. As new open-source webservers, databases, and web-friendly programming languages liberated developers from proprietary software and big iron hardware, startup costs plummeted. This lowered the barrier to entry, changed the startup funding game, and led to the emergence of the current Angel/Seed funding ecosystem. In addition, of course, to enabling a generation of webapps we all use everyday.

This same process is now unfolding in the Big Data space, with an open-source ecosystem centered around Hadoop displacing the expensive, proprietary solutions. Startups are creating more intelligent businesses and more intelligent products as a result. And perhaps even more importantly, this technological movement has the potential to blur the sharp line between traditional business and traditional web startups, dramatically expanding the playing field for innovation.

PROBLEM, MEET SOLUTION

Even a modestly successful startup has a user base comparable in population to nation-states. The resultant mass of user data creates problems and opportunities. Problems because understanding the value of every user and transaction becomes more complex. Opportunities because the collective intelligence of the population can be leveraged into better user experiences.

Until just a year or two ago, analyzing this scale of data required the same kind of enterprise solutions that LAMP was created to avoid. Multiyear, multimillion dollar deals with the likes of IBM, Oracle, and Teradata. Of course, almost no startup can afford that kind of expense. Furthermore, the closed-source technological pedigree of these solutions makes them incompatible with startup engineering knowledge and culture.

Enter Hadoop. Hadoop solves these data processing problems in a way that is both startup-compatible, and technologically superior. As an open-source project developed by and for engineers, it’s very practical and squarely in the mainstream of startup engineering practice. And its architecture of map-reducing across of a cluster of commodity nodes is more flexible and cost effective than traditional data warehouses.

Given the pent-up demand, it’s no surprise then that Hadoop is blowing up. To see how an open-source project is doing, the first thing you do is look at the developer mailing list traffic. If you gather all the Hadoop-related mailing lists and plot the number of messages, you get the classic hockey stick growth curve:

In today’s world, it’s not much of a stretch to say that the future of technology is shaped by developer adoption. As more engineers and more startups adopt Hadoop, its capabilities are becoming a default assumption when designing the products and businesses of the future.