It’s The API, Stupid! (Part 3)

Posted by: admin Posted date: July 05, 2011 In: Big Data, Hadoop, MapReduce | comment : 0

Last week was definitely a busy one in the MapReduce world! At the annual Hadoop Summit, Yahoo! officially announced the spinoff of HortonWorks (possibly the worst kept secret in the Hadoop community), and Cloudera and MapR both announced new distributions. With even more fragmentation coming to the Hadoop community, what better time to wrap up this series on the state of MapReduce.

In my previous two posts, I argued that the current state of MapReduce APIs is the fundamental limiting factor in widespread adoption of MapReduced-based technologies, such as Hadoop. Specifically, I argued that there are two big problems with today’s MapReduce APIs:

The APIs of various MapReduce-based technologies are generally not compatible with one-another
With a few notable exceptions, the APIs of most MapReduce-based technologies are not accessible to the majority of users
In Part 2 of this series, I discussed at length the impact of a lack of compatibility on ecosystem development, and how that holds back enterprise adoption. In this post, I’ll tackle the second problem with today’s MapReduce APIs: most people can’t use them!