We get asked a lot of questions about how to select Apache Hadoop worker node hardware. During my time at Yahoo!, we bought a lot of nodes with 6*2TB SATA drives, 24GB RAM and 8 cores in a dual socket configuration. This has proven to be a pretty good configuration. This year, I’ve seen systems with 12*2TB SATA drives, 48GB RAM and 8 cores in a dual socket configurations. We will see a move to 3TB drives th ...

Hadoop technologies have started to mature and many of us are now tasked at implementing Hadoop MapReduce based solutions at our respective organizations. From our own practical experiences, we can tell that the going is not going to be as smooth as you might have expected. That’s what peaked our interest in this webinar by Platform Computing: Top Issues IT faces with Hadoop MapReduce This webinar claims th ...

Third Eye is launching "Byte Sized Hadoop" classes – for as low as $125/class! These Hadoop classes will be offered byte-sized, 3 hours each, after work hours & conducted by industry veterans with a practical "from-the-trenches" approach. BigDataCloud meetup attendees have heard & met the instructors individually: Paul Baclace - presented "Optimizing bursty Hadoop analysis demands for big data using A ...

I spend lot of time tracking and monitoring Hadoop jobs running across multiple clusters in my current project. Typically I navigate around multiple Job tracker web admin consoles. Although the job tracker web console gives some basic system level statuses and metrics for Hadoop daemons, it leaves a lot to be desired. What’s missing is a monitoring platform at the application level. In my Hadoop job I may h ...

Jobs around Hadoop technologies have exploded while the number of professionals experienced on Hadoop have not! The current job markets around Hadoop technologies is a very different one, one that is opposite to everything else other job markets all over the US are experiencing today. Jobs around Hadoop technologies have exploded, in all parts of the US, in companies big and small – while the number of prof ...

Users share billions of pieces of content daily on Facebook, and it’s the data infrastructure team's job to analyze that data so we can present it to those users and their friends in the quickest and most relevant manner. This requires a lot of infrastructure and supporting data, so much so that we need to move that data periodically to ever larger data centers. Just last month, the data infrastructure team ...

Services is going to play a huge role in the ultimate success (or failure) of wide-spread adoption of Hadoop and related Big Data technologies by “mainstream” enterprises. The question is: Will Big Data services be delivered by commercial Hadoop vendors like Cloudera and Hortonworks as part of their value-add to the open source framework, or will a separate market of third-party services providers and consu ...

A decade ago, the open-source LAMP (Linux, Apache, MySQL, PHP/Python) stack began to transform web startup economics. As new open-source webservers, databases, and web-friendly programming languages liberated developers from proprietary software and big iron hardware, startup costs plummeted. This lowered the barrier to entry, changed the startup funding game, and led to the emergence of the current Angel/S ...

In the past year, big data has emerged as one of the most closely watched trends in IT. Organizations today are generating more data in a single day than that the entire Internet was generated as recently as 2000. The explosion of “big data”–much of it in complex and unstructured formats–has presented companies with a tremendous opportunity to leverage their data for better business insights through analyti ...

The ability to analyze enormous amounts of data and use that to better target an ad, make a new drug or any number of other lofty goals is talked about constantly, but what about harnessing Hadoop for the rest of us? Something as simple as figuring out when the best time to schedule a meeting and get everyone to attend or let a regional manager select what inventory to order can have huge boosts for busines ...