Hadoop, Hadoop everywhere – but not a developer to work on it.
Jobs around Hadoop technologies have exploded while the number of professionals experienced on Hadoop have not!
The current job markets around Hadoop technologies is a very different one, one that is opposite to everything else other job markets all over the US are experiencing today. Jobs around Hadoop technologies have exploded, in all parts of the US, in companies big and small – while the number of professionals with the required abilities are very scarce, localized & pricey!
The Demand Side:
Do a simple test:
Go to Dice.com and search for “Hadoop” jobs. You will find, as of July 22nd 2011, 508 openings all across the US. Yes, the “hadoop” openings are not only in our dear-old, pie-in-the-sky, a-whole -world-by-itself, “bubble-country” Bay Area but ALL across the US. These openings are from places as diverse as Santa Clara, CA and Cary, NC. The openings are not only from startups but also at big companies as the above links will prove.
Thus to recap, across the whole of US, there are 508 openings for software professionals with Hadoop experiences.
The Supply Side:
Now, do another simple test :
On Dice.com, login as an employer and search for “hadoop” developers. Make sure that the filter is for whole of US. You will find a grand total of 262 candidates!
Thus to recap, across the whole of US, there are a total of 262 software professionals with Hadoop experiences!
It’s been sometime since I have seen such a deviation – actually, the last time I saw such gaps in “demand-supply” was during the hey days of the dot.com rush, circa 97-99. Exploring the reasons for such a deviation, the answers are quick & easy:
- Human-kind never had so much of data since the last five year and so software professionals did not have to deal with so much data.
- Only a handful few lucky ones, who happened to work at data giants like eBay, Walmart, Amazon.com etc. had such exposures – and today they are sitting in companies like Google, Facebook & Twitter, with millions in stock options!
- Companies like Google, Facebook & Twitter are talent magnets! They have been able to lure great talent by the thousands by offering them much more than just annual salaries – lucrative stock options, free lunches, on-premise day care etc. This leaves the rest of the companies trying to fill in their open Hadoop positions in a lurch!
- Earlier, technology was simply not accessible to most companies. The Teradata’s of the world typically cost in the millions for anyone to implement and operate and so only a handful of data giants had the financial muscle to afford them whereas Hadoop based technologies are open-source and free! Every company now can afford them – but still need professionals to implement & operate!
Thus, we are in some sort of a hyper state where the demands from employers have increased substantially while the supply of such Hadoop trained professionals simply does not match up. The only way to solve this problem – creation!
We simply need to create more professionals trained on Hadoop – and the act of creation has already started. They are being created in training centers, employer sponsored class-rooms, universities offering Hadoop classes. Soon, educational institutions across countries like India, China would catch up too!
Thus, the future is not as bleak as it is now. These efforts would soon start churning out qualified professionals to catch up with the demand. Pretty soon the price wars among US based professionals would be fought, then off shoring will happen, then politicians will start complaining jobs are being lost & yada, yada, yada …
But we all are used to those cycles and heard these things before – haven’t we