While experts debate if Big Data is still scaling the heights of Gartner’s Hype Cycle Peak of Inflated Expectations or already freefalling down the Trough of Disillusionment, what’s not in doubt is that demand for data scientists, the brainy boffins expected to transmute ridiculous amounts of data into business gold, is exploding. Demand for data scientists has tripled over the past six years as companies pursue the benefits of predictive analytics and Big Data, said Stephen Purpura, founder and CEO, Context Relevant, a developer of predictive analytics applications.
Predictive analytics require scientists with extensive training in computer science and statistics, but these people are rare, he said. “As a person who has worked on providing Big Data solutions for about a decade, I can tell you that the reason we started the company is it’s so hard to build them using existing tools, so no wonder why organizations struggle. Hiring them is difficult, especially if you’re not Google or Microsoft or a big player who has had success. Customers were telling us they couldn’t find these people.”
IDC predicts that investment in Big Data technologies and services will grow to nearly $10 billion this year and $23.8 billion by 2016, while Nemertes Research has found that nearly 30% of organizations have initiated Big Data projects. Gartner is even more bullish, expecting this segment to account for $32 billion in 2013 and $232 billion through 2016.
However, in addition to the cost, Gartner has warned that finding the right people to take advantage of Big Data will be a huge problem.
“By 2015, 4.4 million IT jobs globally will be created to support big data, generating 1.9 million IT jobs in the United States,” said Peter Sondergaard, senior vice president at Gartner and global head of Research. “In addition, every big data-related role in the U.S. will create employment for three people outside of IT, so over the next four years a total of 6 million jobs in the U.S. will be generated by the information economy.”
Organizations are looking for Java programmers who can write code on Hadoop, said Purpura, but these people are expensive and rare. The problem becomes more complex because even the best Java developers may not have the statistical background to get into what’s interesting about the data, he said.
Until now, scientists were required to develop models through trial and error. “The challenge isn’t Big Data or analysis, it’s providing immediate business benefit in a world where data scientists are scarce and tools are limited.”
Big Data and data scientist demand may appear to be soaring, but the urgency may be just a bit overblown, cautions Evan Quinn, Senior Principal Analyst, Enterprise Strategy Group, in a new blog. In ESG’s just published 2013 IT Spending Intentions Survey, which suggests around a 2% overall increase in IT spending, the closest proxy to “big data,” specifically “improved data analytics…,” tied for fifth place, among a longish list of “Business Initiatives with the Greatest Impact on IT Spending Decisions.” Note that if one only focuses on IT priorities not tied to business initiatives per se, BI/ analytics ties for ninth place.
2013 will be an excellent year for big data, Quinn writes, but crazy claims like big data will augment the IT workforce by 25% over the next two years, or that big data will grow from a $5b to $50b in 5 years, and other wacky market sizing and forecast exercises need zapping. Thankfully, ESG’s IT spending survey for 2013 helps take the inaccurate creep out of big data forecasts, but also clarifies that big data will do more than merely crawl along.
Whether the Big Data market is $5 or $50 billion, its demand is keeping Context Relevant busy, said Purpura. “The customer pull is fantastic. The venture capitalists have never seen demand like we’ve seen.”