Is big data a big failure?

Big data this, big data that. It's talked up as a business revolution, and has caused many a boring PowerPoint presentation, but are businesses actually using – and benefiting from – big data?

According to recent pan-European research into the impact big data has on organisations, over half of businesses have missed opportunities they didn't see coming because they lacked accurate information at a time when they really needed it – and it's cost them as much as £20 million (around $29 million, AU$41 million) per year.

What's more, Pure Storage's research showed that over 72% of businesses admit to collecting data they never use, calling it time-consuming and expensive to process.

"The reason we're seeing these trends emerge is because it is now cheaper for businesses to retain the data they are collecting, than to destroy it – so the volume of data a business holds is growing rapidly," says James Petter, VP EMEA, Pure Storage. "But at the same time, it is complicated and costly to access usable information fast enough to make a difference."

So is big data going to waste?

Are companies making the most out of big data?

Business transformation consultancy Moorhouse recently commissioned some independent research into FTSE250, UK public sector and UK-headquartered multinationals that showed only 11% of such organisations believe they are leveraging big data effectively to inform their strategic decisions.

"First the industry promised untold benefits, we then moved on to scepticism and we're now at the point of seeing the first practical wins," says Matt Davies, Head of EMEA Marketing at Splunk. "We're only at the beginning of the big data story and the next couple of years should see the industry focusing on the value and delivering on that early promise of big data."

Is big data all talk?

It's a buzz-phrase, it's a popular hashtag on social media, and it looks great on a CV. Big data is so 'zeitgeisty'.

"IT directors and marketing directors face growing corporate peer pressure to invest in big data," says Laurence Armiger, Sales Director at Zizo. "Some IT directors are dabbling with big data because they want to have it on their CV – 'I've put two million records into a Hadoop database' etc. – not because they think it will help their business."

Meanwhile, the sales and marketing departments can't get their heads around exactly what big data is for.

Is there a lack of workers skilled in data management?

Data management and analysis may have the power to unlock the potential of a business, but educated, professional big data experts are thin on the ground. "Most analysts in business are still enthusiastic early adopters, motivated to self-educate," says Giles Slinger, Director at data analytics firm Concentra.

However, universities are now starting to create courses in analytics. In the UK, the Nuffield Foundation's Q-Step programme is advancing analytics at 15 universities, while international business schools are now offering MBAs in big data and business analytics.

"These will generate graduates in one to three years," says Slinger. "In the meantime, businesses have an opportunity to educate employees about the benefits of data analysis in supporting their daily tasks and integrating operations."

Finding the right people

It could be a long three years; recruitment website Indeed.com currently lists over 750,000 data analytics jobs, yet these positions are not being filled. "75% of companies are now actively recruiting people with skills in advanced data analytics," says Tom Pohlmann, Head of Values and Strategy at Mu Sigma, "but it can be difficult to find candidates with the creativity and experimental mind-set to truly revolutionise the handling of big data to truly transform a business' approach."

The new trend for personalised advertising will put even more pressure on graduates, who'll need a "scientific, artistic and emotive mind-set" if they're to cope with the coming phase of extreme experimentation.