Is big data a big failure?

The 'black box' approach

If you want genuinely useful business insight, you have to work at it. "Too many think an analytics strategy means choosing a specific black box to be fed with specific data feeds, and applying specific pre-set algorithms," says Nick Clarke, Head of Analytics at international analytics, software and consulting services company Tessella. "But many problems are too complex and too subtle to automate in this way … the correlations that pop out the other end are not magically wrapped up in valuable business insight."

All big data is not the same, and not even Google can get insight from any unstructured data set. See the failure of the Google Flu Trends (GFT) project for proof of that – complex issues require sophistication when it comes to big data.

Getting it right

However, there are plenty of examples of getting it right. "The pharmaceuticals industry has long used vast complex data sets to identify profitable areas for new research," says Clarke. "Although still far from perfect, their maturing combination of data analytics and clarity of vision of what they want to achieve, puts them at the front of the game."

Their secret is the embedding of scientifically literate data experts within specialist research groups. "Human expertise is vital when framing the problem, contextualising the insight, and uncovering bias, both in the data feeds and in our assumptions," says Clarke. "The data strategies of too many organisations fail at this level."

On the cusp?

Some think we're now on the cusp of being able to deliver insights on a computational scale. "The information age is finally finding its 'steam engine'," says Pohlmann. "This alignment of scale and technology means that we can now create algorithms to support things like machine learning processes, which sees big data evolve from being purely a numbers game to an integral part of any corporate planning process."

Is big data a big failure so far?

"No," says Pohlmann, who thinks that the problem is that many organisations are wedded to just their existing Enterprise Data Warehouse (EDW), usually in silos, and practicing business intelligence techniques that fall far short of the proactive, predictive operational analytics that can actually deliver on intelligent decision making.

"With the continuing rise of IoT, AI and machine learning, there's going to be more pressure than ever applied to organisations that are stuck in the past to adopt new processes and techniques," he says, "and actually start to benefit from what big data has to offer."

The pace of change

Benefitting from big data can take time, but companies paying only lip service to big data are putting everything at risk. "As companies gather more and more granular data on what they do, the potential to gain understanding and plan accordingly is not just a profitable undertaking, it is a necessity," says Petter.

"Transformation is being forced on organisations at an ever-increasing pace. They must adapt to new ways of doing business, new markets and new practices." Or what? "Or die," says Petter.

Jamie Carter

Jamie is a freelance tech, travel and space journalist based in the UK. He’s been writing regularly for Techradar since it was launched in 2008 and also writes regularly for Forbes, The Telegraph, the South China Morning Post, Sky & Telescope and the Sky At Night magazine as well as other Future titles T3, Digital Camera World, All About Space and Space.com. He also edits two of his own websites, TravGear.com and WhenIsTheNextEclipse.com that reflect his obsession with travel gear and solar eclipse travel. He is the author of A Stargazing Program For Beginners (Springer, 2015),

Latest in Pro
Epson EcoTank ET-4850 next to a TechRadar badge that reads Big Savings
I searched for the best printer deal you won't find in the Amazon Spring Sale
Microsoft Copiot Studio deep reasoning and agent flows
Microsoft reveals OpenAI-powered Copilot AI agents to bosot your work research and data analysis
Group of people meeting
Inflexible work policies are pushing tech workers to quit
Data leak
Top home hardware firm data leak could see millions of customers affected
Representational image depecting cybersecurity protection
Third-party security issues could be the biggest threat facing your business
An image of network security icons for a network encircling a digital blue earth.
Why multi-CDNs are going to shake up 2025
Latest in News
An image of Pro-Ject's Flatten it closed and opened
Pro-Ject’s new vinyl flattener will fix any warped LPs you inadvertently buy on Record Store Day
EA Sports F1 25 promotional image featuring drivers Oscar Piastri, Carlos Sainz and Oliver Bearman.
F1 25 has been officially announced, with this year's entry marking a return for Braking Point and a 'significant overhaul' for My Team mode
Garmin clippd integration
Garmin's golf watches just got a big software integration upgrade to help you improve your game
Robert Downey Jr reveals himself as Doctor Doom to a delighted crowd at San Diego Comic-Con 2024
Marvel is currently revealing the full cast for Avengers: Doomsday, and I think it's going to be a long-winded announcement
Samsung QN90F on yellow background
Samsung announces US prices for its 2025 mini-LED TV lineup, and it’s good and bad news
Nintendo Switch Lite
Forget the Nintendo Switch 2, the original Switch is getting one last hurrah in a surprise Nintendo Direct tomorrow