The rise of big data – and the evolution of discussion around how to make the most of it – has been swift, and shows little sign of slowing down. In the past few years, experts and IT vendors the world over have been setting the stage for an analytics-fuelled future in which we will constantly be uncovering new insights from the immense volumes of valuable information we collect, allowing us to make more informed decisions more quickly.
In the case of big data, this hype is indeed justified. But while a great deal has been said about the immense potential of modern analytics and the extraordinary speed at which data can now be analysed, much less focus has been placed on breaking down exactly how big data can impact, and in many cases improve, our lives.
Wide impact
Data analysis on such a large scale stands to completely change how people approach decision-making across a number of fields, including scientific research, banking, medicine, and even professional sport. Of course, the immense volume and variety of data being collected is only the fuel for this analysis. Big data tools allow users to tie all this information together and assess it from multiple perspectives at once in the hope of uncovering new insights, to approach difficult questions from a fresh angle, or merely to process huge data sets very quickly.
Otto Scharmer, a senior lecturer at MIT, recently commented on the value of more meaningful data analysis when he said that "the key to transformative change is to make the system see itself". What this means is that we now have tools that allow us to be more introspective when analysing information. With the right combination of technologies in place, people can assess how efficient and effective their systems and processes are and pinpoint what improvements would make these perform better.
In the grander sense, by looking inward to assess their performance, organisations can begin to uncover insights that will help them fundamentally transform the way they operate.
From a business perspective, big data opens the door to enterprise-scale self-improvement. That being said, the value of this technology extends well beyond the corporate world. The best way to illustrate the value of big data in practical terms is to scale back on the future gazing that has dominated the discussion among businesses and instead explore how the technology is making a noted impact in the world we live in – beginning in this case with the field of medical research.
Sprinting towards the cure
Health professionals dedicate their lives to treating patients and combating disease. Those more inclined towards research spend their days seeking cures for life-threatening illnesses. When it comes to medical studies, the truth is that patients present doctors with the most compelling data set available.
Medical experts have always collected valuable information on each patient's symptoms, the state of their vitals, their medical records, and treatment history each time they check into hospital. This data continues to provide them with the context they need to choose the most appropriate treatment for people dealing with serious illness.
Today, doctors are beginning to examine this detailed patient information alongside data from similar cases on a global scale. With the ability to consolidate patient records and progress from hospitals around the world, and trend that against geographical factors – weather, local customs, family histories – in real-time, researchers stand to gain a more complete view of how a disease works, what does and doesn't affect symptoms, and where to focus their efforts to find a cure.
To add to this, big data tools make the collection and processing of this data an exponentially faster process than it would have been even a few years ago. Taking an analytical approach that is at once broader and more refined can help doctors prescribe more effective treatments for patients in need, and accelerate the drive to cure serious disease.
Pioneering research
The MD Anderson Cancer Centre in Texas has taken a step in this direction and is pioneering the use of big data in research. In traditional medical research organisations, information on what medications patients have taken, what surgeries they've had, and what tests they've undergone has been collected and stored separately. The Texas centre recently centralised all this data onto one "data warehouse" so that all these results now feed into one easily accessible database.
As a result, the onus that was once placed on doctors to investigate and generate research data themselves has now shifted onto purpose-built systems. Studies that previously took months to complete are now being run in a matter of minutes, and the time saved will be invaluable in speeding up the race to cure cancer.
Diving deeper down
Admittedly, sometimes making the most of big data analysis comes down to easily collecting and processing an enormous amount of information.
In the case of pure scientific research, data analysis becomes a very granular process indeed. Scientific studies often have sweeping implications in the long term – the discovery that the Earth is round, the location of the atomic nucleus, and the Human Genome Project have all had a dramatic impact on how people view the world – but the road to enlightenment is often paved with a series of painstakingly slow and iterative processes.
Researchers must collect, consolidate, and extract value from a seemingly endless number of experimental results as they build up to these breakthrough moments.
Big data analysis is ideally suited to quickly consolidating and extracting value from a huge volume of experimental data, and can help researchers delve even deeper into their fields of study to solve the mysteries that continue to elude us.
The Italian National Institute of Astrophysics (INAF) is currently carrying out a research phase for the Gaia mission, an initiative launched by the European Space Agency (ESA) to create a precise three-dimensional map of the Milky Way.
In the next few years, INAF will be in charge of managing and providing access to up to 1 petabyte (1,000 terabytes) of astronomical data generated by the Gaia satellite's observation activity. This information will feed into systems which control and verify that the satellite and its instruments are functioning properly, and therefore that the research is moving forward successfully.
The database employed by INAF to do this will hold a very precious heritage of data for the whole 21st century and beyond, and without modern big data resources it's unlikely a project of this magnitude could actually be carried out.
