You've may have heard a lot about the concept of 'big data' in the last couple of years, probably in ways that have resulted in either figuratively scaring or boring you to death.
We've been subjected to endless, dry (we're talking, like, sandpaper-toilet-paper-dry here) business-speak on how it helps corporations gain a competitive advantage, pinpoint target markets and maximise consumer engagement. Eugh.
If it's not that, it's terrifying chatter about how your friendly neighbourhood government (wilfully assisted by those lovely "who us?" tech companies) is sweeping up your digital footprint to construct a yellow brick road leading directly to the end of liberty.
And what about that moniker? With a title as un-engaging as big data it's little wonder most folks are utterly turned off by the concept.
Despite what we've been subjected to, there is a human face to big data that is benefiting us in untold ways.
In short, as a society, we're creating and being exposed to more data than ever before. In two days, we create more than 5 exabyes of data. That's more than we did in total from dawn of time right up until 2003, according to Google's Eric Schmidt. Likewise, on the first day of a baby's life they are exposed to 70,000 times the amount of information that exists in the library of US Congress.
Author Rick Smolan, who wrote the book The Human Face of Big Data, points out, "our smart devices are turning us into human sensors that leave a trail of digital exhaust."
Big Data is the acknowledgement of the power this data has to change the world.
In many cases it can refer simply to new forms data of we hadn't been able to quantify or access before. In others, it represents the powerful computing systems and software that can harness and cross-reference significantly larger amounts of data and data points, resulting in a combined output that offers a far greater insight than each piece of data in isolation.
In our own lives, we can look at the impact of wearable fitness trackers and apps. Instead of 'manually' looking at your sleeping patterns, your resting heart rate, your exercise schedule and more, these digital devices look at everything together and the whole quickly becomes greater than the sum of its parts. The availability of this data offers unprecedented insight for real people into their health and fitness. In essence, it gives them more control over their own destiny.
More than just big bad data
Yes big data has its dark and snoozy side, but there also plenty of faces that are making your life better. This is the part you should be excited about.
Take Spotify for example. It "strives to be entirely data driven" making it one of the most outwardly data-centric consumer tech companies out there.
Every time you start a song, skip a song, subscribe to a playlist or start a radio station, Spotify knows and is using this to refine your experience. The direct benefits are greatly improved artist and playlist suggestions, and smile-raising songs appearing on your custom radio station. Give us that over suffering through myriad radio stations any day of the week.
"We are a company full of ambitious, highly intelligent, and highly opinionated people and yet as often as possible decisions are made using data," Spotify says. "Don't have enough data? Get more."
Video streaming giant Netflix operates under similar principles. There are few companies who know more about its customers. It once boasted, "there are 33 million different versions of Netflix," that's one for every subscriber (or rather 44m, since that's now the number of subscribers). Doesn't that make you feel special?
"Almost everything we do is a recommendation. Recommendation is huge, and our search feature is what people do when we're not able to show them what to watch," Netflix engineering director Xavier Amatriain once said.
Netflix declined to offer us more up-to-date stats, but two years ago it was analysing the habits of 25 million users and 30 million plays a day. It was gobbling up information on every time users pause, rewind, fast-forward, and stop. It also had access to four million ratings and three million searches a day. Given Netflix's growing influence and more refined methods of analysing data, those numbers will have grown significantly since 2012.
On a personal level it knows your age, sex and location (remember 'a/s/l?' from the AOL chat days?), your viewing history, the devices you use, when and where you watch, the content you share on social media and what your friends are watching.
It is from this and its huge army of engineers, mathematicians and statisticians that Netflix is able to create algorithms that lay on top of each other giving you those ultra-personalised "rows" of content it calls "alt-genres." You're also giving Netflix a better idea of which shows to keep in the archive and which to ditch. You're also informing the company's decision making process when commissioning new shows like House of Cards and Orange is the New Black. In that regard, it attempts to attract different demographics to those shows by presenting tailored trailers for different sectors of the audience. It's less about why it's universally good and more about why you would like it.
Facebook is another example of how big data can improve the experience for the average user. When the company introduced Trending Topics in January, many rolled their eyes and accused the social network of simply aping Twitter. However, it quickly became apparent that these trends were far more useful to the average user than Twitter's, which are based on two rather facile pieces of information; location and frequency.
Facebook's version also harnesses popular conversations, but those stories you see in the top right corner are also very much personalised. Thanks to what it knows about you, your posts, your location, the music you've listened to via your connected Spotify account, the episodes watched on Netflix and the articles you've commented on through Facebook Connect. Combining this information with your friends, your likes, your gender and your age, it is able to accurately curate your own personal news digest.
So, instead of seeing #1DNiallDay in trends, you may actually see stuff you're interested in. Speaking personally, I currently see stories about the NBA, Major League Baseball, Marilyn Manson, Sons of Anarchy, Reading Rainbow, Dave Chappelle, Edward Snowden and Robert De Niro. It's almost as if they know me...
Personal shopper
Big data has also been penetrating our shopping habits for years. Tesco's Clubcard programme, which has existed for two decades, did a lot to pioneer that by collecting information on everything we purchase to provide suitable coupons, discounts and rewards in exchange for that knowledge.
Apps like Groupon and Foursquare have built on that by offering personalised offers and coupons based on your location and preferences, while Amazon keeps a very close watch on what you're buying and what you're browsing and tailors ads and communications accordingly.
The next level could be Apple's iBeacon technology. These small, low-frequency Bluetooth receivers located in brick and mortar stores can offer shoppers customised discounts and promotions, via smartphone push notifications, depending on your precise location within a store.
These offers could be based on how many times you've visited the store and looked at the product, the other stores you've visited that day, your previous purchasing habits and items your spouse may have taken a shine to. Of course it requires your co-operation and willingness to participate, but it could revolutionise window shopping.
Indeed, amid all of the fears over whether we're feeding too much personal information into the system, it's easy to overlook what we're getting out.
Take the Waze navigation application for iPhone and Android, for example. The Google-owned tool offers real-time, crowd-sourced information on traffic problems, accidents, speed cameras, police traps and petrol prices. Drivers are contributing to this giant data cloud, but those users are also experiencing unprecedented benefits.
In that sense Waze is an excellent metaphor for the dawning of this new, as yet largely unregulated, big data awakening. The benefits and drawbacks are there for all to see, but this ever-growing data set is enriching experiences across practically every facet of our digital lives.
