However, let's go back to the question of how Twitter mines its data, and what deeper insights can be gained from that data. Back in 2008, I worked with one member of the Twitter development team and produced the first public trending tool, which went live as @secretbear on Twitter (and gained a total of exactly 15 followers!). It ran for a couple of years before I stopped updating it.
The biggest challenge is sorting the wheat from the chaff.
Picking the same time range as I did when I was looking at Twitter's worldwide trends above, my newly reactivated Twitter trending tool shows (amongst a few others):
- Mompati Merafhe
- Scotland Yard
- Commonwealth Games
- Australian Ethical Investment
- Christina Perry
I feel a little more enlightened about the state of the world, but it goes to show that with gigabytes of data pouring through second by second, it can still be a little more like sticking a sewing needle in the ocean and hoping to spear a shark.
But the sheer volume of data is useful if you are looking for longer term trends.
Emotional analysis
One continuing area of research is around emotional analysis of the Twittersphere, i.e. can you draw conclusions about how the world is "feeling", and if so, can that be applied to other sets of data. And of course, can you make money from it?
In 2010, researchers at Indiana University studied almost 10 months of tweets, and subjected them to mood and sentiment analysis using a number of different tools. They then looked to see whether this could be correlated against changes in the Dow Jones on a daily basis. And in 87% of cases, the changes in mood over the day could be used to predict whether the market rose or fell at the end of the day.
On paper, this looks like the ultimate path to riches. Track Twitter, see if people are happy or sad, and then bet on whether the market will end up or down at the end of the day. Except of course, all of this analysis is done after the event. The researchers were able to place themselves in the past and say that had they known at the beginning of the day how people would be feeling over the course of the day, they could have predicted how the day would end. What they were not suggesting is that how people felt yesterday would affect how the stock market would perform the next day.
But there is light at the end of this particular tunnel, and it comes from an unusual place: Gamers.
Parallel power
Usually associated with darkness and acne cream, gamers have demanded one thing from their computers: speed. They need ultra-realistic graphics rendered smoothly for a fully immersive gaming experience. Traditional CPU technology is not fast enough to cope with this, and has one other major drawback – it processes each instruction one at a time, (i.e. it is a serial process). When you need all 24 million+ pixels to appear on your screen at roughly the same time, plotting their movement one instruction at a time gives you lag. And lag gets you killed (virtually, at least).
To get over this, manufacturers such as Nvidia produced graphics cards that could perform a reduced number of types of instructions (called RISC computing), but all at the same time, which is known as parallel processing. This type of power and parallel processing was previously the realm of supercomputers costing millions of pounds. Now for £700 (or dollars) you can get a supercomputer on your desktop.
This type of speed and power has led to the resurrection of one of the brave new technologies of the 80s, the neural network. A neural network is designed to emulate the learning patterns of the brain, by getting a computer to "learn" from data that is given to it. It fell out of fashion because while the mathematics was well understood, the processing power was just not available.
Stock answer
Today, by feeding millions of emotional scenarios into our neural network running on hardware that costs less than a second-hand car, the computer can learn the likely effects of different emotional states on the other patterns it is monitoring, such as the stock market. This way, rather than waiting until the end of the day to see if the stock market is going to go up or down, we can tell by the end of the next tweet that comes in. And this technology is available now (and the people in the know are keeping it to themselves, of course)
Twitter has changed communication for a whole generation, but it has also provided an incredibly rich seam of data for analysts to work on. I have described some in this article, but prepare to be amazed at the insights into human behaviour that 140 characters can give in the next few years.
- Nigel Cannings is CTO at Intelligent Voice.
