In part one of this two-part entry I pointed out that I am not a great fan of conspiracy theories. Hint: you may want to read part one before reading this. Trust me, you need to read it. Otherwise you may worry about people deliberately doing very bad things, when this post is all about people doing very bad things by accident. Unintended bad things, according to modern moral standards, are nowhere near as bad as intentional bad things, if you know what I mean. So you need to appreciate why I worry about well-intentioned goofs.
As I said before, I think conspiracy theories are daft. However, I do think people sometimes do things, sometimes as large groups, with unintended but important consequences. Those consequences may then by exploited by others in ways that had not been imagined at the outset. So Albert Einstein did not intend to revive the flagging career of Keifer Sutherland, who at one low point considered joining the rodeo. But without the work of Einstein, Niels Bohr and the rest, Sutherland would be doing something useful like roping cows instead of pretending to be agent Jack Bauer looking for terrorist nukes in the tv show “24″.
So Microsoft is now using us all to gather data, in order to test its software. To do this, it has effectively developed a new paradigm for experimentation and gathering of results. Customers are effectively expected to test software – even if it goes beyond the beta stage – and are asked to permit automated feedback whenever something adverse happens. But in doing this, customers are not just helping Microsoft test software. They are also engaged in an altogether different kind of experiment, though the method of gathering data is equally sophisticated. In this experiment people do not just provide the data, they are its subject.
Science Fiction writer Isaac Asimov came up with an interesting idea in his Foundation stories. The essential premise was that a scientist, Hari Seldon, would conspire to manipulate the future, although this conspiracy was positive in that it was in the best interests of the human race. Seldon could do this using his scientific technique of “psychohistory” which was defined in the story as
“…that branch of mathematics which deals with the reactions of human conglomerates to fixed social and economic stimuli…
…Implicit in all these definitions is the assumption that the human conglomerate being dealt with is sufficiently large for valid statistical treatment…”
In other words, Seldon devises a statistical science which predicts the behaviour of very large groups of people in a way that cannot be applied to individuals. Of course, for Seldon to have generated a science of prediction, he must have amassed sufficient data to use as a basis for his calculations.
Okay, so that is science fiction. But you get my point. Right now, in a very crude way, many are trying to accumulate data about us. Some are consciously doing it with the intention of learning about our behaviour; others are doing it for other reasons. But just as Einstein’s theories started a chain of events that brings us to our current fears about the proliferation of nuclear weapons, so the mass of data accumulated about people invites research into predicting our behaviour. Such predictions may be for financial gain, or like Asimov’s fictional conspiracy for the good of all humanity, or (like the popular conception of conspiracy theories) for the good of the few at the expense of the many.
This generation is the beta generation. For the first time in history, people are able to come up with theories about the mass of human behaviour, and then actually test them out. The most basic example is to be found in marketing. To market successfully, you need to understand the market. And use that understanding to influence it. So marketing will be at the forefront of the experimentation (take a look at the depth of analysis here if you do not believe me). Traditional methods of gathering data, through questionnaires and surveys, will be increasingly replaced by the power of enforced gathering of data at the point of sale. The supermarkets have the power to track purchases and build up an understanding over time. Personal finance companies will have more and more data about someone’s credit history and spending habits. And on-line purchases will demand purchasers answer questions for the sole purpose of marketing.
Of course, you may assume that you are protected by some legal voodoo from anybody unreasonably analysing your data. If you are in the EU, forget about your rights under data protection legislation – for laws to work they have to be enforced by someone. And, thanks to global terrorism, governments have finally solved the problem of how much data is too much data for a company to keep. So, after wasting some taxpayers money debating its data protection directive back in the late 90′s, the EU has since stepped up to the mark and reversed some of that by passing the data retention directive. For those of you not familiar, the basic idea of the data retention directive is that telcos must keep lots and lots of data about who made what call to whom, what was said in emails, that kind of thing, just in case governments need it to spy on their people…ahem, I mean protect good people from bad people. So, after some worrying in the 90′s that companies might be tempted to keep too much data, now governments are bending over backwards to persuade or force (whichever works faster) those same companies to keep ever more. They are even going to give the telcos taxpayer’s money – though nobody is sure how much yet – to keep and store all that data, just in case it turns out handy for fighting terrorism, or serious crimes, or important things like tax evasion. You know, important things. Of course, nobody in the telcos would ever misuse that data – no no no no no no no no. The fact that a few years earlier the same Eurocrats were demanding increased powers over telcos to ensure that telcos did not hold on to unnecessary data is irrelevant. A few years ago, if telcos had data they could not be trusted, and governments had to protect the people from the evil that telcos would do with all that data. Now we have global terror. So telcos can be trusted with all that data. Obvious, really. Of course, if you are outside of the EU this is all irrelevant as you never had any rights anyhow.
If you are inclined towards conspiracy theories, then this should all be troubling. One relatively benign example of how technology increases the power for monitoring human behaviour is the use of virtual communities like Second Life for research. But it can get much cleverer than crude questionnaire-based methods of researching behaviour. As well as training computers to read number plates, in order to charge drivers for using congested areas, computers can be trained to read the actual behaviour of people. Telcos have used simple neural networks to try and spot for fraudsters for a long time. The basics for this kind of approach were always in place – high degree of automation, very large volumes of data representing behaviour all in a standard format, and enough financial incentive to make it worth investing in developing the technology. But sophisticated analysis of data and even neural networks are being used for far wider automated predictions of human behaviour – just see this list of abstracts from an academic conference dedicated to fighting crime.
The recent Vodafone-Yahoo advertising deal gives an example of the increasing opportunities for assimilating data about people. Supermarkets know what you buy, but not much else about you. But suppose you know what someone buys (because you are monitoring on-line purchases) AND their movements (because of their mobile phone) AND who their friends are (because you know which numbers they call, who they email, or their activity on social networks). You know an awful lot about that person. And the potential uses are extensive. So, imagine someone appears to be a “net promoter” of a particular product, in that they would promote it to their friends. If you know Rachel is a net promoter, why wait for her to recommend the product to friends? Why not just target an offer to Rachel’s friends straight away, mentioning how happy Rachel is with her purchase. Now this could backfire – perhaps the friends will be none too happy with Rachel. But then again, if you give Rachel a credit for every friend that buys, and each friend gets an exclusive discount too, then maybe everybody feels like a winner.
Anyhow, I do not work in marketing, so maybe my scenarios are daft or maybe they make sense. But the important thing is that it will, in theory, not be necessary to speculate any more. You can try out different marketing models and steadily learn which are the ones that work best. So it seems we are destined to be increasingly manipulated, or better understood, depending on how you look at it. But my bet is that the future will not work out that way. Not because they will not try to sift through the data and test us all. But for a few very important reasons that none will care to admit.
First, the data that will be gathered will often be garbage, and chances are very few will realise how bad it is. If you read part one of this two-part entry, you got my point of view on how clever people are. They think they are cleverer than they really are. So in practice, they make mistakes, especially when there is no straightforward feedback to highlight those mistakes. Poor data is very hard to identify, if you have no source of information that highlights poor data, and no simple way of testing it. Let me give you some examples of why a lot of data will be garbage.
This week I booked a flight with Easyjet. Rather annoyingly, the jolly fat Greek’s airline portal forced me to click a box stating where I was staying at my destination. I was given no choice – either I clicked one of the options or I would not be able to book on-line (so would have had to pay more to book over the phone or else fly with another airline). The point of gathering the data is clear – to understand the kind of customer I was and the potential to sell accommodation. So I did the only sensible thing. Which was to give a dishonest answer. But instead of picking an answer at random, I made sure I picked the answer that was most misleading. Instead of clicking the box saying I was going on business and staying in a hotel, I clicked the box saying I was visiting friends and staying at their home. That way I also minimised the risk that the jolly fat Greek will spend money on yet more unwanted advertising aimed at me.
Of course, it did not stop there. Next thing I did was book my car park space at the airport on-line. This required me to explain how I knew the car park existed. So I lied again. This time I rationalised that the best lie would be say it had been a recommendation by friends.
Even supermarkets cannot totally rely on their data about who buys what. A letter printed in the Financial Times last week tells a funny story about a customer shopping at Tescos without having one of those ubiquitous loyalty cards. What does the checkout person do? The checkout person generously offers the points – and unrelated data – to the next customer in the queue, who gladly accepts.
But not all data will be garbage. If you pay for something, hopefully the record of what is bought is right. When mobile phone companies record your location, chances are they will be right. One goal of enhanced 911 in the US is to pinpoint cellphones to within 50-300 metres. This may be a boon for emergency services, but it is also a potential boon for tracking the location of people. So if the data is correct, then the next problem becomes how to use the data to make predictions.
Making predictions is in our nature. Whether it be the biblical interpretation of dreams by Joseph, the rhymes of Nostradamus, the stories of hard science fiction writers like Asimov or Clarke or the predictions of professional futurologists like BT’s Ian Pearson (yes, Ian Pearson is for real – though I agree he is more like a spoof), there seems to be an eternal and unquenchable desire for prediction. So the question will be how predictable we are. My hope is that we are like the climate, where the facts can be so readily disputed by experts like Danish scientist Bjorn Lomborg. Many predictions down the ages turned out to be very wrong, and not just by weather forecasters. The aforementioned Isaac Asimov was known as a “hard” science fiction writer because he was a proper scientist who tried to extrapolate from proper science, but he predicted we would have invented positronic brains for walking, talking robots, and populated 50 planets before we would have managed artificial insemination – so he was just a little wrong there. And in Asimov’s Foundation, the character Seldon’s main goal was just to compile a source of all human knowledge – the Encyclopaedia Galactica – when had Asimov predicted the rise of the internet he would have realised Wikipedia is going to get there first :)
But the power to change is not dependent on making accurate predictions. Al Gore making a film about the environment is more influential than Bjorn Lomborg writing a book. Arguably Futurama making a cartoon about global warming is more influential than proper academic research. Even father-and-son Rupert and James Murdoch take global warming seriously, screening Al Gore’s film (and getting Al to come along and talk too) to News Corp execs. [The fact that the Murdochs could make Attila the Hun seem like a hippy should make it very tough for the beardy shagger Richard Branson - who finds himself outflanked by the Murdochs both commercially and ethically]. The Y2K bug may have turned into an anti-climax, with no planes falling out of the sky and everyone’s elevators still working fine, but it still motivated enormous change. Karl Marx may have inspired many changes around the world, but pretty much everyone now has given up on the worldwide communist revolution that Marx argued was inevitable. Thomas Malthus was wrong that population growth would outstrip food supply, but his name is still known because so many thought it credible. The “new paradigm” of dotcom boom evaporated along with a lot of paper profits. Confident predictions of WMD in Iraq were shown to be unjustified. So enormous changes can be inspired by bad predictions. And predictions will be accepted as fact if argued for persuasively by people in authority, even if their reasoning or data is flawed. That is the bit that really scares me. Responding to accurate predictions makes sense even if it is creepy, but responding to invalid predictions that may be harum scarum nonsense, generated by self-serving experts who only make guesses so that they can generate rewards for themselves. Disraeli had good reason to warn about “lies, damn lies and statistics.” With all this data being accumulated, the potential for bad, but persuasive statistics is great. A prediction does not have to be right to have profound consequences. The current gathering of lots data may motivate people to believe that humans are more intelligent than they really are. It is tempting to think that a poor conclusion is right, so long as it is supported by lots of data. With the current beta testing of theories being performed across the whole human race, premature conclusions may be hard to argue against. Worse still, if the data is available only to the elite, like the world’s governments, how will the rest of us be able to determine when their opinions are reasonable, and when not?