Archive for January, 2012

UK mobile operator O2 found themselves in trouble last week, when it was discovered they automatically sent the user’s telephone number to any website they visited. Any. The problem is fixed now, but the error that caused the privacy violation was created on January 10th and remained uncorrected until January 25th, according to O2’s blog. Whilst O2 were quick to resolve the issue once they had become aware of it, the most important lesson learned (we can only hope it is learned this time) is that customers have become adept at spotting the mistakes not caught by the operator’s formal testing. Lewis Peckover, a system administrator at a mobile gaming company, not only found the problem, but helpfully shared it with the world, writing a script so anyone could check if their O2 phone was broadcasting their number. You can see his script page here.

A knee-jerk reaction might be to demand more controls, more testing, more whatever. Without knowing how O2 works, I would not like to comment. Mistakes happen and will keep on happening; a demand for more and more controls can only lead us in ever decreasing circles. My reaction to this incident is quite different. First, O2 reacted quickly. That is important. Agility demonstrates concern and limits risk. Second, they were transparent. They did not keep quiet and hope the problem would disappear. Instead, they were proactive in talking to customers, press and regulators. This also limits the eventual damage done. Third, all telcos could benefit from making it easy for customers to give them information, and being responsive when acting upon it. It was easy for Lewis Peckover to tell the whole world about O2’s bug. O2 got the message, but they got it via a public route. If the same information had been passed to them privately, and if they responded as rapidly, no more harm would have been done to customers, but less harm would have been done to O2’s reputation. Customers can and will keep finding flaws, and the internet makes it easy for them to share them publicly. As with Facebook’s bug bounty, telcos can get error-spotting customers to work for the telco, not against the telco. Trying to avoid mistakes is obviously important and deserves the investment of time and effort. However, a tiny investment in helping customers to help the business means the issues can be resolved quickly and quietly – thus generating a positive return for the firm.

Bookmark and Share

5 Whys is perhaps the best known technique for finding the root cause of problems. Developed in Japan and pioneered in Toyota, it belongs with quality management and Kaizen as one of the factors behind the high-precision, low-error Japanese manufacturing revolution. Part of its genius is that the method itself is so simple. Start with a problem. Ask why you have the problem. Whatever the answer, ask why again – why are things like that, why is that answer true, why is the business like that? Keep repeating the question every time you get an answer, until you have forced yourself to get to the very bottom, the essential root cause that underlies the problem and where it becomes impossible to ask why again. To get to the bottom may take more than 5 whys; it just happens that 5 is a reasonable guess at how many it will take. So to give a simple illustration of how 5 Whys would apply to RA, imagine a bill is in error. Why? Because a CDR is missing. Why? Because the switch was not working as expected. Why? It got overloaded. Why? Because actual call volumes were above the design spec. Why? You get the idea by now – just keep asking why until there is nothing left to learn.

Simple, huh? Well, getting the answers might be a lot of hard work. But the point is right – RA should keep asking why until it finds the real root causes of the leakages. And if not RA, then who should ask why? The answer to that is plain: if it is not RA’s job to uncover and address these root causes, then it is nobody’s job to uncover and address these root causes. RA is unusual in having a job spec that gives it the freedom to research, find, and drive the response to root causes which lead to leakage, wherever and whatever those root causes are.

But does every RA function habitually ask the 5 Whys? No. Why? Because they can get results more quickly without asking the 5 Whys. Why? Because drilling to the root cause may be time-consuming and RA can get credit for just treating the symptoms. Why? Because finding the root cause often involves more than analysing data that RA already has (or wants) and RA can choose to measure the benefits they add by just treating symptoms. Why? Because addressing symptoms does deliver nominal benefits that can keep their bosses happy and nobody is pushing RA to measure how well they dealt with root causes. Why? Because executive management is comfortable with just addressing symptoms and RA is comfortable with satisfying those more limited expectations of executive management. Why? Because nobody in RA or the executive team is motivated to change the business more fundamentally.

Have I made my point? I think so; RA can and should deal with fundamentals and lead the discovery of root causes in order to drive fundamental change. Why? Because I believe that if RA does this, then businesses will be better… and that is the root cause of this blog.

Bookmark and Share

I have always been preoccupied by bias, but last year I spent a lot more time trying to understand what drives and enables it. Bias is important – no amount of data leads to a good decision if either the data or the decision-making logic is biased. If you appreciate that, you understand why bias is the greatest opponent to robust risk management, and why what seems like good risk management may be utterly flawed. The financial collapse is an excellent and relevant example. Bias is insidious; it often works beyond the limits of our comprehension. Its subtleties lie so deep they may be invisible even to scientists, the group in society that struggles to eliminate bias more than any other. A recent article in The New Yorker beautifully explains how endemic bias plagues the scientific community, even though scientists are not conscious of it. You can read it here. If we can understand why science fails to eliminate bias, leading to expensive and mistaken decisions, we can better understand how all-pervasive bias is a constant challenge to risk management.

Bookmark and Share

In October, the Managing Director of Teleonto, the fringe Indian RA vendor, was arrested by police for allegedly siphoning INR18.6M (USD340k) of investor funds; see here. The CEO is also reportedly wanted by police. Teleonto’s website has seemingly disappeared, but their corporate LinkedIn profile is still visible here.

Newsgopher holds his paws up and admits this story initially slipped by him. There are dangers when dealing with unproven providers, because they may make grossly inflated claims about their business. Teleonto were always something of a mystery, despite the fact they called themselves “innovation leaders” in the field of revenue assurance. For several years they occasionally obtained publicity for supposed product developments and for their fund-raising efforts, but their history appears to be that of a venture which entered the market too late and never developed a viable offering. A year ago the CEO gave an interview where he stated that the company had focused its strategy on providing RA SaaS; see here. This scandal highlights the need for care in determining who has access to sensitive RA data. It also highlights the importance of thoroughly checking the viability of suppliers, especially now that the RA gold rush has come to an end and unprofitable suppliers find themselves unable to secure investment to keep themselves going.

Full disclosure: Newsgopher has a Teleonto-branded mug at the back of his cupboard. They were handed out free at an RA conference several years ago ;)

Bookmark and Share

The astonishing news from China is that 10% of all Chinese web users had their passwords compromised over the Christmas period. Hackers stole data on the 40M users of Tianya chat site, and separately on the 6M users of the CSDN programmer’s forum. In both cases, the user details were stored in plain text. You can read more here.

Bookmark and Share