I saw one industry association – I won’t name names but smart people will correctly guess who I am talking about – that recently pushed a ‘scary number’ to all of its members, based on a survey conducted by another firm. But the association forgot to mention that the scary number completely contradicted the results of their own telco benchmarking exercise!!!

Your article illuminated a serious telecom and political problem. I found the New York Times story particularly alarming.

I’ve long known that the politicians of my United States are skilled at lying with statistics. But even if you can’t find a clever way to present your statistics, isn’t it nice that politicians can employ the fallback tactic of magnifying the costs or benefits?

When I was in college back in the 1970s, we heard a lot of talk about the “military/industrial complex”. And the effort to correct that problem led, in part, to the rise of the regulatory czar.

But forty years later, the military/industrial complex never really went away. In fact, we have added an energy/environmental/government complex and the banking/government complex. The sad result is that referees today are no longer just regulating. The effect of their policies is to penalize some teams in support of their opponents.

General Motors can now compete well with Chrysler, Ford and Toyota because the regulator contributed several hundred billion dollars to GM. Alas, even the best intentions of our wise politicians sometimes goes for naught. The heavily subsidized hybrid car, GM’s Chevy Volt, is no longer being produced because not enough people want to pay $40,000 for it.

Telecoms have never been as politically connected as the banks, automakers, and green energy producers. Witness the recent slap in the face that AT&T got when it tried to merge with T-Mobile USA. Now a smart company like AT&T doesn’t make a huge bid to acquire another mobile operator unless it knows its back is covered by the regulators. So apparently, AT&T got bad intelligence or was double crossed.

Still, I need to temper my criticism of regulators somewhat, because sadly, my wife and I were once ruthless regulators.

You see, we have 5 cats in our house, and in their early years of growing up, we regulated our cats severely by not allowing them to roam outside the house. And why did we do this? Because we couldn’t bear the thought of seeing one of them killed by a car running over them on the street.

But notice something. We had put value on our cats\’ <u><b>existence</b></u>, not on their everyday <u><b>living</b></u>. The cats themselves were begging and pleading to be set free to roam in the outdoors, to catch mice, climb trees, and live the natural life of a cat in the wilderness.

Fortunately, we live in a rural area with plenty of woods behind the house, so we eventually did set our cats free, installing a cat door on the house so they could go in and out anytime. Now it all went fine till the day I awoke to find a raccoon in our kitchen – luckily, an Auto-ID cat collar solved even that issue!

My point is by allowing our cats to take some risks, it enriched their lives immeasurably. Yes, they’ve had to compete for territory with other cats in the neighborhood, and a couple times they came home injured from cat fights, but on the whole, they learned to survive and prosper once my wife and I gave up the regulatory game and opened the door for them.

Rather than arguing theoretically about whether some telecoms operators may do wrong, I would much rather that Malawi’s regulator shows some hard evidence to justify the cost of implementing these systems. You mention that you have evidence of wrong-doing and dishonesty by operators. That I can believe. However, let me assure you that I have plenty of evidence of wrong-doing and dishonesty by regulators. If Malawi’s regulator cannot furnish any evidence of the operators’ fraud, then why not? A cheap and simple comparison between interconnect bills and reported retail transactions would give a useful indication of under-reporting, and there are other simple checks that can be performed. After all, the company’s accounts, and hence its profits, are audited… this audit should also be able to pick up evidence of discrepancies between record-keeping and cash collected, so it would make more sense to focus efforts on this area first, before jumping to the conclusion that automated verification is needed. After all, if the transactions are under-reported, then profits will likely be under-reported too, hence affecting the amount of corporation tax taken. If Malawi’s regulator can show evidence of fraud by operators, then it should do so. If it cannot, then it should not assume the guilt of the operators, nor smear them by implying they are dishonest. Regulators should justify their actions before they impose burdens upon business, and even more so in this case, given the threat posed to Malawi’s citizens. I also consider myself an expert, and I share the privacy concerns voiced by many Malawians. Or would you rather suppose that all government agencies always act in the best interests of their countries and of their people, and that none ever acts cynically to serve its own interest?

The focus on Root Causes is certainly something I have not heard other software vendors talking about – though “root cause” analysis is commonplace in the service assurance and cybersecurity camps.

I am interested to hear comments and follow-on questions from the experts reading this forum.

Dan, you have raised very intelligent questions and I shall try to answer them. However just before I start, I wanted to clarify – “first time ever”. I still stand by this, since I have yet to see or hear from any operator globally in the RA space tell me (or mention in passing in any site) about how their RA system helps by directly presenting them the Root Cause. Subex didn’t invent machine intelligence (and if you ever hear me claiming this, please feel free to beat me on the head with a Sunfire X series server), but we are certainly the first to bring it to RA for the purpose of Root Cause detection. Of course, this is a very bombastic claim – but I am, as always, open to correction.

“How does Zen make use of meta-data? Or put in another way: how does it interact with the human RA experts who are teaching the machines what is relevant?”

This line tells me that you have in fact clearly understood exactly what I was attempting to explain in my blog, since you’ve hit the crux of the matter. Now I will attempt to answer your query (at a generic level, since the actual algorithms cannot be disclosed).

Zen works at two levels -

a) Operational data (eg. CDR)
b) Case Management (i.e. system internal meta-data for analyst interaction)

Operational data is for Zen to perform data pattern analysis on a minimum of two data sets (eg. MSC and Mediation output). It involves automated field level (including the values in the fields) data gathering, classification and presentment. This creates a list of potential root causes (lets call them PRC for simplicity).

Case Management data is for Zen to perform a recursive, weighted analysis of assigning a “Reason” to each PRC. This is performed by the system “looking and learning” from what an analyst has noted in the past. The case management functionality of the ROC RA solution provides a single consolidated workflow environment where all the information pertaining to a detected discrepancy is captured, eg.:

a) What data sources were involved
b) What is the discrepancy level
c) What is the threshold as per the defined KPI
d) CDR pertaining to this audit
e) Auto-assignment to analyst
f) Communication and escalation paths
g) etc.

Now, in each new “case”, the analyst has various fields to fill in based on their investigation. We call a set of such steps as a “Workstep”. At the end of each workstep, the case gets assigned to the next team in that particular workflow (eg. Network team, mediation team etc.).

As you can probably surmise, the system captures a significant amount of data (both CDR and Meta-data) in each Case docket.

This information is fed through the Zen platform on a daily basis for it to keep updating the associated “Reason” database. The platform also analyses the “Acknowledgement” trend in inter-workstep communications to ascertain what is the “Positive Hit-Rate” (or PHR) in investigations. Only those reasons with a high “PHR” are carried forward to the next stage (there is an algorithm which scores all the reasons and keeps updating reason scores periodically based on the PHR updates).

Now, the drawbacks to this approach are:

a) The system is not accurate from day 1. Building the Reason and PRC database will take time. The PRC can be generated from Day 1, since it is an engine which is based on pattern analysis only. But associating it with Reasons will take time as it is directly dependent on analyst inputs which need to be built over a period of time.

b) If the operator does not believe in a strong RA process and does not use the case management functionality, Zen will only be able to deliver Potential Root Causes (PRC) without any level of associated Reasons. This is a situation where only half the platform is being used, and is not the recommended approach.

Luckily for us, in the case of ROC RA, we aren’t necessarily in the same boat as a Google search. Throughout the application, we religiously link all the components of an audit (in a tightly coupled/bi-directional linked manner), right from the raw files all the way to dashboards and mail communication regarding the audit. As a result, the meta-data management is quite structured and the curse of free-text, unstructured information is mitigated to a large extent.

Dan – I believe I’ve ranted for long enough, but does this help you gain a deeper understanding of the Zen platform?

Though Hakan was perhaps a bit harsh, talkRA is no stranger to highly opinionated columns on RA, real-life Big Foots, and all subjects in between. Sir Eric has trained us well: bark often and loudly to scare away imposters.

Any vendor who tries to pass a fast one over talkRA readers has another thing coming. And to his credit, Ashwin is bloody but unbowed.

I found it interesting that this post was closely followed by an interview with Lionel Griache on his open source RA initiative. Both stories are about the value of RA software, and I think both parties have more educating to do to please the many skeptics out there – I being one of them.

First off – Lionel, we congratulate you for your achievement. Now you face the daunting challenge of building a user community, first class documentation, consulting support, training, and marketing around your new tool. Those challenges certainly put the advantages in the court of existing RA software vendors who have many years’ experience addressing these concerns.

Ashwin, thank you for your fine introduction to Zen and for fielding some tough questions here. Unfortunately, you’re not out of the woods yet, my good colleague :- )

What raised my eyebrows a bit was the sentence: “Zen is an automated Root Cause Advisory engine which provides, <b>for the first time ever</b>, machine intelligence for pattern identification and presentment.” [my emphasis added]

Data warehousing and machine intelligence has been around an awfully long time. Back in 1994, I authored a research report on Telecom Data Warehousing, so the claim to have a “first time ever” technology sounds like a stretch. It may very well be true, but you haven’t yet made the case to justify that broad claim – though I forgive your exuberance because I know you’re proud of the work you and your fellow Subexians put into Zen.

As luck would have it, a week before I ran across your column today, I found a relevant blog article by Jeffrey Phillips, an expert who has written a few books on the subject of business innovation. His article is appropriately named, “<a href=http://workingsmarter.typepad.com/my_weblog/information_management/\">Finding Needles in Haystacks</a>”.

Phillips makes the case that meta-data management is the secret sauce for the next breakthrough in analytics. He argues that the average business must catalog information at the rate of 5 Megabytes a day. For a 50-person firm, that translates to more about 1 Gigabyte of information a month. Not only is the rate of information to be searched growing fast, but we lack a consistent way of classifying that information. So we quickly run into the common situation where a powerful a search engine like Google often can’t even get close to the specific information we need to know.

So my question to you, Ashwin, is: how does Zen make use of meta-data? Or put in another way: how does it interact with the human RA experts who are teaching the machines what is relevant?

http://workingsmarter.typepad.com/my_weblog/information_management/

Though Hakan was perhaps a bit harsh, talkRA is no stranger to highly opinionated columns on RA, real-life Big Foots, and all subjects in between. Sir Eric has trained us well: bark often and loudly to scare away imposters.

Any vendor who tries to pass a fast one over talkRA readers has another thing coming. And to his credit, Ashwin is bloody but unbowed.

I found it interesting that this post was closely followed by an interview with Lionel Griache on his open source RA initiative. Both stories are about the value of RA software, and I think both parties have more educating to do to please the many skeptics out there – I being one of them.

First off – Lionel, we congratulate you for your achievement. Now you face the daunting challenge of building a user community, first class documentation, consulting support, training, and marketing around your new tool. Those challenges certainly put the advantages in the court of existing RA software vendors who have many years’ experience addressing these concerns.

Ashwin, thank you for your fine introduction to Zen and for fielding some tough questions here. Unfortunately, you’re not out of the woods yet, my good colleague :- )

What raised my eyebrows a bit was the sentence: “Zen is an automated Root Cause Advisory engine which provides, for the first time ever, machine intelligence for pattern identification and presentment.” [my emphasis added]

Data warehousing and machine intelligence has been around an awfully long time. Back in 1994, I authored a research report on Telecom Data Warehousing, so the claim to have a “first time ever” technology sounds like a stretch. It may very well be true, but you haven’t yet made the case to justify that broad claim – though I forgive your exuberance because I know you’re proud of the work you and your fellow Subexians put into Zen.

As luck would have it, a week before I ran across your column today, I found a relevant blog article by Jeffrey Phillips, an expert who has written a few books on the subject of business innovation. His article is appropriately named, “Finding Needles in Haystacks”.

Phillips makes the case that meta-data management is the secret sauce for the next breakthrough in analytics. He argues that the average business must catalog information at the rate of 5 Megabytes a day. For a 50-person firm, that translates to more about 1 Gigabyte of information a month. Not only is the rate of information to be searched growing fast, but we lack a consistent way of classifying that information. So we quickly run into the common situation where a powerful a search engine like Google often can’t even get close to the specific information we need to know.

So my question to you, Ashwin, is: how does Zen make use of meta-data? Or put in another way: how does it interact with the human RA experts who are teaching the machines what is relevant?

http://workingsmarter.typepad.com/my_weblog/information_management/