ISP Death By A Thousand Duck Bites
For many years much of Internet community has continued to point a finger at ISPs, claiming they’ve turned a blind eye towards the botnet problem. While sometimes their arguments seem well founded, often, those claiming ISPs are “feeding us dirty water” don’t fully understand the ISP’s predicament. I hope to shine a bit of light on some of the extraneous factors gating the service provider considerations space as it relates to network convergence, bots and the surrounding business aspects.
Most of the discussion here applies primarily to residential broadband and consumer access providers such as cable MSO and DSL providers, which is where the bulk of the ISP-bot discussions seem to be. Hybrid networks, more akin to those of enterprises, academic, and government organizations, typically have responsibility and authority over end systems, as well as infrastructure and connectivity services, and are a bit of a different animal.
Over the past decade all forms of Internet access have become a commodity, in particular, in the broadband services market. Broadband service providers compensate for low-margin services several ways. The most obvious is by expanding market share, or simply put, increasing the number of subscribers. This comes in two forms, either by additional cost for build-out and OPEX associated with expanding service offering footprints in new markets, or lowering cost to be more competitive in existing markets.
The second option is to move away from strictly low-margin service offerings and aim at increasing average revenue per user (ARPU). This is what all the hype surrounding triple play (i.e., data, video and voice) and quad play (triple play ++mobile) is about. If you can offer more services with only incremental infrastructure investment, and the services provide higher returns, you’ll find more opportunity for profitability. Easy enough.
Well, as you might suspect, this has lots of implications. One of the more obvious is that if service providers that were strictly providing DSL services start offering VoIP and IPTV services, they’re stepping into cable/MSO space. And, so to do the cable/MSO folks start offering data and voice services; expanding your market opportunities increased your competitive landscape considerably, even in your traditional markets.
Another implication is that of traditional services availability. If you start offering voice services over cable plant with converged IP packet services (e.g,. VoIP), your service offerings will be bounded by the similar requirements and regulatory constraints that the traditional services mandate. A simple example of this is emergency dialing services such as VoIP Enhanced 911.
One of the things that frustrates me when having discussions on this topic is that ISP customers whose hosts have been compromised are often referred to as malicious, or bad actors, or simply ‘infected hosts’. Well, while strictly speaking this is true, fundamentally, they’re victims, and most often, unwitting victims. The nefarious activities for which their systems are being employed are usually quite malicious and further violate the compromised systems owners.
Back to that ISP broadband services business model. I care primarily about two things; Subscriber churn and ARPU. Just subsequent to ARPU, or somewhere in the mix, is profitability. So, I don’t want to lose customers to competitors, and I want to increase revenue per user and profit margins. Makes sense…
Now, let’s consider an example service offering. I’m offering voice service, and Internet data services, and IPTV to my subscribers, all via a single local access loop, connected to a single subscriber aggregation device, with a good bit of new overlay services infrastructure required to enable the voice and video services.
For Internet data services my ARPU is ~US$25/month (fixed, unlimited, not usage based). I amortize capital expenditures (CAPEX) costs over, say, 60 months, factored all other operational expenditures (OPEX), and came up with a monthly per subscriber cost of goods sold (COGS) of ~US$20/month. I calculate gross margins: ARPU – COGS, and find profit per subscriber to be in the range of $5/month — with a fully loaded system. IPTV and voice services provide for higher margins, but further increase focus on churn and services availability.
One of the largest costs associated with OPEX for broadband services providers is that of customer support, or help desk services. Costs associated with maintaining a customer support organization are considerable. They include the employees themselves, salaried or hourly, benefits, facilities, etc.. A reasonable estimate would be an average of US$100/hour per customer support representative.
*This model and all the numbers provided here are purely for explanatory purposes, as all factors vary widely.
99% of the time, when a broadband subscriber is infected with malware (and becomes aware), or “the Internet breaks”, or they have connection or download problems, or their car won’t start, they call their ISPs customer support group. And, 99% of the time, unless the actual connectivity service is broken, whatever it is their problem might be, it won’t be fixed by the help desk. But figure a reasonable estimate of time to deal for customer interaction with the help desk is 30 minutes per call – or US$50.
Let’s stop here for a second. If each customer call cost averages $50 and my gross margin per subscriber is $5, I just blew 10 months worth profitability by answering the phone. Regardless, let’s continue. Now, recall that the ISP doesn’t own the subscribers PCs, and more than likely, most of the folks in the customer support group couldn’t find a root kit, much less tell a customer how to remove one – especially when they’re anti-virus or spyware detection tools can’t do the job. So, 99% of the times, if it’s not a connectivity services problem they:
- refer them to their OS vendor for patches, which often provide little or no help
- refer them to AV or spyware detection companies
- refer them to a consumer technical support services organization like Geek Squad
- or help them realize they might be better off simply getting a new PC, as it’s cheaper and easier
More and more now providers are beginning to offer these enhanced technical support services themselves, such as AT&T Support+ Service Packages. Note that these services aren’t cheap, and justifiably so, given the skill required and the breadth and depth of issues they deal with.
So, when events like the FBI’s Operation: Bot Roast cause a thousand ducks to come a quack’n, this disrupts the factored OPEX as call center volumes increase by orders of magnitude or more, with considerable financial implications for service providers. The right answer is for ISPs to put solutions in place that enable automatic quarantine and subscriber management for compromised hosts. However, it’s not as simple as it might seem. Believe it or not, many ISPs have attempted to automate services of this sort and backed out. Some providers employ partial solutions today.
However, it’s not as simple as taking a host offline until it’s cleaned. You can’t, because it can’t get clean unless it’s connected. And the administrator (think of your sister or uncle) may not have a clue about how or what to do in order to sanitize a system. If they need OS patches from their system OS vendor, you have to continue providing them with that connectivity to those system, whoever the OS vendor is. If they need to download AV or anti-malware software, you need to provide them with that connectivity. If they’ve contracted some remote assessment and management service, you have to maintain that connectivity. And you have to maintain services availability for voice and emergency dialing services. And you’d better not impact that IPTV service or they’ll move back to cable or satellite services — so you’ve got to do all of this without pissing off the customer and increasing churn.
Ohh, BTW, most of those subscriber management devices you’ve deployed have limited capability to segment services at the Network Layer, much less at the Transport or Application Layer.
And how’d you get that intelligence information that triggered quarantine in the first place? What if it’s wrong, or stale, or was triggered by someone stealing wireless access from a neighbor? And what about consumer rights and privacy advocates? Why is my “pipe provider” analyzing my network activity and payloads in the first place?
The service providers want to offer services in this area, I assure you of that. As a matter of fact, the money is flowing somewhere, why not exploit the problem to increase ARPU? And many security and network equipment companies are happy to help them do this — but it’s not such a binary thing as folks might think.