What Vox Doesn’t Get About the “Battle for the Future of the Internet”

by on May 2, 2014 · 8 comments

My friend Tim Lee has an article at Vox that argues that interconnection is the new frontier on which the battle for the future of the Internet is being waged. I think the article doesn’t really consider how interconnection has worked in the last few years, and consequently, it makes a big deal out of something that is pretty harmless.

How the Internet used to work

The Internet is a network of networks. Your ISP is a network. It connects to the other ISPs and exchanges traffic with them. Since connections between ISPs are about equally valuable to each other, this often happens through “settlement-free peering,” in which networks exchange traffic on an unpriced basis. The arrangement is equally valuable to both partners.

Not every ISP connects directly to every other ISP. For example, a local ISP in California probably doesn’t connect directly to a local ISP in New York. If you’re an ISP that wants to be sure your customer can reach every other network on the Internet, you have to purchase “transit” services from a bigger or more specialized ISP. This would allow ISPs to transmit data along what used to be called “the backbone” of the Internet. Transit providers that exchange roughly equally valued traffic with other networks themselves have settlement-free peering arrangements with those networks.

How the Internet works now

A few things have changed in the last several years. One major change is that most major ISPs have very large, geographically-dispersed networks. For example, Comcast serves customers in 40 states, and other networks can peer with them in 18 different locations across the US. These 18 locations are connected to each other through very fast cables that Comcast owns. In other words, Comcast is not just a residential ISP anymore. They are part of what used to be called “the backbone,” although it no longer makes sense to call it that since there are so many big pipes that cross the country and so much traffic is transmitted directly through ISP interconnection.

Another thing that has changed is that content providers are increasingly delivering a lot of a) traffic-intensive and b) time-sensitive content across the Internet. This has created the incentive to use what are known as content-delivery networks (CDNs). CDNs are specialized ISPs that locate servers right on the edge of all terminating ISPs’ networks. There are a lot of CDNs—here is one list.

By locating on the edge of each consumer ISP, CDNs are able to deliver content to end users with very low latency and at very fast speeds. For this service, they charge money to their customers. However, they also have to pay consumer ISPs for access to their networks, because the traffic flow is all going in one direction and otherwise CDNs would be making money by using up resources on the consumer ISP’s network.

CDNs’ payments to consumer ISPs are also a matter of equity between the ISP’s customers. Let’s suppose that Vox hires Amazon CloudFront to serve traffic to Comcast customers (they do). If the 50 percent of Comcast customers who wanted to read Vox suddenly started using up so many network resources that Comcast and CloudFront needed to upgrade their connection, who should pay for the upgrade? The naïve answer is to say that Comcast should, because that is what customers are paying them for. But the efficient answer is that the 50 percent who want to access Vox should pay for it, and the 50 percent who don’t want to access it shouldn’t. By Comcast charging CloudFront to access the Comcast network, and CloudFront passing along those costs to Vox, and Vox passing along those costs to customers in the form of advertising, the resource costs of using the network are being paid by those who are using them and not by those who aren’t.

What happened with the Netflix/Comcast dust-up?

Netflix used multiple CDNs to serve its content to subscribers. For example, it used a CDN provided by Cogent to serve content to Comcast customers. Cogent ran out of capacity and refused to upgrade its link to Comcast. As a result, some of Comcast’s customers experienced a decline in quality of Netflix streaming. However, Comcast customers who accessed Netflix with an Apple TV, which is served by CDNs from Level 3 and Limelight, never had any problems. Cogent has had peering disputes in the past with many other networks.

To solve the congestion problem, Netflix and Comcast negotiated a direct interconnection. Instead of Netflix paying Cogent and Cogent paying Comcast, Netflix is now paying Comcast directly. They signed a multi-year deal that is reported to reduce Netflix’s costs relative to what they would have paid through Cogent. Essentially, Netflix is vertically integrating into the CDN business. This makes sense. High-quality CDN service is essential to Netflix’s business; they can’t afford to experience the kind of incident that Cogent caused with Comcast. When a service is strategically important to your business, it’s often a good idea to vertically integrate.

It should be noted that what Comcast and Netflix negotiated was not a “fast lane”—Comcast is prohibited from offering prioritized traffic as a condition of its merger with NBC/Universal.

What about Comcast’s market power?

I think that one of Tim’s hangups is that Comcast has a lot of local market power. There are lots of barriers to creating a competing local ISP in Comcast’s territories. Doesn’t this mean that Comcast will abuse its market power and try to gouge CDNs?

Let’s suppose that Comcast is a pure monopolist in a two-sided market. It’s already extracting the maximum amount of rent that it can on the consumer side. Now it turns to the upstream market and tries to extract rent. The problem with this is that it can only extract rents from upstream content producers insofar as it lowers the value of the rent it can collect from consumers. If customers have to pay higher Netflix bills, then they will be less willing to pay Comcast. The fact that the market is two-sided does not significantly increase the amount of monopoly rent that Comcast can collect.

Interconnection fees that are being paid to Comcast (and virtually all other major ISPs) have virtually nothing to do with Comcast’s market power and everything to do with the fact that the Internet has changed, both in structure and content. This is simply how the Internet works. I use CloudFront, the same CDN that Vox uses, to serve even a small site like my Bitcoin Volatility Index. CloudFront negotiates payments to Comcast and other ISPs on my and Vox’s behalf. There is nothing unseemly about Netflix making similar payments to Comcast, whether indirectly through Cogent or directly, nor is there anything about this arrangement that harms “the little guy” (like me!).

For more reading material on the Netflix/Comcast arrangement, I recommend Dan Rayburn’s posts here, here, and here. Interconnection is a very technical subject, and someone with very specialized expertise like Dan is invaluable in understanding this issue.

  • John Jackson

    I disagree, but you put forth the best defence for the other side.

  • sumJesusSuspention

    Thank you.

  • Mbps

    The reason Interconnection fees are paid to Comcast is because Comcast refuses to augment congested Interconnection points unless they are paid. And it can force that choice (between paying its arbitrary fees or suffering through congested links) because it faces no real competiton in most places, and it has a monopoly over the only connection to its subscribers. You can say NTFLX did not buy a fast lane, but that’s exactly what it did–it bought a good and apparently readily available connection directly from Comcast because its other options (i.e. CDN/ transit via Cogent) were congested by Comcast at their interconnection location with Comcast. I agree that Comcast isn’t supposed to charge for a fast lane, but that appears to be what it did.

  • Uri

    This was an enlightening read, thank you.

    But, you explained perfectly how “the little guy” is harmed. The CDN negotiates payments and quality of service on your behalf, but it used to do so on your and Netflix’s behalf. There’s a positive feedback loop where the CDNs lose leverage and get worse terms, which leads more big players to “vertically integrate”, which again reduces the CDN leverage. The gap in price and quality of service keeps growing and forms an ever increasing barrier to entry for “little guys” trying to compete.

  • John Jackson

    A few points on why I disagree:
    1) Comcast’s disingenuous argument about spreading cost of connecting to Netflix to all it’s subscribers, when that is exactly the model they have on the TV Video side of the house. Everyone pays for ESPN if they are into sports or not.
    2) Interconnections are not that expensive for CableCo, they a very very few ports on network equipment out of millions of ports the CableCo manages. The CDN pays the cost to bring the packets to the CableCo doorstep, The CableCo just has to assign the ports for the connection.
    3) The idea that the good of assigning the cost to the right subscribers outweighs to the bad of CableCo’s monopoly position needs more justification.

  • Ryan Radia

    Responding to (1), last year Comcast unveiled “Internet Plus” in many markets. It’s a $70/month tier that includes broadband (with 25mbps downstream), local channels, HBO Go, and a handful of other channels.

    As for (2), if network upgrades to facilitate interconnection are cheap, why doesn’t Netflix just pay Comcast for them? Or are you claiming Comcast is overcharging Netflix? If so, isn’t Comcast encouraging Netflix to add a “Comcast surcharge” that other ISP subscribers (including Comcast competitors) won’t have to bear?

  • John Jackson

    Yes Comcast is overcharging because it’s customers are paying to connect to the Internet. The only reason the ports are over subscribed is because Comcast’s customers are requesting content on the other side of the connection, ie The Internet.
    The cost of ports is in the thousands (not even tens of thousands) of dollars. Netflix (or Level3) is paying all the delivery cost to bring the connection to Comcast doorstep. Netflix will even pay all the equipment cost for local caching of content. All they are asking for in return for 1) Having the content Comcast Customers want, and 2) delivering that content to Comcast, is for a few ports that cost a few thousands of dollars. That’s it.
    Instead Comcast is leveraging its market size and last mile monopoly to extract rents.
    Yes they are overcharging by charging anything at all.

  • Cincinnatusdc

    Eli, how much $ has your program received from Comcast and NCTA in the past 5 years?

Previous post:

Next post: