Buzz Out Loud’s Epic Misunderstanding of Peering

by on December 2, 2010 · 20 comments

I love listening to podcasts, yet I’m increasingly disappointed with popular tech news podcasts like CNET’s Buzz Out Loud, which despite being staffed by tech journalists, consistently fail to grasp the basic economics of the Net.  The latest case of this arose on Episode 1360 of “BOL,” which took on the recent dispute between Comcast and Level3 over their peering agreement.

To provide some background, Comcast and Level3 have had a standard peering agreement for years, meaning the balance of incoming and outgoing traffic on either side is so similar that the two have simply agreed to exchange data without exchanging any dollars.

In the past, Comcast and Level3 had a different arrangement. Comcast paid Level3 for access to their network in a “transit” agreement.  This sort of agreement made sense at the time because Comcast was sending a lot more traffic over Level3′s network than it was taking in from Level3, hence it was a net consumer of bandwidth and was therefore treated by Level3 as a customer, rather than a peer.

Now, the tables have turned thanks to Level3 taking on the huge tasks of delivering Netflix streaming video, which takes an impressive amount of bandwidth—up to 20% of US peak traffic, according to CNET.  So, logic and economics compel Comcast to start charging Level3, as Level3 is now the net consumer.

None of this background was understand by the folks at Buzz Out Loud, which probably explains why the hosts acted as though this peering dispute was a sign of the coming Internet apocalypse, decrying the action on the podcast and summarizing their feelings on the action in the episode’s show notes by stating:

We break down the Level 3 and Comcast battle–no matter how you slice it, it’s still very, very, VERY bad for the Web.

No, it’s really, really, REALLY not.In fact, Comcast has actually been rather nice to Level3, according to Nate Anderson at Arstechnica:

Comcast “was able to scramble and provide Level 3 with six ports (at no charge) that were, by chance, available and not budgeted and forecasted for Comcast’s wholesale commercial customers.”

Anderson goes on to explain that Comcast then had to consider how this affected their agreement with Level3:

After providing these six additional ports, Comcast concluded that the existing settlement-free peering agreement with Level 3 was still (barely) valid, but if Level 3 really wanted another 21 to 24 ports, this was simply too much traffic. Level 3 would have to pay for those ports like any commercial paid peering customer.

To sum up, the two companies used to be exchanging bits at a roughly one-to-one ratio, that ratio has changed, so therefore one is charging the other for delivering some bits.  Simple, right?

Yet somehow Buzz Out Loud has managed to turn this mundane non-news into a Chicken Little story and they even went so far as to suggest that Comcast may be scheming shut off Netflix traffic entirely in order to favor their own content, especially now that they’re in the process of merging with NBC.  To say that this is a baseless theory is an understatement.

Buzz Out Loud often ducks criticism by noting that they’re just a podcast, that they’re reporting on “buzz” not hard facts, and that they’re really supposed to be entertaining.  Fine, but if BOL is really supposed to be the Daily Show of tech podcasts, then they shouldn’t make grand pronouncements on what’s good or bad for the web, especially when the basic facts of a given situation aren’t understood.

But I’m sure this isn’t the last net neutrality doomsday scenario that Buzz Out Loud will concoct out of something humdrum.  This sort of thing makes me wonder how much of the current debate on neutrality is grounded in a solid understanding of the underlying principles of the Net and how much is based on this sort of misinformed, ignorant, lazy “journalism.”

  • John

    Comcast explains in its recent FCC filing that its normal ratio of traffic exchange with L3 was 2:1. This is not “roughly one to one.” You would expect L3 to pretty much always send more data to a residential broadband network than to receive from one. By itself, this does not mean that Level 3 should pay Comcast. After all, most of the data that L3 sends to Comcast is not transit data, but data that has been requested by Comcast customers.

    Comcast’s problem is that L3 is trying to take advantage of an anomaly in the system. Backbones either send data to eyeball networks for free, or are paid for the privilege. CDNs have historically paid for the privilege. L3 decided to take advantage of this and underbid CDNs, and Comcast has moved to stop them.

    As “backbone” providers take on CDN roles, residential broadband networks build their own backbones, and so forth, the traditional arrangements between networks begin to make less sense.

    I’m not taking sides on who’s right, but I don’t think it’s accurate to say that Comcast always has been paid for unequal traffic flows. Just unequal traffic flows in some contexts and not others.

  • http://www.cordblomquist.com cordblomquist

    So is the really interesting part of this story that Level3 is trying to take advantage of its backbone status by dumping Netflix traffic onto consumer ISPs and expecting them to shoulder that added traffic at no charge?

  • Anonymous

    It is not “dumping” the traffic onto the ISPs. It is delivering the contents of the requests of the ISPs’ customers. This is why people call ISP networks “eyeball” networks, and why ISPs traditionally have transit agreements with backbone providers where they pay them for service despite the fact that there is always in imbalance of traffic such that the backbone sends more traffic to the ISP than vice-versa. Comcast’s attempt to make this about volume is far less important than the fact that Comcast is trying to leverage its market power over near-captive consumers into the relatively competitive backbone market.

  • Anonymous

    Oh, and I also think you’re incorrect that “In the past, Comcast and Level3 had a different arrangement.” Comcast evidently had and still has a transit agreement in which it pays Level 3. It claims to have also have a settlement-free peering agreement for “on-network traffic” that originates from Level 3 itself (although I admit their FCC letter really muddies the water here). Comcast seems to want to define Netflix traffic as “on-network” and thus claim that it is subject to the peering agreement, and to furthermore claim that it can renegotiate that agreement to charge Level 3. At least that’s the best I can figure.

  • Ryan Radia

    Steve’s essay on this topic is well worth a read: http://www.freedom-to-tinker.com/blog/sjs/trying-make-sense-comcast-level-3-dispute

    It seems that Comcast is indeed trying to break from how transit arrangements have historically worked. But that’s hardly evidence that its move bodes ill for consumers or that it merits the sort of government intervention that Public Knowledge and Level 3 have sought. If Comcast ultimately prevails in its current standoff with Level 3, that means Comcast won’t lose the revenue that it was previously earning from CDNs. What’s wrong with that? Yes, it means Level 3 will be out some serious cash, but Level 3 is presumably makingserious cash on account of its deal with Netflix, right? Where’s the harm?

    Of course it’s true that Comcast has significant power in the last-mile broadband market. But we know from fellow TLFers Prof. Josh Wright and Dr. Geoff Manne that government is extremely ill-equipped to identify instances of anti-competitive conduct, even when they’re allegedly undertaken by a firm with substantial market power. Even if there is a nontrivial chance that Comcast will harm competition if allowed to proceed with unfettered negotiations with Level 3, the market will self correct in time.

  • John

    Yes, I think that’s what’s interesting. In the first place Comcast doesn’t “peer” with a backbone–peer networks exchange traffic that actually is roughly 1:1, and networks that peer are typically transit networks anyway.

    It very well could be the case that the proper result is for Comcast to stop charging any other networks for incoming data per se, as opposed to Comcast starting charging anyone. It’s not at all clear to me what result the “market” would result it if every eyeball network started playing hardball with other nets. It’s true that Comcast has a strong position, but they could easily overplay their hand and end up degrading their customers’ experience. To really know who’s in the “right” here, as measured by standard practices, we’d need to know the traffic ratios between other eyeball ISPs and other backbones, how CDNs have impacted that, etc.

    I said above that CDNs have traditionally paid. That’s not quite true. Many ISPs welcome the likes of Akamai with open arms because it lowers their transit costs.

    Also, it’s pretty clear by now that traditionally-understood CDNs don’t need to work by edge caching. I don’t think that’s what L3 does, nor what Limelight does. If your backbone is hellacious enough it doesn’t matter where, exactly, the servers are.

  • Anonymous
  • Anonymous

    George, your analysis there and in your inflammatory article seems to simply assume that Comcast can arbitrarily decide when to charge for “CDN peering” data and when to pay for transit data. As many commenters on your article and folks on NANOG have observed, there is no public evidence of such a peering agreement, and even if such an agreement exists there’s no explanation of why Comcast can unilaterally say that Netflix traffic counts as “on-net” traffic, and that all “on-net” traffic must be peering vs. transit traffic. You can draw lines on a diagram between the “CDN Division” of Level 3 and Comcast, but that does not make it so.

  • Anonymous
  • Anonymous

    A very good blog post explaining the situation here:
    http://www.voxel.net/blog/2010/12/peering-disputes-comcast-level-3-and-you

  • Ryan Radia

    Question 9 is revealing: “. If Comcast believes that the monthly payment is inadequate, Comcast can either lower the cap or charge more for higher usage as many other broadband access providers have already done. It is important to remember that Comcast’s subscribers have requested the content delivered by Level 3, and that Level 3 must also add capacity as these requests increase. We try to charge our customers a fair price for the expense associated with meeting requests from Comcast’s customers. We do not understand why Comcast cannot do the same, especially when caps and bandwidth limits are already in place.”

    I don’t understand why I, as a Comcast subscriber, should be concerned about this dispute — so long as I don’t suffer from degraded Netflix connectivity. But if you’re concerned about Comcast’s dominant position in last-mile broadband, shouldn’t you want its subscribers to revolt? Doesn’t that anger encourage investors to build competing infrastructure?

  • Anonymous

    “I don’t understand why I, as a Comcast subscriber, should be concerned about this dispute — so long as I don’t suffer from degraded Netflix connectivity. But if you’re concerned about Comcast’s dominant position in last-mile broadband, shouldn’t you want its subscribers to revolt? Doesn’t that anger encourage investors to build competing infrastructure?”

    Are you implying that it would be smart business for Level 3 to call Comcast’s bluff, theoretically resulting in degraded service? This doesn’t work because the backbone market is competitive (ie: Netflix can go to another backbone or CDN) as is the online streaming market (at least to an extent, for example Amazon streaming). Last-mile service is not.

  • Ryan Radia

    No. I agree that it would be unwise for Level 3 to refuse Comcast’s terms, which is why I don’t foresee Comcast subscribers suffering harm due to this dispute (unless the FCC intervenes). I simply meant that even if Level 3 were to rebuke Comcast, self-corrective forces would eventually mitigate consumer harm.

    On a related note, because federal antitrust laws bar horizontal restraints in many instances, CDNs and backbone providers might not be legally permitted to join forces against Comcast even if they wanted to. That’s unfortunate, because while no single CDN or backbone provider has anywhere near Comcast’s market power, their bargaining power would increase dramatically were they to negotiate the terms of transit arrangements with Comcast on a group basis.

  • Anonymous

    So your theory is that if L3 said “no” to Comcast, and if therefore Comcast customers couldn’t get Netflix, that (“eventually”!) somebody think of some economically viable way of building a last-mile competitor?

    Netflix can go elsewhere.

    There are (at least partial) alternatives to Netflix (including, conveniently, Comcast products!)

    Potential last-mile competitors already have plenty of incentive to build out.

  • Anonymous

    @Sjschultze

    “Comcast can simply decide when particular traffic counts as “CDN peering” that they can charge for and when it counts as transit that they have to pay for”

    I’ll make it simple as possible for you, and this is pretty standard in the industry.

    Comcast traffic going or coming from the Internet through Level 3 requires Comcast to pay Level 3 for transit.

    Level 3 CDN traffic (not “open” Internet traffic) going to Comcast over private peering capacity, beyond the free 260 Gbps of private capacity, and not used for transit, requires Level 3 to pay Comcast.

  • Anonymous

    And I will make it as simple as possible for you.

    You have provided no actual evidence (by drawing it in your diagram or asserting it here) that this was the agreement, or that Level 3 is trying to send this traffic over a “private peering” link that is different from their general transit interconnect point.

    Industry practice is for eyeball networks to pay upstream providers for transit. Sure, some edge-facilities-based CDN providers that rent capacity from others have struck such a deal in reverse, but L3 is not one of those.

    In any event, the relative market power of each company within their relevant markets makes it quite clear that competitive forces will not discipline this negotiation into optimal efficiency.

  • Anonymous

    I’ve confirmed with both Comcast and Level 3 that there is both a transit and peering component of their agreement. Comcast also publicly confirmed this in a published statement http://www.digitalsociety.org/2010/12/comcast-confirms-my-observations-on-peering-dispute/.

    So you can stop reciting the debunked myth that there is no peering agreement.

  • Anonymous

    Great. I too suspected that there might be more than one type of agreement at work, but didn’t simply assert it to be true (nor did I “recite a debunked myth”). Your apparent confirmation is however a far cry from proving that Netflix traffic necessarily counts as peering traffic under these agreements, and even further from demonstrating why Comcast’s demands reflect the behavior of a competition-disciplined actor (and/or the optimal policy outcome). It’s possible that as a narrow legal matter about the terms of the contract(s) Comcast would win a suit, but that at the same time such contract(s) do not make for a well-functioning market overall (and that given such a hypothetical outcome of the dispute L3 would probably rethink its way of doing contracts with Comcast). I see that on your blog you continue to try to engage in comparative bean-counting of the costs of each company even though this is largely irrelevant to the overall question of market power and solutions that mitigate market failures rather than relying on some third-party to dictate what is “fair” (an inherently subjective exercise).

  • Anonymous

    You just can’t stop beating that dead horse can you? Both Level 3 and Comcast agree on this issue that Netflix is peering. The only disagreement between them is that Level 3 wants the additional 300 Gbps of peering capacity settlement free, but Comcast will only give an additional 60 Gbps free and they want money for anything above that.

    To dispute these incontrovertible facts that are not disputed by the relevant parties is to live in denial of reality.

    Netflix can choose to come through transit if they wish to use direct delivery or some other CDN, but not if they want to use the higher performance Akamai or Level 3 CDN service. For that it has to go through some peering connection and those CDNs have agreed to pay for it.

  • Anonymous

    To begin with, as I have stated from the beginning, it DOES NOT MATTER whether you call this a “peering” or “transit” or “interconnection” dispute. The market analysis is the SAME. However, with respect to your claim that “Both Level 3 and Comcast agree on this issue that Netflix is peering”:

    http://www.level3.com/index.cfm?pageID=491&PR=965

    4. Q: Is the disagreement between Level 3 and Comcast “just a good old fashion peering dispute”?

    A: No. The dispute between Level 3 and Comcast is not a peering dispute, which relates to connection of Internet backbone networks. At issue is a fundamental interconnection disagreement between Comcast, as a provider of local high speed Internet access to consumers who pay Comcast for access to content, and Level 3, which delivers content to residential broadband access providers like Comcast in response to consumer requests. Unlike “peering” in the Internet backbone, where competition abounds and prices have been declining steadily, Internet carriers that have content requested by Comcast subscribers have no choice but to exchange traffic with Comcast. Comcast is using this dominant position to demand payment for traffic delivered at its customers’ requests. You simply cannot “route around” Comcast to provide requested content to Comcast’s subscribers.

    Second, Netflix and/or Level 3 cannot simply route around the problem by using transit. As I have noted here, on your blog and elsewhere:
    http://www.voxel.net/blog/2010/12/peering-disputes-comcast-level-3-and-you
    http://www.dslreports.com/shownews/Claims-Resurface-Concerning-Congested-Comcast-TATA-Links-111818

    My fundamental point, which you continue to try to distract from by getting lost in the weeds, is that the last-mile terminating access monopoly (and market power more generally) makes the resolution of this dispute impossible to resolve in a market-oriented way without first addressing that market failure. You wish to introduce an approach which is fundamentally not free-market oriented, in which a self-declared sage decides what is “fair” and what is not.

Previous post:

Next post: