In several of our previous podcasts (see episodes 34, 35,and 37), we’ve discussed what we’ve called the “Comcast Kerfuffle,” which was the controversy surrounding the steps Comcast took to manage BitTorrent traffic on its networks. Critics called it a violation of Net neutrality principles while Comcast and others called it sensible network management.

This week we saw a new kerfuffle of sorts develop over the revelation in a Monday front-page Wall Street Journal story that Google had approached major cable and phone companies and supposedly proposed to create a fast lane for its own content. What exactly is it that Google is proposing, and does it mean – as the Wall Street Journal and some others have suggested – that Google is somehow going back on their support for Net neutrality principles and regulation? More importantly, what does it all mean for the future of the Internet, network management, and consumers. That’s what we discussed on the TLF’s latest “Tech Policy Weekly” podcast.

Today’s 30-minute discussion featured two of our regular contributors at the TLF, who both wrote about this issue multiple times this week. Cord Blomquist of the Competitive Enterprise Institute wrote about the issue here and here, and Bret Swanson of the Progress & Freedom Foundation wrote about it here and here.  To help us wade through some of the more technical networking issues in play, we were also joined on the podcast by Richard Bennett, a computer scientist and network engineer guru who blogs at Broadband Politics as well as Circle ID and he also pens occasional columns for The Register.  Also appearing on the show was Adam Marcus, Research Fellow & Senior Technologist at PFF, who wrote a “nuts and bolts” essay full of excellent technical background on edge caching and net neutrality.

You can download the MP3 file here, or use the online player below to start listening to the show right now.

[display_podcast]

The introduction below was originally written by Adam Thierer, but now that I (Adam Marcus) am a full-fledged TLF member, I have taken authorship.

My PFF colleague Bret Swanson had a nice post here yesterday talking about the evolution of the debate over edge caching and network management (“Bandwidth, Storewidth, and Net Neutrality“), but I also wanted to draw your attention to related essay by another PFF colleague of mine. Adam Marcus, who serves as a Research Fellow and Senior Technologist at PFF, has started a wonderful series of “Nuts & Bolts” essays meant to “provide a solid technical foundation for the policy debates that new technologies often trigger.” His latest essay is on Network neutrality and edge caching, which has been the topic of heated discussion since the Wall Street Journal’s front-page story on Monday that Google had approached major cable and phone companies and supposedly proposed to create a fast lane for its own content.

Anyway, Adam Marcus gave me permission to reprint the article in its entirety down below. I hope you find this background information useful.

Nuts and Bolts: Network neutrality and edge caching

by Adam Marcus, Progress & Freedom Foundation

December 17, 2008

This is the second in a series of articles about Internet technologies. The first article was about web cookies. This article explains the network neutrality debate. The goal of this series is to provide a solid technical foundation for the policy debates that new technologies often trigger. No prior knowledge of the technologies involved is assumed.

To understand the network neutrality debate, you must first understand bandwidth and latency. There are lots of analogies equating the Internet to roadways, but it’s because the analogies are quite instructive. For example, if one or two people need to travel across town, a fast sports car is probably the fastest method. But if 50 people need to travel across town, it may require 25 trips in a single sports car. So a bus which can transport all 50 people in a single trip may be “faster” overall. The sports car is faster, but the bus has more capacity. Bandwidth is a measure of capacity, of how much data can be transmitted in a fixed period of time. It is usually measured in Megabits per second (Mbps). Latency is a measure of speed, of the time it takes a single packet data to travel between two points. It is usually measured in milliseconds. The “speeds” that ISPs advertise have nothing to do with latency; they’re actually referring to bandwidth. ISPs don’t advertise latency because its different for each different site you’re trying to reach. The Internet consists of devices and wires connecting those devices. The speed of data along the wires is fixed–there are no fast lanes and slow lanes. The only way to increase speeds is to either travel a shorter path or to get priority at the routers, the virtual traffic lights of the Internet. ISPs advertise bandwidth because with more bandwidth, more data can get to you in fewer trips, making your broadband connection seem much faster than a dial-up connection.

Sometimes latency and bandwidth are important and sometimes they’re not that important. The typical response time between any two points on the Internet is 1/5th of one second, so the difference between a relatively fast and relatively slow connection isn’t much. If you’re sending an email (without any attachments) or chatting with someone using an Instant Messaging program, you’re not using much bandwidth and if your messages are delayed by a second it’s probably not a problem. Or when Microsoft Windows is downloading system updates in the background, whether the download completes in a few minutes or an hour really doesn’t matter–as long as it completes. The emails and IMs are low-bandwidth and the system updates are usually high-bandwidth, but in both of these examples, latency is not that important. But if you’re playing a real-time online multiplayer game, making a VoIP phone call, videoconferencing, or remotely connecting to another computer using pcAnywhere, GoToMyPC, or Remote Desktop Services, both bandwidth and latency are important. Without a high-bandwidth low-latency connection, you’ll experience drop-outs and lag. NOTE – Latency is a measure of time, so the lower the latency the better.

Latency is most affected by the Internet equivalent to traffic lights: routers. Data transmitted over the Internet is sent in packets which contain a header that specifies, among a few other things, the IP address of the intended destination computer. Between every connection sits a router. For every packet that arrives at every router, the router must look at its header to determine where to send it, and then forward the packet out along the proper connection. Normally, routers inspect and forward packets with almost no delay. But when there are too many packets for a router to handle or the tubes get filled, the packets are temporarily queued in the router’s memory. This queuing imposes some delay. If the memory becomes full, the router drops (deletes) some of the packets and tries to keep going. If the sending computer doesn’t get a response in a certain amount of time, it assumes the packet has been dropped and sends it again, resulting in even more delay. On average, about 6% of packets are lost.

One way to deal with overloaded routers is to simply install more and bigger routers. Another method is to build more connections so packets don’t have to travel through as many routers. But both of these options are costly and it’s not clear whether simply increasing capacity will be enough to keep pace with increasing demand. A third option is to prioritize the packets. Prioritizing packets is kind of like the Mobile InfraRed Transmitter (MIRT) system that allows emergency response vehicles (e.g. fire, police, and EMS) to immediately turn specially-equipped traffic lights green. Most people would probably agree that this form of traffic priortization is a good idea. But when referring to the Internet, talk of traffic prioritization starts arguments.

The Network Neutrality Debate: What’s It All About

The network neutrality debate is a debate about the best method to manage traffic on the Internet. Those who advocate for network neutrality are actually advocating for legislation that would set strict rules for how ISPs manage traffic. They essentially want to re-classify ISPs as common carriers. Those on the other side of the debate believe that the government is unable to set rules for something that changes as rapidly as the Internet. They want ISPs to have complete freedom to experiment with different business models and believe that anything that approaches real discrimination will be swiftly dealt with by market forces.

But what both sides seem to ignore is that traffic must be managed. Even if every connection and router on the Internet is built to carry ten times the expected capacity, there will be occassional outages. It is foolish to believe that routers will never become overburdened–they already do. Current routers already have a system for prioritizing packets when they get overburdened; they just drop all packets received after their buffers are full. This system is fair, but it’s not optimized.

The network neutrality debate needs to shift to a debate on what should be prioritized and how. One way packets can be prioritized is by the type of data they’re carrying. Applications that require low latency would be prioritized and those that don’t require low latency would not be prioritized. But who makes the determinations? What happens if someone hacks their computer to prioritize packets that shouldn’t be? Another method is for ISPs to offer prioritization for a fee. ISPs could determine who should get prioritization based on the source or destination IP address in the packet header, or content providers could pay ISPs to prioritize only packets they tag with a special marker.

Opponents of network neutrality mandates argue that it’s simply not feasible to increase capacity to the extend that would be necessary without prioritization. They believe that with prioritization, they will be able charge more for faster access to those willing to pay, and the increased revenue will provide the funding necessary to upgrade the networks, which will benefit everyone. As the saying goes, a rising tide lifts all boats. Network neutrality advocates fear that if ISPs are allowed to charge for prioritization, they will have no incentive to increase speeds for those who don’t pay for prioritization. While that may be true, price discrimination is very different from other forms of discrimination. It would be a real shame if the net neutrality debate over latency hampered efforts to increase bandwidth. Even common carriers were not restricted from setting different prices for different classes of service, they simply had to offer the same rates to all comers. If those who claim the Internet should be a completely level playing field applied the same logic to the phone system, toll-free numbers wouldn’t be allowed.

Edge Caching: What It Is and Isn’t

Monday’s Wall Street Journal ran an article suggesting that Google is abandoning its stance as an advocate for Network Neutrality because of a plan to set up edge caching servers. Edge caching is just a way to more efficiently balance the costs of storage space and bandwidth in an attempt to decrease latency. It a way to move content “closer” to the end-users that view it to avoid the latency that occurs as packets traverse longer distances across the network.

To continue the roadways analogy, imagine the Internet arranged like a city. The end-users are all in the suburbs and the data they want to access is downtown in the network’s “core.” With this model, every request from a user needs to “commute” from the suburbs to the core, and the requested data needs to then travel from the core all the way back to the suburbs. Just like companies realized that setting up satellite offices nearer to its workers would decrease commuting times and increase productivity, content providers have realized that setting up edge caching servers at major ISPs decreases latency and saves on bandwidth costs.

Edge caching doesn’t work for all types of Internet content. If the content changes rapidly, edge caching doesn’t save much bandwidth because you’re constantly pushing new content to the edge servers. But for popular YouTube videos, edge caching is a great way for Google to save on bandwidth costs. Before Google bought YouTube, YouTube outsourced the hosting of its videos to edge caching provider LimeLight. So its no surprise that Google is now looking to do the same with its own edge caching servers.

The fact that Google can afford to set up edge caching servers around the network does give it a bit of an advantage. But the advantage is mostly a savings in bandwidth costs for the content provider. The use of edge servers is meant to be almost unperceptable to users. Accessing content from edge servers may be a bit faster for users, but nobody is being discriminated against and most content on the Internet is not latency-sensitive. In the example of Internet video, the difference between playing a video hosted on an edge caching server versus playing video from a server located far away may be just a matter of a few seconds delay before the video begins playing.

Some, like the Wall Street Journal, argue that even edge caching violates the net neutrality principle of the Internet being a level playing field. I would suggest that only discriminatory practices, such as an ISP offering packet prioritization to only some companies, should be considered a violation of net neutrality principles.

As Google points out, other companies are free to set up their own edge caching servers or use one of the many companies that offer edge caching services. There have been economies of scale in other industries for generations. The fact that edge caching provides economies of scale for Internet content providers is not a game changer. On the Internet, just as in other media industries, it’s not who can get their goods to market the fastest, it’s whose content best satisfies their audiences.

— Adam Marcus (adamm@pff.org)

Claims that Google has abandoned its stance on network neutrality have been thoroughly debunked, as Cord and Adam note below. Over at Broadband Reports, Karl Bode explains that Google is seeking edge-caching agreements, not preferential treatment. Edge-caching involves Google housing its content on servers located inside consumer ISP networks, cutting bandwidth costs by allowing users to access Google content located just a few hops away.

Even though edge-caching doesn’t violate network neutrality as defined by Google, it’s still one of the many advantages that big players have over new entrants. Edge-caching isn’t a “fast track,” as the WSJ imprecisely terms it, but rather a short track—functionally, there’s a lot of similarity between the two. As Richard Bennett has explained time and time again, being close to end users is quite advantageous even without preferential treatment, as it eliminates the need to push vast amounts of data across the congestion-prone core of the public Internet.

We’ve heard about how edge-caching enables content providers and ISPs to cut their bandwidth bills and make more efficient use of finite network resources. Both of these are true, but there’s more—edge caching makes it much less likely that users will experience long load times or buffering hiccups while watching streaming video online. That high-def YouTube clip might take a few extra seconds to buffer if it has to make its way through congested central network exchanges—not so, however, if that video is housed just a few hops away, within your ISP’s network.

Of course, as Larry Lessig points out, there’s nothing stopping anybody from negotiating edge-caching arrangements with ISPs directly or with content delivery networks like Akamai. Like many network enhancing-technologies, though, edge caching costs money. But what’s wrong with that? A blog post over at NYTimes.com quips that Google has a “Treat All Rich Companies the Same” vision of net neutrality. The post goes on to point out that:

The reality today is that rich companies already get first class service, and most network neutrality proposals aren’t going to change that. Big sites buy faster Internet connections and get better service from their providers. Moreover, those with money can buy services content delivery networks like Akamai, or in the case of the superrich, they can set up their own networks, as Google is trying to do.

Neutrality supporters’ refrain—that the Internet must remain open to all, free from any arrangements that grant a leg up to established players at the expense of the little guy—would seem to go against edge-caching. This apparent disconnect between how net neutrality regulation is argued and how it’s actually defined is quite revealing, as ArsTechnica points out.

Wayne Crews argues on OpenMarket that arrangements such as those sought by Google benefit the Internet by making it a more valuable network than it would otherwise be:

Special deals like Google’s, as well as future proprietary services that use Internet technology, but may or may not ride the same pipes as the “capital-I” Internet, increase the Net’s overall functionality. Policy should not discourage the possible emergence of such a “Splinternet” by catering to the old-school model of infrastructure socialism and sleepy-headed “openness.”

And as Cord correctly states, efficient agreements that harness the Internet’s power to deliver rich content and give us greater access to information should welcomed, not shunned.

Over just the past 24 hours, there’s been quite a hullabaloo surrounding the Wall Street Journal’s controversial front-page story on Google’s edge caching plan and whether it violates Net neutrality. (See Cord’s post and Bret’s). Lessig calls it a “made-up drama“, David Isenberg says it’s “bogus” and “bullshit,” and Google’s Rick Whitt has said it’s much ado about nothing.

Regardless, here’s the important thing not to overlook about this episode: It is a prime example of the what Tim Lee has referred to as “the fundamental problem of backlash” that ensues whenever there is even a hint of a potential violation of network neutrality (however one defines it). As Tim argued in his excellent Cato paper on Net neutrality, “No widespread manipulation would go unnoticed for very long,” and a “firestorm of controversy would… be unleashed if a major network owner embarked on a systematic campaign of censorship on its network.” (p. 23). Indeed, this (non-)story about Google’s edge-caching plans have spawned an intense “firestorm of controversy” over the past 24 hours and it doesn’t even involve serious network meddling or censorship! I’ve been trying to keep up with all the traffic about this on TechMeme and Google News during that time, but I have given up trying to digest it all. (Take a look at those snapshots I pasted down below to get a feel for the volume we are talking about here).

In that regard, I love this quote from the always-bloodthirsty Tim Karr of the (inappropriately-named) regulatory activist group Free Press:

If Google or any other tech company were secretly violating Net Neutrality, there would be an absolute and cataclysmic backlash from the grassroots and netroots who have made Net Neutrality a signature issue in 21st Century politics. The Internet community would come crashing down on their heads like Minutemen on Benedict Arnold.

Indeed, that’s exactly what we saw today. But it wasn’t just pro-regulatory fanatics like Free Press. The entire tech and business blogoshere and even some of the mainstream media were on top of this. That’s the “fundamental problem of backlash” at work, and with a vengeance.