As a means of introducing myself to TLF readers, this is an article that I wrote for the PFF blog in September that has not been previously mentioned on the TLF. Most of my other PFF blog posts have been cross-posted by Adam Thierer or Berin Szoka, but I’ve taken ownership of those posts so they appear on my TLF author page.

This is the first in a series of articles that will focus directly on technology instead of technology policy. With an average age of 57, most members of Congress were at least 30 when the IBM PC was introduced in 1981. So it is not surprising that lawmakers have difficulty with cutting-edge technology. The goal of this series is to provide a solid technical foundation for the policy debates that new technologies often trigger. No prior knowledge of the technologies involved is assumed, but no insult to the reader’s intelligence is intended.

This article focuses on cookies–not the cookies you eat, but the cookies associated with browsing the World Wide Web. There has been public concern over the privacy implications of cookies since they were first developed. But to understand them , you must know a bit of history.

According to Tim Berners Lee, the creator of the World Wide Web, “[g]etting people to put data on the Web often was a question of getting them to change perspective, from thinking of the user’s access to it not as interaction with, say, an online library system, but as navigation th[r]ough a set of virtual pages in some abstract space. In this concept, users could bookmark any place and return to it, and could make links into any place from another document. This would give a feeling of persistence, of an ongoing existence, to each page.”[1. Tim Berners-Lee, Weaving The Web: The Original Design and Ultimate Destiny of the World Wide Web. p. 37. Harper Business (2000).] The Web has changed quite a bit since the early 1990s.

Today, websites are much more dynamic and interactive, with every page being customized for each user. Such customization could include automatically selecting the appropriate language for the user based on where they’re located, displaying only content that has been added since the last time the user visited the site, remembering a user who wants to stay logged into a site from a particular computer, or keeping track of items in a virtual shopping cart. These features are simply not possible without the ability for a website to distinguish one user from another and to remember a user as they navigate from one page to another. Today, in the Web 2.0 era, instead of Web pages having persistence (as Berners-Lee described), we have dynamic pages and “user-persistence.”

This paper describes the various methods websites can use to enable user-persistence and how this affects user privacy. But the first thing the reader must realize is that the Web was not initially designed to be interactive; indeed, as the quote above shows, the goal was the exact opposite. Yet interactivity is critical to many of the things we all take for granted about web content and services today.

Stateful Sessions

On the original World Wide Web designed by Berners-Lee (Web 1.0), Web servers responded to each client request without relating that request to previous requests. There was no need to remember what other pages the user had requested because the requests were for static pages. But if you’ve used a Web-based email system like Gmail, Hotmail, Yahoo! Mail, etc., you know that once you log in, the service remembers who you are as you click from message to message. When a website can keep track of a user as they move from page to page within a site it is called a “stateful session.” The website doesn’t necessarily need to know anything about the user, it just needs to be able to distinguish that particular user from all other users. For example, if you go to an online store and place a few items in your virtual shopping cart, the site still does not know your name, email address, or billing information. But it does know what you’ve placed in your cart–or more precisely, it knows what someone using your browser has placed placed in a particular cart. If you leave the site before buying anything and then go back an hour later, it’s possible that the site will have completely forgotten about you. In that case, the unique identifier persists during your “session” on the site, but it doesn’t persist between sessions.

URLs and HTTP Requests

Web 1.0 sites achieve Web page persistence by having a unique address or Uniform Resource Locator (URL) for each Web page, which is displayed in the address bar at the top of your browser as you browse the web. For example, http://www.pff.org/about/ is a simple URL pointing to a specific Web page. Every user that visits the PFF site at www.pff.org and clicks on the “About” link will be taken to the exact same page.

URLs can also store information about the user. For example, if you search for “test” on Google, the URL of the resulting page may look like the following: http://www.google.com/search?q=test&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a.[2. http://googlesystem.blogspot.com/2006/07/meaning-of-parameters-in-google-query.html] The URL contains a number of different pieces of data, separated by ampersands. There is the search query (“q=test”), the character encoding of the input (“ie=utf-8”), the character encoding of the output (“oe=utf-8”), the type and language of the client (“rls=org.mozilla:en-US:official”), and the Web browser used (“client=firefox-a”). None of this information can be used to uniquely identify the user, but this basic example illustrates how URLs can be used to specify more than simply static Web pages–and how some information can be remembered as a user navigates a website even without using cookies. Knowing how this works, you can create your own advanced searches or change the way the results are formatted (e.g., changing the language).

So how did Google know I speak English and use Firefox? That information is included in the HTTP request that my Web browser sends to the Google Web server when it requests a page. HTTP requests specify (among a few other more technical things) the desired language and a “User-Agent” field that includes the name of the browser and sometimes your operating system. This information allows websites to customize their content for different Web browsers (e.g., to ensure that it displays properly). HTTP requests also include your IP address so the Web server knows where to send its response, and geotagging allows Web servers to associate an IP address with a geographic area (though the area is rarely more accurate than the country or state). HTTP requests can also contain HTTP cookies.

HTTP Cookies

URLs can be used to uniquely identify individual users and allow stateful sessions, but unless a user bookmarks the URL containing their unique identifier, there is no way for the site to associate the same unique identifier with the same user on subsequent visits. Another option is to have users create an account and then log in each time they access the site. The website could then include the user’s unique ID in the URL on subsequent pages, so that the user only needs to log in once per session. Having to bookmark or create an account on every site you want to remember you would quickly become unmanageable. It would be nice if mapping and weather websites, for example, just remembered your location. It would be nice if the blogs you follow remembered what post you last read and displayed only unread posts when you next visit their site. What was needed at this point in the Web’s evolution was a way for websites to automatically store a unique identifier on the user’s computer and send it back to the website automatically[3. A site could also try to uniquely identify users by the IP address of their computer, but this is unreliable as there can be many computers behind a firewall sharing a single IP address.]—which is precisely what a cookie does.

To quote Wikipedia,

“HTTP cookies, or more commonly referred to as Web cookies, tracking cookies or just cookies, are parcels of text sent by a server to a Web client (usually a browser) and then sent back unchanged by the client each time it accesses that server. HTTP cookies are used for authenticating, session tracking (state maintenance), and maintaining specific information about users, such as site preferences or the contents of their electronic shopping carts.”

A cookie can contain one or more pieces of data, a description and/or URL for an online description of the cookie, how long the Web browser should store the cookie, and the domain, path, and port that the cookie should be limited to. Cookies can be set to expire after a specified interval, or can be “session cookies” that will expire when the Web browser is closed. When a cookie expires, it is deleted by the Web browser. Unexpired cookies are automatically sent back to the originating Web server when the Web browser makes any subsequent requests to the same server (the same domain, path, and port).

Neither Web servers nor Web browsers are required to support cookies, but a server may refuse to work with a Web browser that does not return the cookie(s) it sends. Cookies do not contain any executable code and are extremely small in size. They only contain data sent by the website and the data is not changed by the client computer, so there generally should be no privacy concerns about sending a cookie back to the website that created it (“First-party cookies”).

First-Party and Third-Party Cookies

Cookies are normally only sent to the server setting them or a server in the same domain ( e.g., a cookie set by mail.google.com could be shared with calendar.google.com). These are called first-party cookies because they’re set by the site displayed in the address bar of the Web browser. These cookies are typically used to tailor the website for the user. Third-party cookies, on the other hand, are typically used by advertising networks to track users across multiple Web sites where the networks have placed advertising–which allows the advertising network to target subsequent advertisements to the user’s presumed interests and also to limit the number of times a user is shown a particular ad. This targeting allows the delivery of “smarter” advertising that is less annoying and more informative to the user–and therefore more valuable to the advertiser, who will be willing to pay websites more for their ad space. However, this targeting also raises privacy concerns.

It is trivial for a Web page to contain images or other components stored on servers in other domains (“third-party elements”). In fact, it is often easier to link to an image already hosted online elsewhere than it is to host an image on your own Website.

Examples:

• Typical first-party embedded image:
• Typical third-party embedded image:

Whenever a Web browser loads a Web page or component of a Web page, it will include in its request for that component any cookies already stored on the user’s computer that are associated with the domain hosting the content. The Web server, in turn, can send a cookie or update a cookie already existing on the user’s computer.

Although your Web browser will not send a third-party cookie to the first-party Web server (and it won’t send a first-party cookie to the third-party Web server), the first-party Web server can send information to the third-party Web server by embedding it in the URL for the third-party content. The most common form of this communication between the sites you visit and the sites they rely on for content or ads is called a “web bug”–a small (usually 1 pixel by 1 pixel) graphic not meant to be noticed by the user. Its purpose is to cause the user’s Web browser to load the third-party embedded content from the external Web server, which will allow the third party (usually an advertising network) to track the user.

• Example third-party embedded web bug:

While this all may seem scary and invasive,the fact that a website or ad network can uniquely identify your browser does not mean that they have any clue who you are. Even if you provide your name, email address, or other personally-identifiable information to the first-party Web site, most sites’ privacy policies state that they will not share this information with their advertising partners. To use a real-world analogy, third-party advertising is equivalent to a marketer in a mall watching you come out of a music store and then offering you a flyer for a concert: The marketer may know that you’re interested in music (because you were shopping at the music store), but they have no idea who you are. And as my colleagues Adam Thierer and Berin Szoka explained in their post on Adblock Plus, websites (especially smaller independent websites) depend on advertising as a source of revenue and to cover their overhead costs.

Alternatives to Cookies

Cookies are not the only way websites can do stateful sessions. As has already been mentioned, Websites can put unique identifiers in URLs. But custom URLs don’t last between sessions. Websites that need to remember users ( e.g., websites that charge a fee for access) can require users to create an account and log into the site every time they use it.

But most websites do not require users to create an account and log in every time. And more and more users are configuring their Web browsers to delete all cookies when they close the browser. In response, Web site operators have found other methods to uniquely identify users by storing a unique identifier on users’ computers.

The cookie alternatives listed below are not any more or less invasive of privacy than cookies if the user is aware of them and manages them the same way they manage cookies. But most Web browsers don’t give users the same amount of control over cookie alternatives that they do over cookies, and few users know about these alternatives.

Per-session cookie alternatives – These cookie alternatives are not saved to disk and thus are not accessible after you close your Web browser.

• Hidden form fields – Web pages can contain hidden Web forms that submit data back to the Web server when an on-screen button is pressed. This method is quite limited because it requires the user to click a specific button, and there is no method for saving data after you’ve navigated away from the site. Beyond these limitations, the only way to detect hidden form fields is to inspect the HTML code for a page. There is also no easy way to block hidden form fields.
• window.name – JavaScript embedded in a Web page can set or read the this internal value that’s not really used for anything else. The value can be up to 32 megabytes in size and once set a value can be accessed by any Web site. Although the only way to detect this is to inspect the HTML code for a page, you can disable JavaScript.

Persistent cookie alternatives – These cookie alternatives are like cookies in that they are saved on your computer and can be accessed even after you’ve closed your Web browser.

• Flash Cookies – Also known as Local Shared Objects, Flash cookies require Adobe Flash to be installed on your computer. Whereas HTTP cookies are limited to 4 kilobytes, Flash cookies can contain up to 100 kilobytes by default and can contain an unlimited amount of data if the user desires. To view and delete the Flash cookies stored on your computer, go to this page (although accessed via a Web page, the Flash cookies shown are stored on your computer). You can also permanently disable Flash cookies on that page.
• DOM Storage – DOM storage was designed specifically to allow Web 2.0 applications to work offline, saving data locally when they are unable to access the host website and to save data that would otherwise be lost if a page is accidentally reloaded. DOM storage is currently only implemented in Firefox (and Internet Explorer 8 Beta). If cookies are disabled, DOM storage is also disabled. Users can also manually disable DOM storage even when cookies are enabled.
• userData behavior – The userData behavior does for Internet Explorer what DOM storage does for Firefox. Each “document” is limited to 128 kilobytes of storage, with a per-domain limit of 1024 kilobytes. The data is stored in Internet Explorer’s cache and are deleted when you delete cookies using the Delete Browsing History dialog box.

Conclusion

This article should give you a better sense of what cookies are used for and how they work. You should now see that per-session cookies and cookie alternatives are completely harmless. Persistent cookies (and cookie alternatives) can make your Web browsing a bit easier, but deleting them will not (in most cases) cause any problems. If you are concerned about your privacy, you will need to do a bit more than just delete cookies–you also need to delete or disable the above-mentioned cookie alternatives.

The Progress & Freedom Foundation has just launched the new Center for Internet Freedom.  CIF offers an alternative to the proliferation of advocacy groups calling for government intervention online by offering timely analyses and critiques of proposals that diminish the vital role of free markets, free speech and property rights.  We aim to drive the Internet policy debate in new directions by emphasizing a layered approach of technological innovation, user education, user self-help, industry self-regulation, and the enforcement of existing laws consistent with the First Amendment.  Such an approach is a less restrictive—and generally more effective—alternative to increased regulation.  

Here are some of the issues I’ll be working on as CIF’s Director in conjunction with my esteemed colleagues Adam Thierer, Adam Marcus, and adjunct fellows: 

• Defending online advertising as the lifeblood of online content & services, especially in the “Long Tail”;
• Emphasizing market solutions to problems of privacy protection, especially regarding the use of cookies and packet inspection data;
• Protecting online speech and expression both in the U.S. and abroad;
• Defending Section 230 immunity for Internet intermediaries;
• Opposing online taxation and legal barriers to e-commerce and digital payments, especially at the state and local levels; and
• Ensuring that Internet governance remains transparent and accountable without hampering the evolution of the Internet.

By Berin Szoka & Adam Thierer Progress Snapshot 4.19 (PDF)

Since the fall of 2008, a debate has raged in Washington over “targeted online advertising,” an ominous-sounding shorthand for the customization of Internet ads to match the interests of users.  Not only are these ads more relevant and therefore less annoying to Internet users than untargeted ads, they are more cost-effective to advertisers and more profitable to websites that sell ad space.  While such “smarter” online advertising scares some—prompting comparisons to a corporate “Big Brother” spying on Internet users—it is also expected to fuel the rapid growth of Internet advertising revenues from $21.7 billion in 2007 to $50.3 billion in 2011-an annual growth rate of more than 24%. Since this growing revenue stream ultimately funds the free content and services that Internet users increasingly take for granted, policymakers should think very carefully about what’s really best for consumers before rushing to regulate an industry that has thrived for over a decade under a layered approach that combines technological “self-help” by privacy-wary consumers, consumer education, industry self-regulation, existing state privacy tort laws, and Federal Trade Commission (FTC) enforcement of corporate privacy policies.

In an upcoming PFF Special Report, we will address the many technical, economic, and legal aspects of this complicated policy issue-especially the possibility that regulation may unintentionally thwart market responses to the growing phenomenon of users blocking online ads.

We will also issue a three-part challenge to those who call for regulation of online advertising practices:

1. Identify the harm or market failure that requires government intervention.
2. Prove that there is no less restrictive alternative to regulation.
3. Explain how the benefits of regulation outweigh its costs.

The Online Advertising Market

While there are other forms of targeted advertising based on who you are (“demographic”) or where you are (“locational”), the most important varieties are based on what you’re searching for, seeing or doing online at any particular moment (“contextual”) and the pattern of what you’re searching for, seeing or doing over time (“behavioral”). The bulk of Internet advertising falls into one or both of these last two categories, with behavioral advertising growing rapidly.

Search engines deliver contextual ads on search results pages based on the search keywords entered by a user, while third-party advertising networks (some of which also run search engines) deliver contextual ads on behalf of website operators who sell ad space to the network, with the ads displayed on each page chosen according to keywords on that page. Contextual advertising is far “smarter” than displaying the same “dumb” untargeted banner ads to every user, because the contextual ad uses keywords to “guess” what the user is interested in based on the context of each page. But the purely contextual ad network doesn’t “remember” what the user has looked at in the past, so its insights into what the user would find relevant are very limited, especially for some websites. Online behavioral advertising (OBA) solves this problem and increases the value of advertising space on all websites by targeting ads based on a “profile” of the user created by tracking websites the user has visited—as well as limiting the number of times a user is shown a particular ad.

The Perceived Harm Driving Calls for Regulation

For a decade, the basic technology behind OBA has changed little: When a user visits the typical webpage, they download not only the webpage contents but also a small piece of code that allows the website to distinguish that user’s browser from other browsers (a “cookie”)—without personally identifying the user. Some cookies are required to make sites work properly (“site cookies”) while others (“tracking cookies”) are used by the third party ad network in which that site participates to recognize that browser across multiple sites participating in the ad network, and thus create a “profile” of what the user might be interested in. Even though such profiles themselves are anonymous, many privacy advocates have pointed to four reasons why online profiling is becoming “too invasive:” (i) It is sometimes possible to infer the actual identity of the user; (ii) though all browsers allow users to opt-out of tracking by “cleaning out” their tracking cookies, a website may be able to restore deleted tracking cookies through the use of cookie alternatives such as “Flash cookies”; (iii) certain vulnerabilities in current browser design make it theoretically possible to “sniff” a user’s browsing history, cache or bookmarks; and (iv) the use of “packet inspection” by Internet Service Providers (ISPs) (instead of the use of cookies) to track online browsing amounts to illegal wiretapping.

The other concerns expressed by the advocates of regulation vary significantly. Some fear that browsing profiles could be captured by hackers, somehow associated with personally identifying information, and used for identity theft. These advocates demand limits on data retention as well as data security mandates. Others demand that users have access to their own profiles—a goal inherently in tension with data security. Most share a vague queasiness about “being tracked” and about advertising in general, while downplaying the effectiveness of self-regulation or user self-help.

Perhaps most legitimately, others fear that the real “Big Brother”—the government—will gain access to a “honeypot” of surveillance data that might be associated with individual users. A variety of other solutions have been proposed to what is, for the most part, a poorly defined problem, including a government-run “Do Not Track” registry to make it easier for users to block tracking cookies; mandating opt-in for some or all forms of profiling; and banning completely the collection of tracking data about sensitive subjects, cross-referencing of data sets, and use of packet inspection data for OBA.

The Less Restrictive Means: A Layered Approach

But how should policymakers decide which, if any, of these interventions are really necessary–or would even be effective? Ironically, those who demand immediate OBA regulation to protect user privacy are often the first to insist on less burdensome approaches whenever a policy “problem” involves purely non-commercial speech. For example, emphasizing personal and parental responsibility is often favored as the more sensible approach to dealing with free speech and child protection concerns. But, as Chapman University Law Professor Tom Bell has asked, why not apply the same standard across the board? Why not expect those especially privacy-sensitive users who object to OBA to do something about it? To the extent effective self-help privacy tools exist, they provide a means of solving policy problems that is not only “less restrictive” than government regulation but generally more effective and customizable as well. Why settle for one-size-fits-all solutions of incomplete effectiveness when users can quite easily and effectively manage their own privacy? Indeed, those who advocate personal responsibility and industry self-regulatory approaches to free speech and child protection issues should be advancing the same position with regards to privacy.

Fortunately, a wide variety of self-help tools and “technologies of evasion” are readily available to all users and can easily thwart traditional cookie-based tracking, as well as more sophisticated tracking technologies such as packet inspection. While cookie management tools that allow users to delete their cookies have been standard in browsers for some time, the latest generation of browsers incorporates far more advanced control over what kind of cookies browsers will accept from websites in the first place. Furthermore,  the extensible nature of modern browsers allows any freelance software developer who sees a way to improve a browser to do so by writing an add-on that “plugs in” to the browser using standard programming interfaces designed by each browser developer.  Many such add-ons are wildly popular, but even those users who never install a single one benefit from the acceleration of browser evolution made possible by add-ons.  We will be documenting examples of these tools in our upcoming Special Report and in an ongoing  series of blog essays.

The Benefits of Smarter Advertising

The “free” Internet economy is based on a simple value exchange: Users get access to an ever-expanding collection of content and services at no cost from websites that are able to generate revenue from “eyeballs” on their pages by selling space on their sites to advertisers, usually through ad networks. The smarter that advertising, the more free content and services it can support. This is the same value exchange that has supported free, over-the-air television and radio content for decades. The only difference is technological: Because websites can connect directly with the user, they need not rely on crude profiling tools such as Nielsen ratings.

There are larger economic benefits of smarter online advertising. First, it makes the overall economy more open and competitive by allowing small market entrants to reach consumers with messages about their products. Second, those who attack the use of packet inspection by ISPs for OBA fail to see that it is precisely the kind of “game-changer” that could disrupt Google’s currently dominant market position. Third, the involvement of ISPs in OBA could help defer broadband costs: Even if OBA revenue does not completely subsidize monthly service costs, smarter advertising could at least keep prices in check and potentially lower them significantly going forward.

But smarter advertising isn’t just about selling products or services. It is ultimately about making all kinds of speech more cost-effective. The ability to “target” listeners more narrowly also increases the ability of political and other not-for-profit speakers to communicate their messages. In short, smarter advertising means more voices, more choices, and more speech. The line between “advertising” and “content” is already blurring rapidly, as the technologies used to customize advertising are also used to customize webpages and ad networks themselves are used to deliver content.

The Larger Implications of Potential Regulation

As if reducing the advertising revenue generated by each web ad didn’t do enough to reduce the total amount of funding for free web content and services, government regulation of targeted online advertising could reduce advertising revenues even further by aggravating the problem of adblocking in two ways. First, the less relevant ads are, the more annoying users will find them, and the more likely users are to try to block them. Increased relevance is perhaps the most important remedy for adblocking and the best way to maintain the implicit value exchange that currently supports free Internet content and services

Second, regulation could short-circuit the eternal battle of technological one-upmanship between online advertisers and those users who rely on the technologies of evasion to “opt-out” of seeing ads or being tracked. Such privacy-conscious users are “free-riding” off of those users who don’t opt-out, since (at present) they generally don’t lose access to the free content and services supported by the targeted advertisements that other users do see. The user who blocks tracking, but not ads, is still free-riding off those users who don’t opt-out of tracking. On a large enough scale, such self-help has the potential to disrupt the value exchange of the Internet, just as automatic commercial-skipping has already disrupted the value exchange of television. As with all “Spy v. Spy” battles, this long-term trend is inevitable: As more sophisticated technologies of evasion are incorporated seamlessly into browsers and can be used without significantly degrading the browsing experience, their use will become increasingly mainstream. But ultimately, just as with television commercial-skipping, market forces can and will, if permitted, respond through technological means and the development of new business models. Today’s implicit quid pro quo may become, of necessity, explicit: Websites and ad networks will have to find increasingly creative ways to grant access to certain content and services for users who do not block ads or the tracking that makes ad space more valuable. Policymakers should take care not to ban such technologies or cripple such business models (e.g., through requiring opt-in), which may rely on more sophisticated forms of targeting such as the use of packet inspection data.

As users face an increasingly clear choice between (i) getting content and services for free supported by behavioral advertising and (ii) paying to receive those same services and content without tracking or even without ads altogether, policymakers will finally see whether users are really as bothered by profiling as the advocates of OBA regulation insist. Given the ongoing and widespread replacement of fee- or subscription-supported web business models with ad-supported models, it seems likely that the vast majority of consumers will continue to choose ad-supported models, including profiling.

Conclusion

The questions raised above—about the harm that supposedly requires intervention, the availability of less restrictive means, and the cost/benefit analysis of regulation—are vital considerations for the future of the Internet. Indeed, if smarter online advertising will not fund the Internet’s future, what will? As both the desire for “free” services and content and the need for bandwidth expand, OBA has the potential to offer important new revenue sources that can help support the entire ecosystem of online content creation and service innovation, while also providing a new source of funding for Internet infrastructure and making ads less annoying and more informative. That would certainly seem preferable to increased user fees or other “pay-per-view” pricing models for Internet content and services.

But looming legislative and regulatory action could stop all of that by replacing the current regime—in which the FTC merely enforces industry self-regulatory policies—with one in which the government preemptively dictates how data may be collected and used. The more enlightened approach is a “layered” approach to privacy protection that combines industry self-regulation, enforcement of industry-established privacy policies, consumer education, and user “self-help” solutions. These and other issues will be addressed in greater detail in our upcoming PFF Special Report.