I guess the search for market failure in the privacy area is interesting to me. I wrote about it the other week too. It’s nice that those who prefer regulation feel obligated to justify that preference. It’s acknowledgment of the fact, increasingly well-accepted worldwide, that functioning free markets do a better job of discovering and satisfying consumers’ interests than any other method for organizing societies’ resources.

A recent market failure blog post called “Privacy and the Market for Lemons, or How Websites Are Like Used Cars,” seems to have piqued Adam’s interest. (See the comments.) In it, privacy and anonymity researcher Arvind Narayanan makes the case for privacy market failure. (Evidently, it’s an argument that others have made before.)

“In the realm of online privacy and data collection,” he says, “information asymmetry results from a serious lack of transparency around privacy policies. The website or service provider knows what happens to data that’s collected, but the user generally doesn’t.” Several economic, architectural, cognitive and regulatory limitations/flaws “have led to a well-documented market failure—there’s an arms race to use all means possible to entice users to give up more information, as well as to collect it passively through ever-more intrusive means.”

Alas, there’s no link at “well-documented.” I would like to see that documentation. But more importantly, what Narayanan appears to be speaking of as market failure—an arms race to get more information from Web users—is not one. That’s market action that Narayanan doesn’t like.

So where’s the market failure? Is it in the “impossibility” of navigating the Web aware of the privacy policies of each Web site? That’s certainly very hard to do. The premise is that markets need information symmetry for them to function. There’s a profoundly difficult question lurking here: Just how symmetrical does information have to be? 42?

Luckily, markets don’t require information symmetry. They work because both parties to nearly every transaction make themselves better off by their own reckoning. In the aggregate, market transactions make everyone better off without making anyone worse off. As with the information symmetry ideal, there’s no need for perfection in transacting, either. If some actors regularly make “bad” trades, but still end up better off on the whole, the market is working for them, too. (Their consistent failure to perceive their own self-interest may ultimately require them to seek the charity of those of us who do, but think carefully before depriving all consumers of the market’s opportunities just to protect some small number of incompetents.)

This logic is not defeated when a consumer gets less in a transaction than he or she thought, or bears a cost he or she didn’t know about. (“I thought I was getting a car that gets 35 mpg, but it only gets 31,” or “I thought the site would collect no information, but it associated my browser and IP address with an interest in golf.”) It is only defeated when the systematic operation of the market will reach sub-optimal results.

Will a series of transactions drive someone to their death of starvation or some other harm or ruin? Will the gears of commerce freeze? Market failure occurs when the rules, signals, and sanctions in and around a given marketplace would cause preference- and profit-maximizing actors to reach a sub-optimal outcome.

Narayanan believes there is a sub-optimal outcome—people are giving up too much privacy unaware of the consequences. I agree that people do not protect their privacy enough online. But correlating market behavior we rue to an “information asymmetry,” where every last detail of what happens to data isn’t clear, has not made the case for market failure.

The same “asymmetry” exists in the real world. I have no idea what my neighbors will do with the information they observe about my comings and goings or what my friends will say behind my back. Does this social market failure demand or justify a regulatory solution? Should I get a privacy notice detailing their plans? No. I’m on notice that stuff like that can happen with anyone and everyone I encounter. I go outside, subjecting myself to observation, because I am better off for doing so even though I pay an unknown privacy cost. Likely, Web surfers are similarly better off by their own reckoning despite paying a similar, unquantified privacy cost. The fact that they could pay less is imperfection, not failure.

As you can see, I love analogical thinking, but the analogy between the online environment and a used car market does not work for me. I also don’t see that the theory of the “lemons market” has borne itself out, either online or in the market for used cars. A lemons market should lead to stagnation—a no-trade equilibrium—but we have a robust online information environment and used car sales continue to flourish (in part because of market innovations).

In the end, Narayanan’s post is . . . a lemon. That is not a market failure—just a computer scientist who I think has hashed the economics a bit. It’s not my field either, so I welcome comments that educate and elucidate things for all of us.

One of the reasons that so many of us here take issue with proposals to expand regulation of communications, broadband, and media markets is because we have studied the horrendous inefficiencies of economic regulation in practice. We oppose regulatory proposals not because of a “blind faith” in free markets, but because we understand that even when markets stumble they correct themselves quicker and more efficiently than regulatory systems do. One can profess the supposed theoretical benefits of enlightened “public interest” regulation all they want, but the facts are the facts. And the facts do not support the proposition that government regulation generally enhances consumer welfare.

In that regard, Tim Lee’s new Net neutrality report for Cato does a nice job of surveying some of the past unintended consequences of regulation. Also, even though it is now 10 years old, I highly recommend “Economic Deregulation and Customer Choice” by Jerry Ellig and Robert Crandall. It’s an outstanding overview of why economic regulation of various industries failed consumers so miserably in the past.

But if you want even more shocking proof of how horrendously inefficient communications regulation can be in practice, then you must read my PFF colleague Barbara Esbin’s two essays this week on the Universal Service Fund (USF): “The High Cost of USF Support,” and “More FCC Support Fund Follies.” In these two essays, Esbin walks the reader through various grim reports and statistics that have been released recently documenting the failures of the USF.

Her first essay notes how a recent FCC Inspector General report found that the USF “High Cost” fund is spiraling out of control. According to a FCC press release, that report found that “a program is at risk if the erroneous payment rate exceeds 2.5% and the amount of erroneous payments is greater than $10 million. The estimated erroneous payment rate for the High Cost Program (“HCP”) was 23.3%. The previous estimate was 16.6%. Total estimated erroneous payments were $ 971.2 million as compared with the previous estimate of erroneous payments of $617.8 million. Accordingly, the FCC-OIG concluded that the High Cost Fund program is “at risk” under applicable [..] criteria.”

Esbin puts these shocking results in perspective:

“At risk” is a surely a euphemism for a program that loses in “erroneous payments” nearly one out of every four dollars collected from telephone subscribers. In 2007, pursuant to FCC rules, telephone consumers were effectively taxed over $4 billion for the high-cost portion of the USF. Thus, nearly $1 billion dollars of subscriber money went out the door in “erroneous payments.” As the report makes clear, erroneous payments include both over- and underpayments, and also instances where the agency is unable to discern whether a payment was proper as a result of “lack of documentation.” The report’s conclusions state that the “rate of improper overpayments is 22.8%, and the proportion of improper overpayments out of total improper payments is 98.2%.” To be considered “erroneous,” an payment “need not be the result of fraudulent misrepresentation, or a corrupt administrative process.” “Nor does it necessarily exclude those factors as potential causes of erroneous payments.” Significantly, nor are “the erroneous payments . . . necessarily recoverable from recipients by process of law.” Fabulous. Not only has nearly $1 billion in erroneous overpayments gone missing, but even if final audits indicate where it has gone, it may not be recoverable! Among the interesting results of this preliminary report are the identified causes of erroneous payments. According to Table 2 of the report, 50% of the causes of erroneous payments can be attributed nearly equally to two factors: either “Inadequate Documentation” (25.3%) or “Inadequate Auditee Processes and/or Policies and Procedures” (24.6%). Another 10% “Disregarded FCC Rules” and 12% had “Applicant/Auditee Weak Internal Controls.” That is, roughly 75% of the erroneous overpayments can be attributed to poor bookkeeping, inadequate internal controls and “disregard” of FCC rules. This is stunning information. No wonder it made its appearance the day before Thanksgiving.

But wait, things get worse. So much worse. In Esbin’s second essay, she notes that:

On Monday, the OIG released its Semi-Annual Report to Congress, discussing the full range of audit activities conducted from April 1, 2008 to September 30, 2008. Thus we learn that in addition to the loss of nearly $1 billion in erroneous overpayments to the High Cost program, another fund the FCC is ultimately responsible for, the “Telecommunications Relay Service” (TRS) Fund, which provides funds for a variety of telephone transmission services for those with hearing and speech disabilities, also appears to be at risk for substantial overpayments due to the lack of adequate controls. Since 1993, according to the FCC’s website, the Commission’s rules have required that each common carrier providing voice transmission services provide TRS throughout its service area. All providers of interstate telecommunications services contribute to the TRS Fund, and TRS providers recover the costs of providing interstate services from the Fund on a minutes-of-use basis. Intrastate TRS funding is generally administered by the states, although some intrastate TRS offerings are supported by the interstate TRS Fund. The current TRS Fund Administrator is the National Exchange Carrier Association (NECA). Although NECA directly manages the Fund, the FCC sets the Fund size and carrier contribution factor annually and is ultimately responsible for Fund oversight. When the TRS Fund started, it disbursed about $31 million, growing to over $38 million by 1999. Since 1999, the OIG report states that the TRS Fund has increased approximately 50-80% each year, to reach $637 million for the Fund’s fiscal year from July, 2007 to June, 2008. The size of the fund for the current fiscal year is $850 million, a 26% increase over the previous fiscal year. That is, in roughly ten years the TRS Fund has ballooned from $38 million to $850 million! What, if any, other communications service has seen 50-80% growth in costs per year?

Indeed, that is a shocking degree of waste and inefficiency by just about any standard. And Esbin goes on to document specific examples of this waste and inefficiency in action within the TRS Fund. It’s shocking stuff and doesn’t make for pleasant reading if you care about good government.

Barbara is actually much more tempered and tolerant than me when it comes to what to do about all this. She recommends a lot more reform and oversight. If you ask me, however, then entire USF program should be dismantled immediately and any future support deemed necessary should be distributed directly to consumers at the state level in the form of a welfare payment. After all, at root, that’s what universal service is: a communications industry welfare program, but one in which most of the support flows to companies instead of individuals. And that makes it one of the most insanely misguided and inefficient regulatory / subsidization systems known to man. 13 years ago, in one of the very first things that PFF ever published ( The Telecom Revolution: An American Opportunity) I was advocating exactly this sort of a plan along with a dozen other think tank colleagues. (And we also set forth another, less radical reform plan than the “voucher-ize & devolve” plan I favored).

But no one listened. Business as usual continued. And so the endless waste and inefficiencies continue. Somebody will have to remind me how any of this benefits consumer welfare. I can’t see how anyone could make such a case, and I would hope the USF follies serve as a cautionary tale for how the best of intentions are meaningless when it comes to what regulation actually means in practice. Because it sure ain’t pretty.

But hey, it’ll all be different going forward right? We just need to have faith in the media reformistas and the Net neutralitistas.  If we click our heels together enough time and just wish hard enough, all our dreams can come true.

Sure.

Geese are flying overhead. Leaves are orange. The election is over. A historic moment. And I will be optimistic, and hope that although the economics of the moment seems to be a return to things past… to the 1930s, it will turn out to be otherwise, for a good bit is known now that was not known then, whatever one’s ideology.

This column offering thoughts from Europe anticipates a wave of hostility to free markets. Well, that would perhaps not be that much of a change. I will venture far out on a doctrinal limb here, why not, and venture to ask where the free marketers went wrong? [Wait, you mean that they did something wrong? Can that be possible? Surely not]. (There is a good bit that went wrong, of course, that is not the fault of markets or their advocates… the fact that markets are not perfect, problems with rent-seeking, the fondness of the press for dwelling on the negative, and so on). But there have been consistent problems with our presentation, which I diagnose as follows:

-Use of nineteenth century models and rhetoric, and too much movement jargon, much of which is pointlessly disparaging and negative.

-Failure to empathize with people’s real concerns, such as concern about the environment or income disparity. There is the perennial addiction of wonks to Reason-and our awkwardness with emotion that leads us to dismiss it as irrelevant. Makes it look like we don’t care–a false impression, but a real factor none-the-less.

-Specializing in the defense of unpopular causes, whether it is free speech, the super-rich, or the large company of the day. Advocates tend to focus on these causes in the hope of getting attention as contrarians-but as a result the image of advocates for the market becomes identified with unpopular interests, and our energy gets expended in short run battles.

The solution? Well, I’ll save that for another day.

Meanwhile, how about this for a thought? In recognition of the nation’s leaning to the left, I’ll make a concession. Have some social programs. Have all the social programs you want. But there is one thing that I will insist on. Just one thing.

Y’all will have to be ruthlessly honest about how well the programs actually work. About the unintended consequences. About the rules that pile up costs with no benefits. About the forces and factors that lead institutions like public schools or regulatory agencies to fail.

If you can just manage a genuine curiousity about whether the plans that you dream up to help people will actually work, then we’ve got a deal. I promise that if the programs don’t work, we can try something else. Institutional re-design. Heck, maybe even a market.

But I don’t think anyone’s going to go for it.