Spontaneous Order Is Counter-Intuitive

by on September 24, 2006 · 16 comments

Luis Villa has an interesting post about the evolving understanding of open source software:

I’ve long thought that in open source software we are seeing a trend away from trust in an institution (think: Microsoft) and towards trust in ‘good luck’- i.e., in the statistical likelihood that if you fall, someone will catch you. In open source, this is most manifest in support- instead of calling a 1-800 # (where someone is guaranteed to help you, as long as you’re willing to be on hold for ages and pay sometimes very high charges), one emails a list, where no one is responsible for you, but yet a great percentage of the time, someone will answer anyway. There is no guarantee, but community practices evolve to make it statistically likely that help (or bug fixing, or whatever) will occur. The internet makes this possible- whereas in the past if you wanted free advice, you had to have a close friend with the right skills free time, you can now draw from a much broader pool of people. If that pool is large enough (and in software, it appears to be) then it is a statistical matter that one of them is likely to have both the right skills and the right amount of free time.

Clay Shirky today makes an argument that this isn’t just something that is occurring in open source, but is hitting other fields of expertise as well: “My belief is that Wikipedia’s success dramatizes instead a change in the nature of authority, moving from trust inhering in guarantees offered by institutions to probabilities created by processes.” Instead of referring to a known expert to get at knowledge, you can ask Wikipedia- which is the output of a dialectic process which may fail in specific instances but which Clay seems to suggest can be trusted more than any one institution’s processes in the long run.

This is an excellent point, but it’s actually not a new one. Two examples that immediately spring to mind are Darwin’s Origin of the Species and Friederich Hayek’s The Road to Serfdom (and, more specifically, his subsequent essay “The Use of Knowledge in Society” ). Darwin and Hayek each described decentralized processes in which the correctness of the result is produced by statistical processes, rather than by the good judgment of a trusted authority.


In Darwin’s case, of course, the trusted authority was God, and the statistical process was natural selection. In Hayek’s case, the trusted authority was the state, whom socialist intellectuals believed could plan a nation’s economy better than the chaos of the market. In both cases, the central insight was that the problem at hand was too big for any one intelligence to solve, but they explained how the problem could be solved by impersonal, statistical processes that might fail in many individual cases, but in the long run will find better solutions that centralized planning could.

It seems to me that the critics of peer produced works have arguments that closely mirror the arguments of Darwin and Hayek’s critics. A few quick examples:

  • Denying that the decentralized process can work at all (or claiming that their occasional successes were flukes) despite the fact that it obviously does. There was a great deal of worry in the middle of the 20th Century about the likelihood that the Soviet economy would out-produce the American one. Likewise, to this day, there are people who reject Darwin’s hypothesis, despite the wealth of evidence in the fossil record. Peoples’ claims that open source software is “unsustainable”, despite the fact that it’s been growing rapidly for decades, are in the same vein.
  • Insisting that the process will only work in small-scale or peripheral cases, but that it doesn’t scale as well as centralized mechanisms. Many Darwinian critics argued that evolution works at the micro-level–that the peppered moth does indeed change colors–but that some other theory is needed to explain the emergence of new species. The socialists of the early 20th Century argued that markets work all right for a simple agrarian society, but that for a modern industrial society greater central planning was needed. By the same token, it’s sometimes argued that open source software is good for databases and web servers, but that for browsers, or office suites, or whatever, more centralized development processes are needed. Of course, these predictions don’t have a very good track record, as peer-produced products continue to succeed in new markets.
  • Focusing on failures in individual cases, while ignoring that the average is moving steadily in the right direction. Critics often point to Enron as an example of what’s wrong with markets. But the important point about Enron is that they’re not in business any more. No one ever claimed that every business in a free market is honest. What we claim is that free markets are more likely than other economic systems to find and weed out the dishonest businessmen. Likewise, when people point to particular open source projects that aren’t very good, or to particular errors in Wikipedia pages, they’re missing the forest for the trees. No one ever claimed that every Wikipedia page would be 100 percent accurate. What we claim is that Wikipedia pages tend to steadily get better (more accurate, more comprehensive, more timely) over time, and so in the long run, we should expect it to be better than more traditionally edited publications.
  • Demanding a detailed description of how the process will solve a particular problem. Critics of Darwin frequently charge that Darwin couldn’t explain the emergence of particular structures like the eye, the wing, or the bacterial flagella. Biologists actually do have pretty good explanations of how many of these things developed, but the broader point is that evolution often relies on serendipity, in which a body part that originally served one purpose becomes used for another purpose instead. So absent direct evidence from the fossil record, it’s often difficult to figure out the exact evolutionary path any given organism took. Likewise, critics of school choice like do demand detailed descriptions of what a free market in education would look like, ignoring the fact that the whole point of having a free market in education is that we need the market process to figure out how to create better schools. In precisely the same fashion, a lot of criticisms of Wikipedia rest implicitly on the fact that we can’t predict in advance who will contribute to any given Wikipedia article: “We know that the Brittanica article on the French Revolution was written by an expert. How do I know the Wikipedia article on the French Revolution wasn’t written by a 6th grader who didn’t know what he’s talking about?” Which, of course, we don’t. All we know is that as Wikipedia becomes more popular, the average number of experts that will review any given article will increase. But we can’t necessarily predict who they’ll be or when or how they’ll come across the article in any given case.

    I think the root cause of these kinds of fallacious arguments is that our brains are not wired to think social systems in statistical terms. In the primitive tribes that shaped the human brain, decisions were made by a tribal leader, and peoples’ confidence in the correctness of the decisions was based on their personal trust in the judgment of the chief. We’re conditioned to ask who’s in charge, and so we find the notion that nobody is in charge to be very disconcerting.

    I also think the socialist example illustrates another important point: even smart and well-educated people get these sorts of questions wrong. Hayek dedicated his book to “socialists of all parties” because the leading intellectuals of the 1940s tended to be socialists. Spontaneous order is counter-intuitive, whether we’re talking about life, markets, or Wikipedia. As peer production becomes a more important part of the economy, explaining how they work to non-participants will be an increasingly important challenge.

    SHARE: 16 comments

    • Tim Lee / Timothy B. Lee (Contributor, 2004-2009) is an adjunct scholar at the Cato Institute. He is currently a PhD student and a member of the Center for Information Technology Policy at Princeton University. He contributes regularly to a variety of online publications, including Ars Technica, Techdirt, Cato @ Liberty, and The Angry Blog. He has been a Mac bigot since 1984, a Unix, vi, and Perl bigot since 1998, and a sworn enemy of HTML-formatted email for as long as certain companies have thought that was a good idea. You can reach him by email at leex1008@umn.edu.

    Comments on this entry are closed.

    Previous post:

    Next post: