Spontaneous Order Is Counter-Intuitive

by on September 24, 2006 · 16 comments

Luis Villa has an interesting post about the evolving understanding of open source software:

I’ve long thought that in open source software we are seeing a trend away from trust in an institution (think: Microsoft) and towards trust in ‘good luck’- i.e., in the statistical likelihood that if you fall, someone will catch you. In open source, this is most manifest in support- instead of calling a 1-800 # (where someone is guaranteed to help you, as long as you’re willing to be on hold for ages and pay sometimes very high charges), one emails a list, where no one is responsible for you, but yet a great percentage of the time, someone will answer anyway. There is no guarantee, but community practices evolve to make it statistically likely that help (or bug fixing, or whatever) will occur. The internet makes this possible- whereas in the past if you wanted free advice, you had to have a close friend with the right skills free time, you can now draw from a much broader pool of people. If that pool is large enough (and in software, it appears to be) then it is a statistical matter that one of them is likely to have both the right skills and the right amount of free time.

Clay Shirky today makes an argument that this isn’t just something that is occurring in open source, but is hitting other fields of expertise as well: “My belief is that Wikipedia’s success dramatizes instead a change in the nature of authority, moving from trust inhering in guarantees offered by institutions to probabilities created by processes.” Instead of referring to a known expert to get at knowledge, you can ask Wikipedia- which is the output of a dialectic process which may fail in specific instances but which Clay seems to suggest can be trusted more than any one institution’s processes in the long run.

This is an excellent point, but it’s actually not a new one. Two examples that immediately spring to mind are Darwin’s Origin of the Species and Friederich Hayek’s The Road to Serfdom (and, more specifically, his subsequent essay “The Use of Knowledge in Society” ). Darwin and Hayek each described decentralized processes in which the correctness of the result is produced by statistical processes, rather than by the good judgment of a trusted authority.


In Darwin’s case, of course, the trusted authority was God, and the statistical process was natural selection. In Hayek’s case, the trusted authority was the state, whom socialist intellectuals believed could plan a nation’s economy better than the chaos of the market. In both cases, the central insight was that the problem at hand was too big for any one intelligence to solve, but they explained how the problem could be solved by impersonal, statistical processes that might fail in many individual cases, but in the long run will find better solutions that centralized planning could.

It seems to me that the critics of peer produced works have arguments that closely mirror the arguments of Darwin and Hayek’s critics. A few quick examples:

  • Denying that the decentralized process can work at all (or claiming that their occasional successes were flukes) despite the fact that it obviously does. There was a great deal of worry in the middle of the 20th Century about the likelihood that the Soviet economy would out-produce the American one. Likewise, to this day, there are people who reject Darwin’s hypothesis, despite the wealth of evidence in the fossil record. Peoples’ claims that open source software is “unsustainable”, despite the fact that it’s been growing rapidly for decades, are in the same vein.
  • Insisting that the process will only work in small-scale or peripheral cases, but that it doesn’t scale as well as centralized mechanisms. Many Darwinian critics argued that evolution works at the micro-level–that the peppered moth does indeed change colors–but that some other theory is needed to explain the emergence of new species. The socialists of the early 20th Century argued that markets work all right for a simple agrarian society, but that for a modern industrial society greater central planning was needed. By the same token, it’s sometimes argued that open source software is good for databases and web servers, but that for browsers, or office suites, or whatever, more centralized development processes are needed. Of course, these predictions don’t have a very good track record, as peer-produced products continue to succeed in new markets.
  • Focusing on failures in individual cases, while ignoring that the average is moving steadily in the right direction. Critics often point to Enron as an example of what’s wrong with markets. But the important point about Enron is that they’re not in business any more. No one ever claimed that every business in a free market is honest. What we claim is that free markets are more likely than other economic systems to find and weed out the dishonest businessmen. Likewise, when people point to particular open source projects that aren’t very good, or to particular errors in Wikipedia pages, they’re missing the forest for the trees. No one ever claimed that every Wikipedia page would be 100 percent accurate. What we claim is that Wikipedia pages tend to steadily get better (more accurate, more comprehensive, more timely) over time, and so in the long run, we should expect it to be better than more traditionally edited publications.
  • Demanding a detailed description of how the process will solve a particular problem. Critics of Darwin frequently charge that Darwin couldn’t explain the emergence of particular structures like the eye, the wing, or the bacterial flagella. Biologists actually do have pretty good explanations of how many of these things developed, but the broader point is that evolution often relies on serendipity, in which a body part that originally served one purpose becomes used for another purpose instead. So absent direct evidence from the fossil record, it’s often difficult to figure out the exact evolutionary path any given organism took. Likewise, critics of school choice like do demand detailed descriptions of what a free market in education would look like, ignoring the fact that the whole point of having a free market in education is that we need the market process to figure out how to create better schools. In precisely the same fashion, a lot of criticisms of Wikipedia rest implicitly on the fact that we can’t predict in advance who will contribute to any given Wikipedia article: “We know that the Brittanica article on the French Revolution was written by an expert. How do I know the Wikipedia article on the French Revolution wasn’t written by a 6th grader who didn’t know what he’s talking about?” Which, of course, we don’t. All we know is that as Wikipedia becomes more popular, the average number of experts that will review any given article will increase. But we can’t necessarily predict who they’ll be or when or how they’ll come across the article in any given case.

    I think the root cause of these kinds of fallacious arguments is that our brains are not wired to think social systems in statistical terms. In the primitive tribes that shaped the human brain, decisions were made by a tribal leader, and peoples’ confidence in the correctness of the decisions was based on their personal trust in the judgment of the chief. We’re conditioned to ask who’s in charge, and so we find the notion that nobody is in charge to be very disconcerting.

    I also think the socialist example illustrates another important point: even smart and well-educated people get these sorts of questions wrong. Hayek dedicated his book to “socialists of all parties” because the leading intellectuals of the 1940s tended to be socialists. Spontaneous order is counter-intuitive, whether we’re talking about life, markets, or Wikipedia. As peer production becomes a more important part of the economy, explaining how they work to non-participants will be an increasingly important challenge.

    • http://www@pff.org Noel Le

      Per professors Chesbrough and Wu, we already see decentralized production; partially through IP enabling firms to specialize and then license their IP assets. I hand it to you though- based on your posts one would have to refute Darwin and Hayek to disagree on the potential of FOSS:) Or one can just say he disagrees w your reading of them:) Based on your post, FOSS should beat proprietary counterparts in most technology spaces (like office apps, accessibility features) and Wikipedia should be 100 times authoratative.

    • Noel Le

      Per professors Chesbrough and Wu, we already see decentralized production; partially through IP enabling firms to specialize and then license their IP assets. I hand it to you though- based on your posts one would have to refute Darwin and Hayek to disagree on the potential of FOSS:) Or one can just say he disagrees w your reading of them:) Based on your post, FOSS should beat proprietary counterparts in most technology spaces (like office apps, accessibility features) and Wikipedia should be 100 times authoratative.

    • http://tieguy.org/ Luis Villa

      Tim: eeeenteresting. Darwin definitely should have come to mind; I admit I’ve never read any Hayek. (Anyone care to recommend a good intro to Hayek and/or Hayekian thought as it is currently recieved?) You’re right, of course, and your analysis is dead on.

      I might note that perhaps this is coming more to the fore at the moment because evolution and economies work at very large scales that not only are difficult to grasp, but which have relatively low impact on individual decision making. I have faith that evolution works, but it really doesn’t matter, because no matter what I think about evolution, it isn’t going to change my day-to-day decision making. In contrast, whether I have faith in the reliability of wikipedia does change my day-to-day behaviors. So that makes coming to grips with these kinds of systems more relevant than they had been in the past.

    • http://peerworks.org/ Jed Harris

      I call thinking that relies on these statistical mechanisms “population thinking” and I do believe that it is hard for people to accept, but conversely that these mechanisms are far more robust and effective in a lot of domains than deterministic mechanisms.

      As a curious sidelight, the standard model for market economics, the neo-classical Walrasian synthesis (actually formalized by Arrow and deBreu), uses a deterministic mechanism, tattonement, rather than statistical trading. Ironically, this makes the Walrasian model very fragile and computationally intractable, while the statistical model has more robust and extensive equilibria (which are also closer to real world data) and is computationally trivial. The statistical methods existed when the Walrasian model was developed and formalized, and the economists chose deterministic ones instead.

      So apparently even theorists of naturally statistical domains, like economics, have a deep aversion to population thinking.

      Similarly AI and linguistics tried to use deterministic mechanisms, but they are finally coming around to a thoroughly statistical approach, since that actually works and the deterministic one didn’t.

    • http://www.techliberation.com/ Tim

      Luis,

      “The Use of Knowledge in Society” is one of his more famous essays, and it’s available online here. That’ll give you a basic introduction to his most famous thesis–that the price system is a decentralized mechanism for coordinating individuals with dispersed information.

      I imagine you’re too busy with law school at the moment to read longer works, but next time you have some time on your hands The Constitution of Liberty is probably the best introduction to liberal (in the classical sense) political philosophy as you’re likely to find.

      The Road to Serfdom is his most famous work, and although it’s well written and historically important, it has limited relevance today because he was arguing against a hard-core socialist philosophy that hardly anyone subscribes to these days.

    • http://tieguy.org/ Luis Villa

      Tim: eeeenteresting. Darwin definitely should have come to mind; I admit I’ve never read any Hayek. (Anyone care to recommend a good intro to Hayek and/or Hayekian thought as it is currently recieved?) You’re right, of course, and your analysis is dead on.

      I might note that perhaps this is coming more to the fore at the moment because evolution and economies work at very large scales that not only are difficult to grasp, but which have relatively low impact on individual decision making. I have faith that evolution works, but it really doesn’t matter, because no matter what I think about evolution, it isn’t going to change my day-to-day decision making. In contrast, whether I have faith in the reliability of wikipedia does change my day-to-day behaviors. So that makes coming to grips with these kinds of systems more relevant than they had been in the past.

    • http://peerworks.org/ Jed Harris

      I call thinking that relies on these statistical mechanisms “population thinking” and I do believe that it is hard for people to accept, but conversely that these mechanisms are far more robust and effective in a lot of domains than deterministic mechanisms.

      As a curious sidelight, the standard model for market economics, the neo-classical Walrasian synthesis (actually formalized by Arrow and deBreu), uses a deterministic mechanism, tattonement, rather than statistical trading. Ironically, this makes the Walrasian model very fragile and computationally intractable, while the statistical model has more robust and extensive equilibria (which are also closer to real world data) and is computationally trivial. The statistical methods existed when the Walrasian model was developed and formalized, and the economists chose deterministic ones instead.

      So apparently even theorists of naturally statistical domains, like economics, have a deep aversion to population thinking.

      Similarly AI and linguistics tried to use deterministic mechanisms, but they are finally coming around to a thoroughly statistical approach, since that actually works and the deterministic one didn’t.

    • http://weblog.ipcentral.info/ Noel Le

      Tim, again, there is a technology industry right in front of us. Didn’t you work in it.

      Based on your analogies, why is FOSS (which has been around longer than most proprietary technologies) not leading in every category. Any why do you continue to push for bending US policy to accommodate it. Darwin and Hayek will not help unless FOSS does something for itself in the market (which I’m supportive of, but remain skeptical). The limitations to FOSS, as far as I see, are principles on which it distinqishes itself from propetiary industry. FOSS should learn to be more practical now that its beyond the phase of proof of concept.

      Whats the most important entity in technological innovation. The market, right? The market: meaning consumers, those who invest, buy and decide the societal value of technologies. So, whats the need for these silly metaphors from Darwin, Hayek, etc, when there is a market right in front of us- unless you need to explain away sub-par market performance with philosophical justifications. Well, Darwin and Hayek will not help FOSS build market share. But still, this discussion is interesting.

      Jed, population thinking is different from the “market.” Thats not an analogy I see you getting at, but one I believe Tim hinted to. A lot of misconception arises when the concepts of population thinking and the market are compared. The market resides on incentives and competition. Population thinking, well, its the driving force behind spam, patent trolls and other societal anomolies. Some claim it is the virtue of things like Wikipedia and FOSS projects, but- it probably does not apply to FOSS development. Wikipedia perhaps..

      The concept of Darwin’s evolution aside (I won’t delve into it, but note that it has no predictive value), there is conflation of two distinct principles when Darwinism is compared to market economics. Darwinist theory predicts that the best technologies naturally arise and lead follow-on and peripheral innovations. Those who apply Darwinist theory *usually* equate the “strong” with the “best technologies.” In reality, the best technologies per se are not always those that win. It’s the technologies that are most adaptive to consumer and market demand that win.

    • http://www.techliberation.com/ Tim

      Luis,

      “The Use of Knowledge in Society” is one of his more famous essays, and it’s available online here. That’ll give you a basic introduction to his most famous thesis–that the price system is a decentralized mechanism for coordinating individuals with dispersed information.

      I imagine you’re too busy with law school at the moment to read longer works, but next time you have some time on your hands The Constitution of Liberty is probably the best introduction to liberal (in the classical sense) political philosophy as you’re likely to find.

      The Road to Serfdom is his most famous work, and although it’s well written and historically important, it has limited relevance today because he was arguing against a hard-core socialist philosophy that hardly anyone subscribes to these days.

    • http://weblog.ipcentral.info/ Noel Le

      Tim, again, there is a technology industry right in front of us. Didn’t you work in it.

      Based on your analogies, why is FOSS (which has been around longer than most proprietary technologies) not leading in every category. Any why do you continue to push for bending US policy to accommodate it. Darwin and Hayek will not help unless FOSS does something for itself in the market (which I’m supportive of, but remain skeptical). The limitations to FOSS, as far as I see, are principles on which it distinqishes itself from propetiary industry. FOSS should learn to be more practical now that its beyond the phase of proof of concept.

      Whats the most important entity in technological innovation. The market, right? The market: meaning consumers, those who invest, buy and decide the societal value of technologies. So, whats the need for these silly metaphors from Darwin, Hayek, etc, when there is a market right in front of us- unless you need to explain away sub-par market performance with philosophical justifications. Well, Darwin and Hayek will not help FOSS build market share. But still, this discussion is interesting.

      Jed, population thinking is different from the “market.” Thats not an analogy I see you getting at, but one I believe Tim hinted to. A lot of misconception arises when the concepts of population thinking and the market are compared. The market resides on incentives and competition. Population thinking, well, its the driving force behind spam, patent trolls and other societal anomolies. Some claim it is the virtue of things like Wikipedia and FOSS projects, but- it probably does not apply to FOSS development. Wikipedia perhaps..

      The concept of Darwin’s evolution aside (I won’t delve into it, but note that it has no predictive value), there is conflation of two distinct principles when Darwinism is compared to market economics. Darwinist theory predicts that the best technologies naturally arise and lead follow-on and peripheral innovations. Those who apply Darwinist theory *usually* equate the “strong” with the “best technologies.” In reality, the best technologies per se are not always those that win. It’s the technologies that are most adaptive to consumer and market demand that win.

    • http://tieguy.org/ Luis Villa

      Thanks for the pointer, Tim. I’ll add Constitution of Liberty to the pile of ‘things to read when I have spare time’. I tell myself the comforting lie that ‘spare time’ may occur before 2010.

    • http://tieguy.org/ Luis Villa

      Thanks for the pointer, Tim. I’ll add Constitution of Liberty to the pile of ‘things to read when I have spare time’. I tell myself the comforting lie that ‘spare time’ may occur before 2010.

    • http://peerworks.org/ Jed Harris

      Noel’s claim that evolution has no predictive power is addressed in this summary.

      I’m not sure why Noel believes that “population thinking is different from the ‘market’”. Evolutionary game theory arose from evolutionary theory, and is now heavily applied in economics. A detailed study of related explanatory mechanisms was done by Sunny Auyang in Foundations of Complex System Theories (a fascinating book about the mechanisms of spontaneous order, by the way).

      As for the relative success of FOSS — the remarkable thing is that FOSS development processes are a viable way of solving some very large, complex coordination problems, without institutional enforcement mechanisms — either bureaucracy or contracts. This was considered impossible in conventional social and economic models. Evolutionary game theory approaches do better at modeling FOSS type coordination.

    • http://peerworks.org/ Jed Harris

      Noel’s claim that evolution has no predictive power is addressed in this summary.

      I’m not sure why Noel believes that “population thinking is different from the ‘market’”. Evolutionary game theory arose from evolutionary theory, and is now heavily applied in economics. A detailed study of related explanatory mechanisms was done by Sunny Auyang in Foundations of Complex System Theories (a fascinating book about the mechanisms of spontaneous order, by the way).

      As for the relative success of FOSS — the remarkable thing is that FOSS development processes are a viable way of solving some very large, complex coordination problems, without institutional enforcement mechanisms — either bureaucracy or contracts. This was considered impossible in conventional social and economic models. Evolutionary game theory approaches do better at modeling FOSS type coordination.

    • http://enigmafoundry.wordpress.com/ enigma_foundry

      I would highly reccommend Cosma Shalizi’s notebooks for those interested in self-organization.

      Or play around with some cellular automata.
      here.

    • http://enigmafoundry.wordpress.com eee_eff

      I would highly reccommend Cosma Shalizi’s notebooks for those interested in self-organization.

      Or play around with some cellular automata.
      here.

    Previous post:

    Next post: