TCP/IP “More or Less by Fiat?”

by on February 29, 2008 · 24 comments

I agree with most of what Tom says here but I have to quibble with his final paragraph:

Oh, and one more thing: it’s probably worth noting that one of the greatest technical (and economic) triumphs in recent memory — TCP/IP and the suite of other protocols and standards that powers the internet — was designed by having a bunch of really smart engineers get together, execute an RFC process and then issue an ISO standard more or less by fiat. This is not to say that markets can’t help us arrive at good solutions — cable vs. DSL vs. FiOS is a good example of such a market working (or would be if the regulatory picture weren’t so complicated). But it ought to be acknowledged that markets are not always an optimal tool for making technical decisions. In fact there are now pseudo-centralized organizations that take responsibility for many of the technical standards that power our world, and nearly all engineers agree that we’re vastly better off for it.

The history of the Internet is something I’ve been researching lately, so this is a question I know something about. And I think Tom’s only telling part of the story.

It’s certainly true that the TCP/IP protocol suite was developed because some military researchers asked some really smart computer scientists to get together and write a new networking protocol. The spec was revised a few times (probably the most important being splitting things into a TCP layer and an IP layer) and then approved by the military, which proceeded to force every user of the ARPANET to switch to it in January 1983. So far, this is consistent with the “central planning” theory.

The problem is that in February 1983, the Internet had nothing like the ubiquity it enjoys today. I just returned all those books to the library so I can’t cite chapter and verse off the top of my head, but in the mid-1980s there were a ton of competing, proprietary network architectures around. A bunch of telecom companies had gotten together and created a standard called X.25 that was widely used for long-haul networking (including France’s Minitel. IBM, which then dominated the server business, had a proprietary networking architecture called SNA. Other hardware vendors had competing proprietary networking architectures. Novell was founded in 1983 and would soon dominate the business networking software. For consumers, going online meant logging on to CompuServe, and later Prodigy and AOL.

I think it would have been easy to envision, ex ante, a scenario in which any of these technologies could have grown to occupy the niche currently occupied by the Internet. Yes, the federal government heavily subsidized the Internet in its early days, but IBM and telecom companies had no shortage of funds to push SNA and X.25 respectively as alternatives. TCP/IP spread like wildfire because it was well-designed, and because its open architecture made it easier to deploy and more flexible than the alternatives. But the only place where TCP/IP was imposed “more or less by fiat” was on the ARPANET, which was a relatively insignificant network of a few hundred hosts. The rest of its growth was, if not exactly the result of market competition, certainly closer to market forces than central planning.

Had the government set out in the 1970s to create the networking architecture of the future, we would have ended up the same place France did. The reason TCP/IP turned out so well was precisely because no one realized it would be the future of world networking, and so its designers were free to take risks with relatively few political pressures. It was precisely because the choice of networking technologies was not “pseudo-centralized” in the late 1970s and early 1980s that TCP/IP had the opportunity to be developed and win a “pseudo-market” competition against networking standards that had much more formidable backers.

  • http://www.manifestdensity.net Tom

    A great point, and more than I knew about TCP/IP’s birth, to be sure. That’s why I hedged my bets by referring to all web technology — I was specifically thinking of W3C, ISO and MPEG. But you’re right that I’m oversimplifying the picture. Even those bodies are influenced and improved by the chaos of the market, like with Microsoft’s introduction of the nonstandard XmlHttpRequest Javascript object.

  • http://www.manifestdensity.net Tom

    A great point, and more than I knew about TCP/IP’s birth, to be sure. That’s why I hedged my bets by referring to all web technology — I was specifically thinking of W3C, ISO and MPEG. But you’re right that I’m oversimplifying the picture. Even those bodies are influenced and improved by the chaos of the market, like with Microsoft’s introduction of the nonstandard XmlHttpRequest Javascript object.

  • http://www.techliberation.com/ Tim Lee

    Right, I think this is a case where a binary division between “the market” and “central planning” is too simple. W3C engage in central planning in roughly the same way that Jimmy Wales and Linux Torvalds do: They’re widely respected and as a result have a lot of “soft power” over the direction of high-tech standards. But if W3C makes a stupid decision and everyone decides to ignore them, there isn’t a whole lot the W3C can do about it. And part of the reason that W3C (and Linus and Jimbo) retain the influence they have is that they’ve proven themselves adept at accommodating the competing agendas of various stakeholders.

  • http://www.techliberation.com/ Tim Lee

    Right, I think this is a case where a binary division between “the market” and “central planning” is too simple. W3C engage in central planning in roughly the same way that Jimmy Wales and Linux Torvalds do: They’re widely respected and as a result have a lot of “soft power” over the direction of high-tech standards. But if W3C makes a stupid decision and everyone decides to ignore them, there isn’t a whole lot the W3C can do about it. And part of the reason that W3C (and Linus and Jimbo) retain the influence they have is that they’ve proven themselves adept at accommodating the competing agendas of various stakeholders.

  • bemused

    As someone who was a young network engineer in the days when TCP/IP was born, I can tell you that central planning had both more and less impact than you admit. TCP/IP ran on the ARPANET which was paid for by DOD. In that period long haul circuits were hugely expensive, as was high speed connection equipemt. Private/commercial networking, barred from using the arpanet, had to cost justify network choices, and the X.25 and proprietary protocols were optimized to work well on lower speed connections. The ARPANET designers were academics studying network design and performance, but the underlying traffic on the ARPANET was mostly email and ftp file transfer between universities.

    When the ARPANET was opened to commercial traffic around 1990, the underlying hardware environment had changed. Computers were faster, the personal computer had become a platform able to run more resource intensive and capable operating systems, and many private companies had installed Ethernet infrastrucure for their private networks. The optimizations made by the telcos in X.25 were now the wrong ones to support high speed peer networking. The proprietary protocols were closed and also not designed to support networks without central administration.

    Thus it came to pass that the design decisions made by the researchers and paid for by the DOD came to the fore. Without the funding that enabled the ARPANET protocols to be designed, developed, tested and improved, they wouldn’t have been ready when the hardware caught up to them.

    An interesting contrast: the ISO protocols developed as an alternative to the ARPANET protocols. Arguably, they had some aspects that were theoretically superior to the ARPANET protocols, and they were an international standard adopted by the telecom community. But because they didn’t have battle tested implementations, they didn’t catch on, and only some of the upper layer protocols still are used.

  • bemused

    As someone who was a young network engineer in the days when TCP/IP was born, I can tell you that central planning had both more and less impact than you admit. TCP/IP ran on the ARPANET which was paid for by DOD. In that period long haul circuits were hugely expensive, as was high speed connection equipemt. Private/commercial networking, barred from using the arpanet, had to cost justify network choices, and the X.25 and proprietary protocols were optimized to work well on lower speed connections. The ARPANET designers were academics studying network design and performance, but the underlying traffic on the ARPANET was mostly email and ftp file transfer between universities.

    When the ARPANET was opened to commercial traffic around 1990, the underlying hardware environment had changed. Computers were faster, the personal computer had become a platform able to run more resource intensive and capable operating systems, and many private companies had installed Ethernet infrastrucure for their private networks. The optimizations made by the telcos in X.25 were now the wrong ones to support high speed peer networking. The proprietary protocols were closed and also not designed to support networks without central administration.

    Thus it came to pass that the design decisions made by the researchers and paid for by the DOD came to the fore. Without the funding that enabled the ARPANET protocols to be designed, developed, tested and improved, they wouldn’t have been ready when the hardware caught up to them.

    An interesting contrast: the ISO protocols developed as an alternative to the ARPANET protocols. Arguably, they had some aspects that were theoretically superior to the ARPANET protocols, and they were an international standard adopted by the telecom community. But because they didn’t have battle tested implementations, they didn’t catch on, and only some of the upper layer protocols still are used.

  • http://bennett.com/blog Richard Bennett

    The ISO OSI suite was the only practical alternative to TCP/IP. Xerox XNS was strictly a proprietary system, and most of its structure was actually “borrowed” by Cerf and Kahn (who were just down the street from Xerox PARC, at Stanford.) IBM never wanted other people implementing SNA, and X.25 was only a data link, not a full-blown protocol suite.

    As bemused mentions, ISO/OSI was technically superior to TCP/IP and was developed in a truly open process, not just by telecoms but by computer companies and users as well. General Motors and Boeing were big supporters of OSI.

    The DOD process was much more a matter of giving contracts to very small groups, like Cerf and Kahn for TCP and IP, and to Jon Postel for FTP. Hence their standards were much more limited and easier to understand and implement with fewer options. There was a whole standards aftermarket for ISO called the OSI Implementors’ Workshop (in which I participated) that did nothing more than choose subsets of the massive ISO protocols to implement in a series of interoperability fairs.

    The government forced the adoption of TCP/IP by paying for software implementations that were given away for free, and by demanding TCP support as a condition of making sales to the federal government. The DOD didn’t want to lose control of networking to a bunch of scruffy foreigners.

    Had the market been allowed to work, with by ISO and TCP software competing on a level playing field, ISO would have won because it incorporated the knowledge gained from early experiments in networking carried out by computer companies with proprietary software suites.

    And we wouldn’t have all the problems we’re having today with the transition to IPv6. But we’d have a different set of problems almost as bad, probably, so it may not end up making all that much difference. As time goes by, the IETF has simply incorporated more and more ideas from ISO anyhow.

    Anyhow, that’s how it looked from where I sat, at a company that was doing both TCP and ISO code at the same time.

  • http://bennett.com/blog Richard Bennett

    The ISO OSI suite was the only practical alternative to TCP/IP. Xerox XNS was strictly a proprietary system, and most of its structure was actually “borrowed” by Cerf and Kahn (who were just down the street from Xerox PARC, at Stanford.) IBM never wanted other people implementing SNA, and X.25 was only a data link, not a full-blown protocol suite.

    As bemused mentions, ISO/OSI was technically superior to TCP/IP and was developed in a truly open process, not just by telecoms but by computer companies and users as well. General Motors and Boeing were big supporters of OSI.

    The DOD process was much more a matter of giving contracts to very small groups, like Cerf and Kahn for TCP and IP, and to Jon Postel for FTP. Hence their standards were much more limited and easier to understand and implement with fewer options. There was a whole standards aftermarket for ISO called the OSI Implementors’ Workshop (in which I participated) that did nothing more than choose subsets of the massive ISO protocols to implement in a series of interoperability fairs.

    The government forced the adoption of TCP/IP by paying for software implementations that were given away for free, and by demanding TCP support as a condition of making sales to the federal government. The DOD didn’t want to lose control of networking to a bunch of scruffy foreigners.

    Had the market been allowed to work, with by ISO and TCP software competing on a level playing field, ISO would have won because it incorporated the knowledge gained from early experiments in networking carried out by computer companies with proprietary software suites.

    And we wouldn’t have all the problems we’re having today with the transition to IPv6. But we’d have a different set of problems almost as bad, probably, so it may not end up making all that much difference. As time goes by, the IETF has simply incorporated more and more ideas from ISO anyhow.

    Anyhow, that’s how it looked from where I sat, at a company that was doing both TCP and ISO code at the same time.

  • http://enigmafoundry.wordpress.com eee_eff

    It’s certainly true that the TCP/IP protocol suite was developed because some military researchers asked some really smart computer scientists to get together and write a new networking protocol. The spec was revised a few times (probably the most important being splitting things into a TCP layer and an IP layer) and then approved by the military, which proceeded to force every user of the ARPANET to switch to it in January 1983.

    TCP/IP spread like wildfire because it was well-designed, and because its open architecture made it easier to deploy and more flexible than the alternatives.

    Tim:

    I think there is a connection between the two statements I’ve repeated above. In other words,
    part of the reason TCP/IP was open and extendable was that it was designed by the military.

    I would refer to the history of the development of the radio as a consumer appliance just after the First World War. Prior to the First World War, Radios occupied room, and were difficult to put together due to patents that different companies held, and any one company couldn’t build a whole radio. The US Navy broke the patents, and made it possible to build what we know today as a radio, and the architecture was much more open than before.

    This is covered in detail in the excellent book “Steal this Idea” by Michael Perelman. In general, it seems that military contributions to consumer markets are under-rated. They are fundamentally different than inputs that come from purely commercial enterprises.

  • http://enigmafoundry.wordpress.com/ enigma_foundry

    It’s certainly true that the TCP/IP protocol suite was developed because some military researchers asked some really smart computer scientists to get together and write a new networking protocol. The spec was revised a few times (probably the most important being splitting things into a TCP layer and an IP layer) and then approved by the military, which proceeded to force every user of the ARPANET to switch to it in January 1983.

    TCP/IP spread like wildfire because it was well-designed, and because its open architecture made it easier to deploy and more flexible than the alternatives.

    Tim:

    I think there is a connection between the two statements I’ve repeated above. In other words,
    part of the reason TCP/IP was open and extendable was that it was designed by the military.

    I would refer to the history of the development of the radio as a consumer appliance just after the First World War. Prior to the First World War, Radios occupied room, and were difficult to put together due to patents that different companies held, and any one company couldn’t build a whole radio. The US Navy broke the patents, and made it possible to build what we know today as a radio, and the architecture was much more open than before.

    This is covered in detail in the excellent book “Steal this Idea” by Michael Perelman. In general, it seems that military contributions to consumer markets are under-rated. They are fundamentally different than inputs that come from purely commercial enterprises.

  • http://bennett.com/blog Richard Bennett

    ef says: “…part of the reason TCP/IP was open and extendable was that it was designed by the military.”

    TCP/IP wasn’t “designed by the military,” it was designed by a couple of computer science Ph. D.’s working at Stanford who happened to get some grant money from the military. They weren’t following orders from the Pentagon, and their work product wasn’t different in any substantial way from a purely private research effort at Xerox a few blocks down the street.

  • http://bennett.com/blog Richard Bennett

    ef says: “…part of the reason TCP/IP was open and extendable was that it was designed by the military.”

    TCP/IP wasn’t “designed by the military,” it was designed by a couple of computer science Ph. D.’s working at Stanford who happened to get some grant money from the military. They weren’t following orders from the Pentagon, and their work product wasn’t different in any substantial way from a purely private research effort at Xerox a few blocks down the street.

  • http://enigmafoundry.wordpress.com eee_eff

    TCP/IP wasn’t “designed by the military,” it was designed by a couple of computer science Ph. D.’s

    Richard:

    My point was not that everyone who worked on the development of TCP/IP was a commissioned officer, but that the client was the military, and that the design consideration were driven by criteria was developed by the military.

    From wikipedia:

    “The Internet protocol suite came from work done by Defense Advanced Research Projects Agency (DARPA) in the early 1970s. After building the pioneering ARPANET in the late 1960s, DARPA started work on a number of other data transmission technologies. In 1972, Robert E. Kahn was hired at the DARPA Information Processing Technology Office, where he worked on both satellite packet networks and ground-based radio packet networks, and recognized the value of being able to communicate across them. In the spring of 1973, Vinton Cerf, the developer of the existing ARPANET Network Control Program (NCP) protocol, joined Kahn to work on open-architecture interconnection models with the goal of designing the next protocol for the ARPANET.

    In March 1982, the US Department of Defense made TCP/IP the standard for all military computer networking.

  • http://enigmafoundry.wordpress.com/ enigma_foundry

    TCP/IP wasn’t “designed by the military,” it was designed by a couple of computer science Ph. D.’s

    Richard:

    My point was not that everyone who worked on the development of TCP/IP was a commissioned officer, but that the client was the military, and that the design consideration were driven by criteria was developed by the military.

    From wikipedia:

    “The Internet protocol suite came from work done by Defense Advanced Research Projects Agency (DARPA) in the early 1970s. After building the pioneering ARPANET in the late 1960s, DARPA started work on a number of other data transmission technologies. In 1972, Robert E. Kahn was hired at the DARPA Information Processing Technology Office, where he worked on both satellite packet networks and ground-based radio packet networks, and recognized the value of being able to communicate across them. In the spring of 1973, Vinton Cerf, the developer of the existing ARPANET Network Control Program (NCP) protocol, joined Kahn to work on open-architecture interconnection models with the goal of designing the next protocol for the ARPANET.

    In March 1982, the US Department of Defense made TCP/IP the standard for all military computer networking.

  • http://bennett.com/blog Richard Bennett

    Dude, when you start throwing wikifacts at me I have to say the discussion is over.

  • http://bennett.com/blog Richard Bennett

    Dude, when you start throwing wikifacts at me I have to say the discussion is over.

  • http://enigmafoundry.wordpress.com eee_eff

    Dude, when you start throwing wikifacts at me I have to say the discussion is over.

    Richard:
    That’s basically an ad hominum attack, but if you have any facts that would show that TCP/IP wasn’t design by a defense agency, please post those facts…

  • http://enigmafoundry.wordpress.com/ enigma_foundry

    Dude, when you start throwing wikifacts at me I have to say the discussion is over.

    Richard:
    That’s basically an ad hominum attack, but if you have any facts that would show that TCP/IP wasn’t design by a defense agency, please post those facts…

  • crack

    I was under the impression that IP came about because the military wanted a robust network that could withstand the loss of multiple nodes. IP being the network layer (layer 4 of the OSI model) left all the lower level stuff to the local network. That would imply that it was designed by PhDs with requirements from the military which would be in line with EF and RB.

  • crack

    I was under the impression that IP came about because the military wanted a robust network that could withstand the loss of multiple nodes. IP being the network layer (layer 4 of the OSI model) left all the lower level stuff to the local network. That would imply that it was designed by PhDs with requirements from the military which would be in line with EF and RB.

  • http://enigmafoundry.wordpress.com eee_eff

    crack:
    that’s exactly right, but Richard seems incapable of admitting that anything good was every done by the govt…plus he likes calling me names …

  • http://enigmafoundry.wordpress.com/ enigma_foundry

    crack:
    that’s exactly right, but Richard seems incapable of admitting that anything good was every done by the govt…plus he likes calling me names …

  • http://None Danny

    In relation to whether TCP/IP was designed by the military or by PhD’s at Stanford, see Stuart Leslie, The Cold War and American Science: The Military-Industrial-Academic Complex at MIT and Stanford – it contains a great deal on the role of the military in shaping scientific research projects, even when not explicitly for military purposes. It also draws attention to the manner in which military funding shapes university educational curricula overall – perhaps if Xerox and Stanford were doing the same work, this is the more important insight.

  • Danny

    In relation to whether TCP/IP was designed by the military or by PhD’s at Stanford, see Stuart Leslie, The Cold War and American Science: The Military-Industrial-Academic Complex at MIT and Stanford – it contains a great deal on the role of the military in shaping scientific research projects, even when not explicitly for military purposes. It also draws attention to the manner in which military funding shapes university educational curricula overall – perhaps if Xerox and Stanford were doing the same work, this is the more important insight.

Previous post:

Next post: