Reverse Engineering and Innovation: Some Examples

In my my recent post on the DMCA, I cited the creation of a Linux DVD player as an example of innovation. Randy Picker responds:

No, certainly the DVD example isn’t evidence of stifling innovation. The DVD example is a choice, nothing more than that. I have little doubt that the DVD format creators could have made it possible to have DVDs played in a Linux environment.

They chose not to do so. We can decide whether that choice should be respected, but we shouldn’t say that it is innovative when someone unilaterally decides not to respect that choice and tries to figure out a way around that choice. That is like saying that it is innovative to drive 70 in a 55 mph zone. Anyone can do it; the trick isn’t in doing it; it is in deciding whether we want to respect the limit in the first place.

Strictly speaking, he’s right. Reverse engineering the CSS encryption scheme, by itself, isn’t an especially innovative activity. However, what I think Prof. Picker is missing is how important such reverse engineering can be as a pre-condition for subsequent innovation. To illustrate the point, I’d like to offer three examples of companies or open source projects that have forcibly opened a company’s closed architecture, and trace how these have enabled subsequent innovation:

Instant Messaging: The instant messaging marketplace is current fragmented among a lot of different networks: AOL Instant Messenger, MSN Messenger, Yahoo Messenger, Google Talk, and others. Although these guys have periodically discussed making their networks interoperable, at present none of them are, despite the fact that they’ve been on the market for years. (Google Talk is an exception–it’s a relative newcomer that uses the open Jabber protocol and is compatible with other smaller IM networks)

That created a problem: Let’s say I’ve got some friends who use AIM, some who use Yahoo, and some who use MSN Messenger. If I wanted to chat with all of them at the same time, I’d have to run three separate instant messaging programs. That’s a pain in the butt.

Fortunately, reverse engineering has come to my rescue. An open source project called Fire built a Mac OS X application that can access all four networks (and some others). That means that I can run one program and have access to all four networks. And Fire’s not the only such client. There’s also GAIM (which my girlfriend uses), the commercial Trillian, and many others.

Not only does this allow me to use four instant messaging networks with one program, but these alternative clients have features that the official clients don’t. And they run on platforms the official clients don’t, too. For example, Center ICQ is multi-protocol chat client that you can use from a Unix command line. Meebo is an extremely slick instant messaging client that runs in your web browser.

It’s likely that none of these programs would exist if the legal system had given AOL, Yahoo, and Microsoft property rights in their respective instant messaging architectures. AOL and Yahoo have attempted repeatedly to change their protocols to break these third-party messengers, but in each case the third-party developers have reverse-engineered the changes and posted new versions of their applications in a matter of days, if not hours. It seems pretty clear to me that the market for instant messenger clients would be impoverished if we gave them the legal power to decide who was allowed to connect to their IM networks.

Samba: Samba’s slogan is “Opening Windows to a Wider World.” It includes client software that allows non-Windows desktops to use Windows-based file and print servers, and server software that allows non-Windows servers to act as Windows file and print servers. Creating Samba required unauthorized reverse engineering without Microsoft’s permission, and indeed, the need for reverse engineering will continue as Microsoft changes its protocols in future versions of Windows.

The existence of Samba has greatly improved the ability of Linux to compete in the small office server marketplace. Indeed, our little office of half a dozen employees has a Linux server that serves as our file server. Linux has a variety of features that (in my opinion at least) makes it a better choice as a file server than Windows-based servers. It’s stable, allows for flexible remote administration, and has a lot of powerful command-line tools.

Samba has also been incorporated into Mac OS X, allowing us Mac users to access Windows file servers. Indeed, my iMac can access files on our Linux-based server using Microsoft’s file-sharing protocol. Samba has transformed the Windows file sharing protocols into de facto open standards. That, it seems to me, is a good thing, because it allows users to choose the operating system that best meets their needs, without worrying about whether it will be able to interoperate with other computers in their offices.

x86 CPUs: The x86 instruction set that lies at the heart of every PC (including as of this year, Macs) was invented by Intel. The first IBM PCs had an Intel 8088 central processing unit in it, and subsequent versions had Intel’s (80)286, 386, and 486 CPUs. The basic instructions executed by the latest Intel-based computers are a superset of the instructions executed by the 8088 and 286.

But of course, Intel’s not the only company that sells x86-based chips. They have several competitors, the most prominent of which is AMD. AMD, too, uses a superset of the x86 instruction set. And not surprisingly, Intel wasn’t happy that AMD was selling compatible chips. Wikipedia tells the story:

In February 1982, AMD signed a contract with Intel, becoming a licensed second-source manufacturer of 8086 and 8088 processors. IBM wanted to use the Intel 8088 in its IBM PC, but IBM’s policy at the time was to require at least two sources for its chips. AMD later produced the 80286, or 286, under the same arrangement, but Intel cancelled the agreement in 1986, and refused to hand over technical details of the i386 part. The growing popularity of the PC clone market meant Intel could produce CPUs on its own terms, rather than IBM’s.

AMD challenged this decision, and subsequently won under arbitration. A long legal dispute followed, ending in 1991 when the Supreme Court of California sided with AMD and forced Intel to pay over $1 billion in compensation for violation of contract. Subsequent legal disputes centered on whether AMD had legal rights to use derivatives of Intel’s microcode. Rulings were made in both directions. In the face of uncertainty, AMD was forced to develop “clean room” versions of Intel code. In this fashion, one engineering team described the function of the code, and a second team without access to the source code itself had to develop microcode that performed the same functionality.

Was this innovative? The early versions of AMD’s CPUs were arguably cheap knock-offs of Intel’s chips. But in recent years, many people believe that AMD has pulled ahead of Intel in the CPU performance race. Most interestingly, a couple of years ago AMD unveiled the Opteron, a 64-bit extension to the 32-bit x86 processor family. A lot of analysts believe that in the last couple of years AMD has become the performance leader for x86 CPUs.

And AMD’s not the only company to build Intel clones, although they’re the most successful. Other notables include Transmeta and Cyrix. Although there has been a lot of legal wrangling over alleged patent infringement and violations of license terms, it’s clear that Intel doesn’t have the right to exclude competitors from making x86 chips simply because Intel invented the x86 instruction set.

So what can we learn from all this? One lesson, I think, is that reverse-engineering to accomplish compatibility with a competitor’s closed, proprietary platform is utterly common in the technology world. There are lots of other examples I could cite (and I encourage readers to add their own examples in comments) but the bottom line is that there’s nothing unusual about Linux users deciding to reverse-engineer CSS in order to build a DVD player.

Secondly, I think that such reverse engineering is plainly a pre-condition to innovation in many cases. It’s highly unlikely that AOL, Microsoft, or Intel would have voluntarily allowed GAIM, Samba, or Cyrix to create their products if they’d had the option to shut them down. Moreover, GAIM and Samba are decentralized open source projects, so even if AOL and Microsoft had been willing to negotiate a license, it’s unlikely that it would have been logistically possible to do so. This is directly analogous to the Linux DVD case. There are a couple of proprietary DVD players for specific quasi-proprietary version of Linux because those version had a for-profit company that could negotiate a license. But an open source project, because it allows anyone to modify its source code, can’t meet the demands of the DVD CCA.

Thirdly, there’s no reason to think the existence of these competing products has reduced the incentive of AOL, Microsoft, or Intel to continue innovating themselves. Intel’s market share is about 80 percent, Microsoft’s is about 90 percent, and the vast majority of AIM users still use the official AOL client. Creating a proprietary architecture is hugely profitable, even if somebody subsequently comes along and forces it open.

I hope this sheds light on why the DMCA is so incredibly unpopular among geeks. The struggle between open and closed systems is probably the central drama in the computer industry. I won’t claim, as some of commenters do, that open systems are always better, although they usually are. But the more important point is that reverse engineering serves as a kind of safety valve, ensuring that unnecessary compatibility problems don’t crop up between competing architectures. It also ensures that should the creator of a proprietary architecture allow it to stagnate, as Intel arguably did with x86 in the late 1990s, competitors have the option of coming in and reinvigorating it. At some point, Apple’s going to release a new iPod that’s a dud. Wouldn’t it be nice if there were iPod clone makers waiting in the wings so that consumers had some alternatives?

SHARE: