FLOSS: The Software Hare that Beats the Proprietary Turtle?

by on March 9, 2007 · 12 comments

Here’s another installment in my series of blog posts analyzing the European Commission’s free/libre and open source software (FLOSS) report. In prior posts, I’ve discussed how the report is a call to action for Europe’s policymakers, that FLOSS’s popularity is growing, and that many FLOSS developers live in the EU. In short, the report contains ambitious policy proposals to transform Europe’s ICT sector through industrial policies — I’ve called it the new Airbus project for the European economy. As I get further into the report, however, I’m becoming more skeptical of the report’s overall claim that open source software can rescue Europe’s software industry.

Here’s why – the authors go to the extreme to show FLOSS in a good light, without giving much emphasis to how FLOSS has so far achieved its growing and respected status. Why do I care? Well, I haven’t read much critique on the report, and it’s often fun to be the dissenting voice. And I’m sick of hearing that open source software will rescue the planet from the evils of commercial software and the capitalist system that spawned it. I know that TLF readers are savvy enough to stay away from such absolute statements, but many others in the blogosphere aren’t.

Moreover, at the public policy level, there’s a lot of confusion I’m anxious to clear up between open source the license and the community development model that will often characterize open source software. Finally, I’m becoming more interested in free and open source software on a personal level, which can be technically challenging and at times overwhelming.

So…here I’ll analyze the report’s claim in Section 7 that “FLOSSers Work Faster”.


To make the assertion that FLOSS development is faster than comparable proprietary software development, the authors compare survey and model data on the effort needed to generate the 221 million lines of code in the Debian FLOSS project. That bit of data is neither conclusive nor convincing, and the authors surely know that the license used for a piece of software says nothing about its development model or the speed of its programmers. Nonetheless, there are some good reasons to think that FLOSS programmers should work faster than developers of proprietary software.

First, FLOSS developers can freely borrow code from other FLOSS projects, saving time in design and coding. Most licenses governing FLOSS software permit its incorporation, modification and redistribution.

On the other hand, proprietary software developers also re-use chunks of code from previous efforts, especially with firms that crank-out multiple versions or a suite of related products. Ironically, The Chairman of the Association for Competitive Technology is a Belgian expat whose company, Sax.net, creates reusable components for proprietary software developers. So the notion of borrowing code probably doesn’t produce a substantial productivity advantage for FLOSS developers.

There’s a more compelling explanation for why FLOSS developers should work faster than their commercial software brethren. It starts with an observation that the important early FLOSS projects were imitations of existing products, or — as the authors put it — “FLOSS projects for which proprietary software with similar functionality exist.” [p51]

Although there are important exceptions (e.g. Apache, Sendmail, Perl), FLOSS developers have focused much of their effort on writing software that imitates features and functionality in existing commercial software products. For example:

Linux, the best-known FLOSS project and the operating system of over 70 million users, was conceived to re-create the widely-used Unix operating system.

OpenOffice, the productivity suite that has been downloaded by over 31 million users, admittedly imitates the interface and functionality of Microsoft Office, and boasts compatibility with all Microsoft file formats.

MySQL is a FLOSS database engine that was designed to emulate commercial database software from Microsoft, Oracle, and IBM. MySQL now has an installed base of five million systems, the largest of any database engine. Another open source vendor, RedHat, offers a FLOSS add-on product to give MySQL a user interface that imitates Microsoft’s Enterprise Manager for its SQL Server database.

Samba imitates the functionality of Windows servers for use with Linux and Unix servers.

In an example that goes beyond imitating applications, the Mono development platform mimics the development framework of Microsoft’s”.NET”.

In projects like those listed above, FLOSS programmers should work faster when they are imitating the workflow and features of software that someone else has already developed. There’s just no need to burn all those man-hours in use-case analysis and screen design when developers decide to create a FLOSS equivalent of, say, Microsoft Word.

With Imitation Comes Reward—with less Risk

Second, many FLOSS projects avoid important risks, meaning less time is spent on market tests, dead-ends, and do-overs. By imitating software that’s already in use, a FLOSS developer instantly escapes two critical risks that keep ordinary software designers up at night:

There is less risk that an imitative software development effort will fail because of limitations in technology, since it’s already been demonstrated that the software can be developed and can operate with acceptable performance characteristics.

There is less risk that imitative software, once created, will not find acceptance by users. The original software that’s being imitated has already proven user acceptance, often after spending significant sums on marketing and user education.

If a proven piece of software has already met the test of customer acceptance, there’s no need to spend months doing user trials and beta tests. Just test the FLOSS product side-by-side against the original. If the FLOSS product is a close copy, you’re good-to-go.

Finally, the authors of the FLOSS report claim less need for centralized coordination and management [p.53-54]. That seems credible when you consider that individual programmers can easily work alone, without the help of designers or system analysts, when they set out to imitate individual components of an existing product.

But after any software is shipped and in wide use, management and coordination become relatively more important than programming. Managing the integration, testing, and distribution of bug fixes, enhancements, and new versions is an ongoing ordeal. It may take fifteen minutes to code a bug fix, but fifteen days of effort to test and distribute the fix to users. Even Linus Torvalds, the chief coder behind the widely-used Linux operating system, now spends most of his time managing relatively small changes to the kernel.

Is Faster Better? Sure, Sometimes.

Even if the data don’t support their conclusion, the authors desperately want to believe that FLOSS developers are more productive than their commercial counterparts. And given that so much of the early FLOSS development has been imitations of existing software, you’d expect FLOSSers to be faster at cranking out the code.

  • http://www.techliberation.com/ Tim Lee

    Braden,

    MySQL is a FLOSS database engine that was designed to emulate commercial database software from Microsoft, Oracle, and IBM. MySQL now has an installed base of five million systems, the largest of any database engine.

    I don’t think this is quite right. MySQL, like Oracle and SQL Server, implement SQL, which has been the industry standard for relational databases for a quarter-century. MySQL isn’t a clone of Oracle or SQL Server any more than SQL Server is a clone of Oracle, or Oracle is a clone of earlier database applications.

    Moreover, writing the code that parses SQL statements is one of the easier aspects of developing database application. The far more difficult tasks are the “back end” challenges of writing code that supports concurrency, scalability, data integrity, and performance. And in those respects—which are what really distinguish industrial-strength databases from toy databases—I don’t think MySQL is in any way derived from proprietary databases.

    To a large extent, the same can be said of the other examples in that quote. Unix was originally a proprietary operating system developed at AT&T;, but by the 1990s it had become a de facto open standard supported by numerous different companies and open source projects. Linux is a clone of Unix only in the sense that it implements interfaces that programmers had come to expect, just as cars are clones of one another because the steering wheel, gas, and brake arranged in a similar way to other cars. The specific implementation details vary greatly among Unix versions.

    Indeed, I would argue that the opposite argument could be made about OpenOffice: it would be far easier to build a word processor from scratch than it is to clone Microsoft Office. A lot of the work of developing OpenOffice involves mimicking the idiosyncratic behaviors of Microsoft’s proprietary file formats. If Office were built on open, well-documented formats, I suspect that open source projects would have been a lot more successful. Thanks to the availability of the open HTML format, for example, the Mozilla project has long since reached parity with Internet Explorer. It’s only when open source projects are trying to replicate the behavior of buggy, proprietary software that they have difficult reaching feature parity in a timely manner.

    On the other side of the ledger, I also think your list of non-immitative free software projects (Apache, Sendmail, Perl) can be expanded on greatly: Python, PHP, BIND, CUPS, BitTorrent, Jabber, Gnome, and KDE are a few that come to mind immediately. And keep in mind that proprietary companies sometimes mimic open source projects. For example, tabbed browsing was first introduced in Mozilla, and only much later was added to Internet Explorer.

  • http://www.techliberation.com/ Tim Lee

    Braden,

    MySQL is a FLOSS database engine that was designed to emulate commercial database software from Microsoft, Oracle, and IBM. MySQL now has an installed base of five million systems, the largest of any database engine.

    I don’t think this is quite right. MySQL, like Oracle and SQL Server, implement SQL, which has been the industry standard for relational databases for a quarter-century. MySQL isn’t a clone of Oracle or SQL Server any more than SQL Server is a clone of Oracle, or Oracle is a clone of earlier database applications.

    Moreover, writing the code that parses SQL statements is one of the easier aspects of developing database application. The far more difficult tasks are the “back end” challenges of writing code that supports concurrency, scalability, data integrity, and performance. And in those respects—which are what really distinguish industrial-strength databases from toy databases—I don’t think MySQL is in any way derived from proprietary databases.

    To a large extent, the same can be said of the other examples in that quote. Unix was originally a proprietary operating system developed at AT&T, but by the 1990s it had become a de facto open standard supported by numerous different companies and open source projects. Linux is a clone of Unix only in the sense that it implements interfaces that programmers had come to expect, just as cars are clones of one another because the steering wheel, gas, and brake arranged in a similar way to other cars. The specific implementation details vary greatly among Unix versions.

    Indeed, I would argue that the opposite argument could be made about OpenOffice: it would be far easier to build a word processor from scratch than it is to clone Microsoft Office. A lot of the work of developing OpenOffice involves mimicking the idiosyncratic behaviors of Microsoft’s proprietary file formats. If Office were built on open, well-documented formats, I suspect that open source projects would have been a lot more successful. Thanks to the availability of the open HTML format, for example, the Mozilla project has long since reached parity with Internet Explorer. It’s only when open source projects are trying to replicate the behavior of buggy, proprietary software that they have difficult reaching feature parity in a timely manner.

    On the other side of the ledger, I also think your list of non-immitative free software projects (Apache, Sendmail, Perl) can be expanded on greatly: Python, PHP, BIND, CUPS, BitTorrent, Jabber, Gnome, and KDE are a few that come to mind immediately. And keep in mind that proprietary companies sometimes mimic open source projects. For example, tabbed browsing was first introduced in Mozilla, and only much later was added to Internet Explorer.

  • Doug Lay

    First of all, although I haven’t read the whole report, and would prefer not to do so, I did check Chapter 7 pretty carefully and found no claim that “FLOSSers work faster.”

    I did note the report’s list of the five largest FLOSS projects by code size, and of these five, only OpenOffice sould really be called an imitation of a commercial product. The other four (Linux kernel, gcc, mozilla/firefox and xfree86) all implement open standards, but none of them is chasing the taillights of any commercial product. Indeed, Linux, mozilla and gcc all arguably represent the state of the art in their respective product segments, and while the X window system is (IMHO) too much of a mess to represent any state of the art, anyone who has worked with it would be hard-pressed to call it a knock-off of the Windows or Mac windowing systems.

    Having been a relatively early adopter of both Linux and MySQL (I used both in production systems as far back as 1997) I also want to emphasize that while the important APIs provided by these two projects were based on standards as opposed to breaking new ground (SQL in the case of MySQL and POSIX in the case of Linux), both projects DID break new ground in terms of providing important new options to developers building Web-based applications:

    Linux allowed (and continues to allow) developers to leverage both the power of Unix/Posix compatibility and the economy of commodity PC architecture. It has proven a near-unstoppable combination, and no commercial product was doing it well at the time Linux emerged in the mid-to-late-1990s (BSD was good but not commercial; SCO was commercial but not good, at least not at Internet applications).

    MySQL provided (and continues to provide) a stripped-down, speedy RDBMS for Web applications, at a time when commercial offerings were either overwrought and expensive (Oracle, Sybase, SQL Server), or woefully inadequate in a multi-user environment (Access).

  • Doug Lay

    First of all, although I haven’t read the whole report, and would prefer not to do so, I did check Chapter 7 pretty carefully and found no claim that “FLOSSers work faster.”

    I did note the report’s list of the five largest FLOSS projects by code size, and of these five, only OpenOffice sould really be called an imitation of a commercial product. The other four (Linux kernel, gcc, mozilla/firefox and xfree86) all implement open standards, but none of them is chasing the taillights of any commercial product. Indeed, Linux, mozilla and gcc all arguably represent the state of the art in their respective product segments, and while the X window system is (IMHO) too much of a mess to represent any state of the art, anyone who has worked with it would be hard-pressed to call it a knock-off of the Windows or Mac windowing systems.

    Having been a relatively early adopter of both Linux and MySQL (I used both in production systems as far back as 1997) I also want to emphasize that while the important APIs provided by these two projects were based on standards as opposed to breaking new ground (SQL in the case of MySQL and POSIX in the case of Linux), both projects DID break new ground in terms of providing important new options to developers building Web-based applications:

    Linux allowed (and continues to allow) developers to leverage both the power of Unix/Posix compatibility and the economy of commodity PC architecture. It has proven a near-unstoppable combination, and no commercial product was doing it well at the time Linux emerged in the mid-to-late-1990s (BSD was good but not commercial; SCO was commercial but not good, at least not at Internet applications).

    MySQL provided (and continues to provide) a stripped-down, speedy RDBMS for Web applications, at a time when commercial offerings were either overwrought and expensive (Oracle, Sybase, SQL Server), or woefully inadequate in a multi-user environment (Access).

  • http://weblog.ipcentral.info/ Noel Le

    Nice summary Braden.

    The relationship between Unix and Linux is closer than Tim lets on above- Linux is an evolved and standardized form of Unix; however, I’ve never seen this line of argument as particularly important. FOSS critics will use it as a way of showing how FOSS is inherently the result of cumulative innovation; and with the length of time that FOSS has been around it should have made more ground by now. FOSS proponents will try to dispel the argument to show that while FOSS does rely particularly on cumulative development, thats its capable of pioneering, groundbreaking innovation. The debate hasn’t changed in the past several years that I’ve followed it, and I believe it ignores several more important issues.

    1) Did FOSS forget the desktop? When is this going to happen, and what does it say about FOSS’ potential reach in the technological community? FOSS will not only continue to co-exist with proprietary technologies, it will need them.

    2) What is really keeping FOSS from reaching its potential? Is it software patents? No, the opposition to software patents from the FOSS community comes from ideological arguments, and ones about the *potential* harms of software patents. Is it Microsoft? Well, yes, if you consider competition harmful. The more comprehensive answer is FOSS, at least its fanatic wing, is holding FOSS back.

    3) FOSS needs to get more serious, period. All these rain dances about a revolution and freedom need to be translated to your basic business language if FOSS wants to appeal to a broader demographic. I notice a lot of the BSD crowd has gotten smart, and practical, however, the religious holdouts, a la the Stallman camp, are stubborn as ever.

  • http://weblog.ipcentral.info/ Noel Le

    My apologies, its late on the east coast.

    I forgot to mention- it takes a gaping lack in proportionality to say FOSS will overtake all proprietary technologies, or that all software will be peer-produced in the future, as some FOSS supporters have claimed.

    This first argument is addressed by the fact that FOSS technologies need proprietary technologies as long as FOSS cannot create it own desktop and mass consumer apps; the second point is addressed by the simply concept that 99% of the population would rather not buy a computer system created by someone who only profits if it doesn’t work well.

  • http://weblog.ipcentral.info/ Noel Le

    Nice summary Braden.

    The relationship between Unix and Linux is closer than Tim lets on above- Linux is an evolved and standardized form of Unix; however, I’ve never seen this line of argument as particularly important. FOSS critics will use it as a way of showing how FOSS is inherently the result of cumulative innovation; and with the length of time that FOSS has been around it should have made more ground by now. FOSS proponents will try to dispel the argument to show that while FOSS does rely particularly on cumulative development, thats its capable of pioneering, groundbreaking innovation. The debate hasn’t changed in the past several years that I’ve followed it, and I believe it ignores several more important issues.

    1) Did FOSS forget the desktop? When is this going to happen, and what does it say about FOSS’ potential reach in the technological community? FOSS will not only continue to co-exist with proprietary technologies, it will need them.

    2) What is really keeping FOSS from reaching its potential? Is it software patents? No, the opposition to software patents from the FOSS community comes from ideological arguments, and ones about the *potential* harms of software patents. Is it Microsoft? Well, yes, if you consider competition harmful. The more comprehensive answer is FOSS, at least its fanatic wing, is holding FOSS back.

    3) FOSS needs to get more serious, period. All these rain dances about a revolution and freedom need to be translated to your basic business language if FOSS wants to appeal to a broader demographic. I notice a lot of the BSD crowd has gotten smart, and practical, however, the religious holdouts, a la the Stallman camp, are stubborn as ever.

  • http://weblog.ipcentral.info/ Noel Le

    My apologies, its late on the east coast.

    I forgot to mention- it takes a gaping lack in proportionality to say FOSS will overtake all proprietary technologies, or that all software will be peer-produced in the future, as some FOSS supporters have claimed.

    This first argument is addressed by the fact that FOSS technologies need proprietary technologies as long as FOSS cannot create it own desktop and mass consumer apps; the second point is addressed by the simply concept that 99% of the population would rather not buy a computer system created by someone who only profits if it doesn’t work well.

  • http://you-read-it-here-first.com John Bailo

    Here’s something that irks me. Even though this is “Sunshine Week”, you’d think there would be tons of Democrats speaking about using the Open Document Format in government and saving taxpayers money with Open Source software. Yet a Google search linking the two produces a paltry number of articles and a Google News search shows no linkage at all!

    Guess those Democrats must have their PACs on the other side of “Sunshine”.

  • http://you-read-it-here-first.com John Bailo

    Here’s something that irks me. Even though this is “Sunshine Week”, you’d think there would be tons of Democrats speaking about using the Open Document Format in government and saving taxpayers money with Open Source software. Yet a Google search linking the two produces a paltry number of articles and a Google News search shows no linkage at all!

    Guess those Democrats must have their PACs on the other side of “Sunshine”.

  • http://techliberation.com Braden

    Thank you all for your comments.  I’ll first respond to Doug Lay, who somehow missed that the report’s authors are asserting that “FLOSSers work faster”.  See page 49 for results of a modeling exercise showing that a software company would take 163,522 person-years to produce the 221 million lines of the code in the Debian 3.1 FLOSS distribution.  By comparison, the table on page 54 shows that FLOSS programmers coded Debian 3.1 in just under 44,000 person-years, and needed only 131,832 person-years for all development over a 5-year period.  And on page 53, the authors muse about possible explanations for this “apparently greater productivity of FLOSS developers.”

    Doug says he hasn’t read the whole FLOSS report, which I admit is a daunting task (look how long it’s taking me to get through it all!).  But this FLOSS report deserves scrutiny, since it’s intended to justify a bold industrial policy initiative for European governments and institutions.  Go straight to the report’s conclusions and you’ll see recommendations for procurement preferences, tax laws, and policies that support FLOSS software in the EU.   These recommendations are based on sections of the report that argue FLOSS is more popular than commercial software, more FLOSS programmers are in the EU, and yes, that “FLOSSers work faster.”

    The point of my post was that if FLOSSers work faster, it’s mainly because there’s less risk and less effort in imitating the functionality of products that are already proven successes in the market.  By imitation, I mean copying the purpose, functionality, style, workflow, and many of the features of a pre-existing product.  When you’re imitating Unix or Microsoft Office, you spend far less time on design and user testing, and waste no time on features that don’t make it into the final release. 

    The report’s authors acknowledge that big FLOSS projects like OpenOffice imitate the functionality of existing proprietary software—namely, Microsoft Office.  Tim Lee, while agreeing that OpenOffice imitates MS Office, makes the strange claim that such imitation actually takes longer than designing and coding from scratch.  Anyone who’s managed a software development project will tell you that imitating the features and workflow of a proven product is the fastest way to get from a blank screen to a finished piece of software that consumers are ready and willing to use. 

    Note that I am talking about imitation, not cloning.  Tim took pains to explain that some imitative FLOSS products are not exact clones of other products.  Cloning is producing an exact copy that is indistinguishable from the original, such that it can be used as a drop-in replacement without having to modify surrounding systems or their interfaces.

    I think Tim misses the point of my post: imitation is faster and easier than innovation, and most important FLOSS software (but not all) has been imitative of products already in the market. FLOSSers may have cranked-out some imitative products pretty quickly, but there’s no evidence to suggest they are inherently more productive than proprietary software developers.   

    Another important point made in my post is that imitation (not cloning) avoids the two biggest risks that raise the cost and lower the success of most software development projects.  There is less risk that an imitative software development effort will fail because of limitations in technology, because it’s already been demonstrated that the software can be developed and can operate with acceptable performance characteristics.  And there is less risk that imitative software, once created, will fail to be embraced by users. The original software that’s being imitated has already proven user acceptance, often after spending significant sums on marketing and user education.

    I appreciate the comments by Doug and Tim, and I encourage them (and everybody else!) to realize more fully the real purpose of the FLOSS report (and the distinction between imitation and cloning).  Moreover, EU policy-makers would appreciate some objective advice before they base their next industrial policy adventure on assertions that FLOSSers are faster.

  • bradencox

    Thank you all for your comments.  I’ll first respond to Doug Lay, who somehow missed that the report’s authors are asserting that “FLOSSers work faster”.  See page 49 for results of a modeling exercise showing that a software company would take 163,522 person-years to produce the 221 million lines of the code in the Debian 3.1 FLOSS distribution.  By comparison, the table on page 54 shows that FLOSS programmers coded Debian 3.1 in just under 44,000 person-years, and needed only 131,832 person-years for all development over a 5-year period.  And on page 53, the authors muse about possible explanations for this “apparently greater productivity of FLOSS developers.”

    Doug says he hasn’t read the whole FLOSS report, which I admit is a daunting task (look how long it’s taking me to get through it all!).  But this FLOSS report deserves scrutiny, since it’s intended to justify a bold industrial policy initiative for European governments and institutions.  Go straight to the report’s conclusions and you’ll see recommendations for procurement preferences, tax laws, and policies that support FLOSS software in the EU.   These recommendations are based on sections of the report that argue FLOSS is more popular than commercial software, more FLOSS programmers are in the EU, and yes, that “FLOSSers work faster.”

    The point of my post was that if FLOSSers work faster, it’s mainly because there’s less risk and less effort in imitating the functionality of products that are already proven successes in the market.  By imitation, I mean copying the purpose, functionality, style, workflow, and many of the features of a pre-existing product.  When you’re imitating Unix or Microsoft Office, you spend far less time on design and user testing, and waste no time on features that don’t make it into the final release. 

    The report’s authors acknowledge that big FLOSS projects like OpenOffice imitate the functionality of existing proprietary software—namely, Microsoft Office.  Tim Lee, while agreeing that OpenOffice imitates MS Office, makes the strange claim that such imitation actually takes longer than designing and coding from scratch.  Anyone who’s managed a software development project will tell you that imitating the features and workflow of a proven product is the fastest way to get from a blank screen to a finished piece of software that consumers are ready and willing to use. 

    Note that I am talking about imitation, not cloning.  Tim took pains to explain that some imitative FLOSS products are not exact clones of other products.  Cloning is producing an exact copy that is indistinguishable from the original, such that it can be used as a drop-in replacement without having to modify surrounding systems or their interfaces.

    I think Tim misses the point of my post: imitation is faster and easier than innovation, and most important FLOSS software (but not all) has been imitative of products already in the market. FLOSSers may have cranked-out some imitative products pretty quickly, but there’s no evidence to suggest they are inherently more productive than proprietary software developers.   

    Another important point made in my post is that imitation (not cloning) avoids the two biggest risks that raise the cost and lower the success of most software development projects.  There is less risk that an imitative software development effort will fail because of limitations in technology, because it’s already been demonstrated that the software can be developed and can operate with acceptable performance characteristics.  And there is less risk that imitative software, once created, will fail to be embraced by users. The original software that’s being imitated has already proven user acceptance, often after spending significant sums on marketing and user education.

    I appreciate the comments by Doug and Tim, and I encourage them (and everybody else!) to realize more fully the real purpose of the FLOSS report (and the distinction between imitation and cloning).  Moreover, EU policy-makers would appreciate some objective advice before they base their next industrial policy adventure on assertions that FLOSSers are faster.

Previous post:

Next post: