My New Empirical Study on Defining and Measuring Search Bias

by on November 3, 2011 · 2 comments

[Cross posted at]

Tomorrow is the deadline for Eric Schmidt to send his replies to the Senate Judiciary Committee’s follow up questions from his appearance at a hearing on Google antitrust issues last month.  At the hearing, not surprisingly, search neutrality was a hot topic, with representatives from the likes of Yelp and Nextag, as well as Expedia’s lawyer, Tom Barnett (that’s Tom Barnett (2011), not Tom Barnett (2006-08)), weighing in on Google’s purported bias.  One serious problem with the search neutrality/search bias discussions to date has been the dearth of empirical evidence concerning so-called search bias and its likely impact upon consumers.  Hoping to remedy this, I posted a study this morning at the ICLE website both critiquing one of the few, existing pieces of empirical work on the topic (by Ben Edelman, Harvard economist) as well as offering up my own, more expansive empirical analysis.  Chris Sherman at Search Engine Land has a great post covering the study.  The title of his article pretty much says it all:  “Bing More Biased Than Google; Google Not Behaving Anti-competitively.”

One clarification is in order.  The  Search Engine Land piece quotes Geoff responding to the author indicating that the research was undertaken independently.  It’s not clear from the way Sherman presents it in the article, but Geoff did acknowledge (as does the disclosure in the paper itself) that Google supports ICLE.  However, as Sherman notes, although the work was indirectly supported by Google, Google had no hand in the conception or execution of the project.  For that I’m the only one to blame, unfortunately.

Following is a summary of the study from the ICLE website.  I plan to blog about the results and their implications in the coming days in a series of posts.

Google has been the subject of persistent claims that its organic search results are improperly “biased” toward its own content.  Among the most influential is an empirical study released earlier this year by Benjamin Edelman and Benjamin Lockwood, claiming that Google favors its own content “significantly more than others.”  The authors conclude in their study that Google’s search results are problematic and deserving of antitrust scrutiny because of competitive harm.

A new report released by the International Center for Law & Economics and authored by Joshua Wright, Professor of Law and Economics at George Mason University, critiques, replicates, and extends the study, finding Edelman & Lockwood’s claim of Google’s unique bias inaccurate and misleading. Although frequently cited for it, the Edelman & Lockwod study fails to support any claim of consumer harm — or call for antitrust action — arising from Google’s practices.

Prof. Wright’s analysis finds own-content bias is actually an infrequent phenomenon, and Google references its own content more favorably than other search engines far less frequently than does Bing:

  • In the replication of Edelman & Lockwood, Google refers to its own content in its first page of results when its rivals do not for only 7.9% of the queries, whereas Bing does so nearly twice as often (13.2%).
  • Again using Edelman & Lockwood’s own data, neither Bing nor Google demonstrates much bias when considering Microsoft or Google content, respectively, referred to on the first page of search results.
  • In our more robust analysis of a large, random sample of search queries we find that Bing generally favors Microsoft content more frequently—and far more prominently—than Google favors its own content.
  • Google references own content in its first results position when no other engine does in just 6.7% of queries; Bing does so over twice as often (14.3%).

The results suggest that this so-called bias is an efficient business practice, as economists have long understood, and consistent with competition rather than the foreclosure of competition. One necessary condition of the anticompetitive theories of own-content bias raised by Google’s rivals is that the bias must be sufficient in magnitude to exclude rival search engines from achieving efficient scale. A corollary of this condition is that the bias must actually be directed toward Google´s rivals. That Google displays less own-content bias than its closest rival, and that such bias is nonetheless relatively infrequent, demonstrates that this condition is not met, suggesting that intervention aimed at “debiasing” would likely harm, rather than help, consumers.

  • Shivaun

    I am the CEO of Foundem, one of the companies at the centre of the European
    Commission’s formal antitrust investigation into Google.


    In October 2009, we launched and defined search
    neutrality as the principle that search engine results should be comprehensive,
    impartial, and based solely on relevance. Clearly, no two search engines will
    produce the same search results; nor should they. But any genuine pursuit of
    the most relevant results must, by definition, preclude any form of arbitrary
    discrimination. The problem for Google is that its Universal Search mechanism
    and its increasingly heavy-handed penalty algorithms are both clear examples of
    arbitrary discrimination.


    By shoehorning prominent links to Google’s own services into its users’
    search results, Universal Search is transforming Google’s ostensibly neutral
    search engine into an immensely powerful marketing channel for Google’s
    secondary services.


    Crucially, Google uses different ranking algorithms and relevance
    signals to place its own services than it uses to place everyone else’s, giving
    Google absolute control over how aggressively and comprehensively it commandeers
    any given vertical. Foundem’s study of the UK and US price comparison markets,
    for example, found that Google consistently exploits this control to place its
    own price comparison service at or near the top of its search results for
    virtually all product- and price-comparison-related searches.


    At the recent US Senate Antitrust hearing into Google, Senator Lee
    referred extensively to this Foundem study during his questioning of Google
    Chairman Eric Schmidt. We have produced a video highlighting some of the more
    important inaccuracies and inconsistencies of Mr Schmidt’s testimony.  Anyone who would like to understand more
    about the subtext of the hearing or the ongoing antitrust investigations on
    both sides of the Atlantic might find this video interesting:


    Shivaun Raff

    CEO and Co-Founder of Foundem and (a Foundem

  • Dwayne Winseck

    By a series of conflations between unrelated things — i.e. search engine bias and “impact on consumers” (para 1) and “unique bias” (para 5) versus bias in general — you simultaneously twist the terms of debate and pave the way to your preordained outcome.  

    Anti-trust does not turn on demonstrating consumer harms but anti-competitive behaviour by industry players. Your conflation of two substitutes the former as a criteria of equal or greater weight. It is neither. 

    The focus on “unique bias” is also misleading, because the question is Google biasing its search results not whether this is industry-wide practice.  The latter is of interest, I would say. 

    The real conclusion, and title of your piece is, is really this, as indicated in Sherman post you reference: Bing more Biased the Google, full stop. 

    Actually, I would add, dominant players fiddle with rankings, for effect, then carefully lay out the details. 


Previous post:

Next post: