Jim Harper

Google Search Won’t Return Links to cato-at-liberty.org

I’ve run across the most curious thing today.

Searches on Google that should turn up the Cato@Liberty blog (at http://www.cato-at-liberty.org) do not return any result with that URL in it.

Berin took great care the other day to report on the temporary demotion of some Progress & Freedom Foundation content by the Google search engine. I want to do a similar, careful job with this because it’s a sensitive area.

Could I ask you, our visitors, to check what you get from Google? Visit Cato@Liberty and then craft the Google search that you think is most likely to return that Web site. (I’ve tried searching “site:cato-at-liberty.org the” for example, which would return instances of the word “the” on the cato-at-liberty.org domain, and gotten no results.)

Next, if you have any technical knowledge, please opine on what might be causing this to occur. Cato@Liberty is a fairly high-traffic site with a large following. Its disappearance from Google search results is unusual. Any ideas on how to get it restored would be welcome.

Update: It’s a problem with robots.txt on the site.

July 24, 2008 | Comments |

Viewing 14 Comments

    • ^
    • v
    As my suspicions about some of Google's motives lately have been growing faster than the National Debt, I've started using Lycos, and going to Google only in the rare instances that the latter returned insufficient results.

    Searches for "cato-at-liberty" and "cato-at-liberty.org" had some results worth mentioning.

    Searching on the domain name should return the website hosted there as result#1. Google didn't. Lycos and hotbot did.

    Searching for just "cato-at-liberty" on Google, returned ONE result from cato-at-liberty.COM ... On the others, "maybe a couple" for cato-at-liberty.org were returned.

    I'd absolutely *love* to be able to accuse Google of something evil, like censoring ideas they oppose, but the other engines denied me of that pleasure :(

    Can you get the access logs for the cato-at-liberty site? If so, you should search them for visits by the search engine "crawlers". Here's who visited me in the last day-and-a half:

    crawler4035.ask.com
    crawl-17.cuill.com
    msnbot-65-55-208-211.search.msn.com
    livebot-65-55-208-211.search.live.com
    crawl-66-249-71-66.googlebot.com

    .. search for "googlebot", for Google's crawler
    "msnbot", "livebot", for Microsoft's
    and, "crawl" for the others.

    I get visited daily. Cato-at-liberty should be crawled by each at least daily, maybe more.. If daily crawls are not happening, that's one thing to fix, otherwise, something is chasing them away, once the get there.

    You can't take potshots at this problem. You need to know whether the 'bots are trying to do their job, first.

    Feel free to use my email address to contact me if you'd like me to help further.

    Regards,
    Nick
    • ^
    • v
    "Dissatisfied? Help us improve" http://www.google.com.au/quality_form has just received this report from me:

    =====
    I was looking for Cato-at Liberty articles around the same time as a specific article by David Boaz, that I know contains the text "remains a temptation to our current candidates".

    The Google result shows the article at David Boaz's own site, but *not* the same article at the URL http://www.cato-at-liberty.org/2008/06/30/our-c...

    The cato-at-liberty site should be prominent in the results, as it is linked to by many other sites, and the very same article at a different site appears in the search results.

    Specifying "site:cato-at-liberty.org" in the search criteria returns *no* results. The expected behaviour is that the site should have its results along with any others that qualify according to the criteria.
    • ^
    • v
    The very same search at clusty.com http://clusty.com/search?query="remains+a+tempt... returns results on cato-at-liberty.org without fuss.
    • ^
    • v
    Have you tried Google Webmaster Tools? There might be a hint in the dashboard why pages aren't getting crawled.

    https://www.google.com/webmasters/tools/dashboa...
    • ^
    • v
    The site configuration is screwed-up. How do you folks expect to rule the world via sheer force of reason if you can't even run a website properly? :-)
    • ^
    • v
    Just slightly out-helpfuling Seth, Tim Lee has figured out that the site's use of robots.txt is sending Google packing. I'm sure Cato's Web staff will fix it in the morning. In the meantime, I'm glad I didn't assume the worst of Google.
    • ^
    • v
    Despite the technical fix, I'm sure our buddy Scott Cleland will still find a way to concoct a pretty good Google conspiracy theory out of all this!
    • ^
    • v

    Searches on Google that should turn up the Cato@Liberty blog (at http://www.cato-at-liberty.org) do not return any result with that URL in it.

    Berin took great care the other day to report on the temporary demotion of some Progress & Freedom Foundation content by the Google search engine. I want to do a similar, careful job with this because it’s a sensitive area.

    Next, if you have any technical knowledge, please opine on what might be causing this to occur. Cato@Liberty is a fairly high-traffic site with a large following. Its disappearance from Google search results is unusual. Any ideas on how to get it restored would be welcome.

    Update: It’s a problem with robots.txt on the site.


    After implying that google wasn't playing fair with Cato at Liberty, and then finding out the problem was with the Cato at Liberty site itself, I would think that at least an apology would be in order.

    In the mean time do a search on "Public issue China" and see who turns up number one....
    • ^
    • v
    I'll happily do so - just as soon as you point to where I implied that Google wasn't playing fair with Cato@Liberty. I felt like I took pains not to do that, and, reading back over what I wrote, I think I did pretty well.
    • ^
    • v
    Jim:

    As I very clearly said you implied that google may not have been fair with Cato at Liberty. Do you know what implied means?

    im·plied /ɪmˈplaɪd/

    –adjective
    involved, indicated, or suggested without being directly or explicitly stated; tacitly understood: an implied rebuke; an implied compliment.

    You implied that google wasn't being fair here:

    Searches on Google that should turn up the Cato@Liberty blog (at http://www.cato-at-liberty.org) do not return any result with that URL in it.

    Berin took great care the other day to report on the temporary demotion of some Progress & Freedom Foundation content by the Google search engine.

    So when will you apologize?
    • ^
    • v
    Oh, and let's not forget the title of the post, which goes beyond implication:

    Google Search Won’t Return Links to cato-at-liberty.org

    But, given the standard of behavior here at TLF, which has been really disgusting at times, with Adam T. using all sorts of vile names and Jerry Brito deleting my posts, to expect suddenly a polite standard of personal conduct from a libertarian is a little naive on my part, I suppose...
    • ^
    • v
    Thanks for the definition, though I already have a handle on the meaning of the term "implied." You realize, don't you, that providing a definition of "implied" implies that I didn't already know it. That's condescending and rude. Let's come back to that, though.

    You point to my citation of Berin being careful not to imply or state that Google had done anything wrong as evidence that I was trying to suggest it? That's a verry close call. I was trying to be as careful as Berin, while holding out the possibility - because it was one, though remote. I was most suspicious of some neutral, if arguably inappropriate, functioning of the Google crawler, such as the one reported on TechDirt the other day. In the last paragraph, I asked people to offer technical explanations and fixes. The strongest inference is that I was searching for a technical explanation and fix. I said nothing about content-based treatment of TLF - as if TLF would be important enough to suffer some derogatory treatment by Google. Please.

    S0 should one apologize for holding out the possibility of some wrongdoing, while carefully seeking all explanations for a thing happening? I just don't think you've got a legitimate beef here.

    Which brings us back to you. Your condescension here, and your pursuit of this infinitesimal slight against Google, are just two among many examples of a unique maladjustment that you portray in your comments here on TLF. Adam, Jerry, myself, and others interact with dozens and dozens of people online. When one of them is consistently difficult to deal with, chances are that it has something to do with that one. So consider the possibility that it's not TLF or any of its authors. Consider the possibility that it's you. I suppose we could be complimented that you obsess over the content of this site, but your participation here doesn't really add value to anyone's experience - writer or visitor.

    Here's an idea that could solve all these problems and make everyone's life better - your own included: stop reading TLF. Stop commenting here. Get another hobby! There are plenty of options for you. There are thousands of communities where you might find a more pleasurable experience and be more pleasurable to others. You might find it a relief to just walk away from TLF and never look back.

    Over and out. I won't be replying to further comments.
    • ^
    • v
    Which brings us back to you. Your condescension here, and your pursuit of this infinitesimal slight against Google, are just two among many examples of a unique maladjustment that you portray in your comments here on TLF. Adam, Jerry, myself, and others interact with dozens and dozens of people online.

    Hey, talk about thin-skinned--I just noticed that you had implied that Google was manipulating the search results for Cato@liberty.

    If you'd simply admitted that you'd been wrong and apologized I wouldn't have had to post the definition of imply.

    And it is fair to say that you'd implied something here--without actually saying it. I'd called you on that and you got upset. That's your problem, not mine.

    In any case there are poster here--Tim Lee and Ryan Radia, for example that I quite like, most of the time, even though I don't always agree with them.

    They, however, don't descend into name calling or deleting my posts. Why is that?
    • ^
    • v
    Here’s an idea that could solve all these problems and make everyone’s life better - your own included: stop reading TLF. Stop commenting here. Get another hobby!

    I feel obligated to comment on some of what I believe is corporate mis-information. TLF-libertarianism of Jerry and Adam is often freedom-depriving ideology cloaked in the very language of freedom. I find that something that should be exposed. If you have an open forum you may actually get comments you disagree with! Grow up!
 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus