Not-for-profit Scroogle, the privacy-friendly Google search scraper, has fallen victim — to Google.
Because this isn’t the first time Google, America’s primary online advertising and data mining company, has muscled tiny Scroogle out of the way.
Back in May, 2010, when I still owned and ran p2pnet, I posted a story headed up Google nails Scroogle,
I adopted Daniel Brandt’s Scroogle as the official p2pnet search engine many moons ago, and it’s done a good job for us.
It describes itself as an “ad-free Google search proxy which prevents the searcher’s data being stored by Google, a Firefox plugin, and tools for webmasters.”
It used to, rather. Because now, when you try a search, this is what you get:
“We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we’ve been scraping since Scroogle started five years ago. Actually, we’ve been using that interface for scraping since Google-Watch.org began in 2002.”
Nor is the first time a Biggie has interfered with Scroogle.
“Scroogle scrapes Google’s website to return its search results without ads – bypassing the Google cookie, and protecting the user’s privacy, because Google is unable to match the searches to any other information- such as your IP address or your GMail account”, said The Register three years ago.
Microsoft’s MSN Messenger service “doesn’t want you talking dirty – and its definition of dirty talk is quite peculiar”, said the story.
“If you send an instant message containing the word ’scroogle.org’ via the Microsoft service, the message never arrives. The sender doesn’t know it was discarded, and the recipient has no indication that it was ever sent, as the original message remains in the chat window and history.”
Scroogle “putters along, makes around 50,000 scrapes per day, without being sued” it goes on, adding, “As Google has failed to challenge the legality of the service, it’s an odd choice of domain for Microsoft to ban.”
Now, “This interface (here’s a sample from years ago) was remarkably stable all that time”, Daniel says, going on:
During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.
That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn’t show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google’s basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched “Universal Search,” which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.
Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google’s main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.
Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.
Thank you for your support during these past five years. Check back in a week or so; if we don’t hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.
In solidarity with Scroogle, we’ll leave things as they are at least until tomorrow.
But if worse comes to worse, any ideas about replacements?
For now, “PS”, Daniels says, adding:
“Many thanks for all the support emails. A word to those who are emailing us with links to simplified Google pages with just the initial search entry box on them: Sorry, but this doesn’t help us. It’s the simplicity of the pages with search results that matter. Scroogle uses it’s own gateway pages, but we parse out the links from Google’s results page before we pass them back to you.
“See the “sample” link in the second paragraph of this page, where you have to mouse-over each link to see the snippet. Results like this were not too difficult to parse. More importantly, they were stable. The query to Google that produced results like this is illustrated by the URL behind the ‘Next’ link at the bottom of that sample (which no longer works, obviously). The parameters in URLs like that could be manipulated for language selection and number of results per page, just like for all Google searches.
“Whether Google cares to restore this simple interface is a question of fundamental corporate policy. Frankly, we’ve always felt that we’d reach this point with Google sooner or later. Years ago it was Microsoft’s decision in a different context, before web search engines even existed:
“From Barbarians Led by Bill Gates: Microsoft from the Inside by Jennifer Edstrom and Marlin Eller (New York: Henry Holt, 1998), p. 117. Eller was Microsoft’s lead developer for graphics on Windows from 1982 to 1995:
Microsoft didn’t want a lot of other companies writing code that could compete. It wanted to keep the barriers to entry very high. The idea, in fact, was to keep raising the bar, putting in more layers of software and APIs, which developers would then have to support. Microsoft wanted to make it so gnarly that anybody who couldn’t devote a team of one hundred programmers to every Windows application would be out of the game.
“Microsoft made a decision for corporate protectionism, and even today many people will never forgive them for their past arrogance. Let’s hope Google doesn’t do the same thing.
It’s not as if Google needs the money — but then, neither did Microsoft.”
Jon Newton My Blog — Dammit
the item below is a straight lift from the Wikipedia ….
screenshot of Scroogle.org as of 2008-10-14
|Type of site||Search Engine|
|Available language(s)||English and 27 others|
|Owner||Public Information Research, Inc.|
|Created by||Daniel Brandt|
Scroogle was a web service that allowed users to perform Google searches anonymously. It focused heavily on searcher privacy by blocking Google cookies and not saving log files. The service was launched in 2003 by Daniel Brandt. After 2005, the service encountered rapid growth before running into a series of problems starting in 2010. In February 2012, the service was permanently shut down by its creator due to a combination of throttling of search requests by Google and a denial-of-service attack by an unknown person or group. Before its demise, Scroogle handled around 350,000 queries daily and ranked among the top 4,000 sites worldwide in traffic.
Scroogle offered a web interface and browser plugins for Firefox, Google Chrome, and Internet Explorer that allowed users to run Google searches anonymously. The service scraped Google search results, removing ads and sponsored links. Only the raw search results were returned, meaning features such as page preview were not available. For added security, Scroogle gave users the option of having all communication between their computer and the search page be SSL encrypted. Although Scroogle’s activities technically violated Google’s terms of service, Google generally tolerated its existence, whitelisting the site on multiple occasions.
Scroogle promised to delete all search terms within an hour and block Google’s cookie files. All logs were deleted weekly, compared to the two years Google saves log files for. “Even if the feds come around and ask me questions I don’t know the answer because I don’t have the logs any more,” claimed company president Daniel Brandt.
The service’s name was a reference to both scraping and Ebenezer Scrooge. Scroogle was supported solely by user donations – an average of $43/day – and run by not-for-profit Public Information Research. At the time of the its demise, Scroogle was running off a total of 6 servers.
Scroogle was created in 2003 by Google critic Daniel Brandt, who was concerned about Google collecting personal information on its users. Brandt said he created the service out of frustration of Google’s transition from an “intellectual technical enterprise” to a big business focused on making money. “They (Google) have nothing but contempt for personal privacy,” Brandt remarked, noting that among Google’s stated goals was to collect more user information going forward. Scroogle was soon blocked by Google. In response, the service was moved to a new server and the source code published so that any who so desired could run Scroogle Scraper on their own computer. Brandt hoped the move would elicit a legal response by Google. He speculated that such a lawsuit would either make Google look bad, or his activity would be ruled legal and the case would “begin to restore a public-interest balance to the Web that has been declining ever since big money got behind the dotcoms.”
Starting in 2005, Scroogle drew expanded media attention and experienced rapid growth. In October of 2005, daily visits numbered around 8,000. A June 2007 story in WorldNetDaily claimed that Scroogle was becoming the preferred search engine of civil libertarians. By December 2007, Scroogle had passed 100,000 visitors a day. By May 2010, daily queries were around 325,000. Brandt attributed the site’s growth to consumers’ growing privacy concerns. In November 2010, CNET reviewed the Scroogle Scraper extension for Google Chrome, giving it 5 stars (out of 5). “Scroogle Scraper lets you take advantage of Google’s search engine without compromising your privacy or allowing your browsing and searching habits to be recorded. We recommend it for all Chrome users,” wrote the reviewer.
In 2010, Scroogle was temporarily offline multiple times, generating headlines about Google blocking the service. In May 2010, Google removed a seldom used interface page that Scroogle relied on causing the site to no longer function properly. Google developer Matt Cutts indicated that Google was not specifically targeting Scroogle, but rather was no longer supporting Internet Explorer 6 for which the interface had been designed. “Due to significant decrease in usage, we discontinued this interface and are encouraging searchers to upgrade their browser,” the company said in a statement. Brandt tweaked his site to rely on a different version of the interface page, which was subsequently removed in July. This time, Brandt was forced to do a complete redesign, which resulted in increased server load on his end. Scroogle went down a third time, when Google redesigned its interface in November 2010.
In 2012, Scroogle began to have problems with Google throttling, being periodically blocked for 90 minute intervals. Google denied targeting Scroogle, saying the site may have been triggering automated anti-spam mechanisms. In February 2012, Scroogle was shut down by its creator. Brandt blamed a DDoS attack by someone with “a personal vendetta” combined with throttling by Google for the site’s demise. Brandt’s other sites – NameBase.org, Google-watch.org, CIA-on-campus.org, and book-grab.com – were also taken offline at the same time.
According to Alexa data, Scroogle was ranked in the top 4000 sites globally in terms of traffic at the time of its demise. It ranked in the top 2500 for the United States, Canada, The United Kingdom, Australia, and other countries. The site was handling around 350,000 searches a day.
 See also
- ^ “PIR Staff Box”. Retrieved 2008-06-25.[dead link]
- ^ a b c “WHOIS – scroogle.org”. Retrieved 2008-06-25.
- ^ a b “Fed up with Google? Try Scroogle.org: Powerful search tool without privacy violations”. WorldNetDaily. 2007-06-04.
- ^ “A note about SSL: How Scroogle’s SSL option protects your privacy”. PIR. 2008-06-07.[dead link]
- ^ a b c d Jeffries, Adrianne (2012-02-21). “Scroogle, Privacy-First Search Engine, Shuts Down for Good | Betabeat — News, gossip and intel from Silicon Alley 2.0”. Betabeat.com. Retrieved 2012-02-27.
- ^ a b c d Patrick Beuth (2012-02-15). “Anonymisierte Google-Suche über Scroogle ist blockiert” (in German). Die Zeit Online.
- ^ a b c Rush, Dominic (2007-12-16). “Fears mount over internet privacy: Google rival Ask.com is promising to wipe out people’s search records within hours. But do the data really disappear?”. The Sunday Times.
- ^ a b c Stonehouse, David (2005-06-18). “Searching for gold”. The Age.