Google Using Human Evaluation

Patented

Integrating humans with the algorithm - http://www.webmasterworld.com/google/3060951.htm

Training documents for these evaluators have been leaked.

Lots of myths are abounding about Google!

It's possible to spot the evaluators visiting a site.

google.com/evaluation/search/rating/task-edit?task=

The question is whether these alleged 10,000 human evaluators are actually improving the google results.

Google employee count is currently around 10,600.

Google is probably counting lots of part time effort when it talk about 10,000 evaluators
12,238 full time employees as of March 31, 2007

Article about this:

http://www.searchbistro.com/index.php?/archives/23-Googles-Human-Quality-Evaluation-How-To-Spot-Offensive-Sites-Googles-Whitelist.html
- Google testers - $10-$20/hr.
- Someone in thread claims to have made $15/hr.
  - Google interview / testing process takes 10 hours, but is paid time.

Example advertisement for Google quality rater
QUALITY RATER - (SPANISH, DUTCH, ITALIAN, FRENCH) This is a temporary role offered through Kelly Services. Google Inc. is recruiting part-time, temporary, home-based workers to help with work on a search quality evaluation on a project basis. You would work at your own pace, and the time and length of any particular work session would be up to you. Candidates will evaluate search results and rate their relevance....

Seattle division of google may be reading blogs and forums.

On a case for lost search engine ranking, court affirmed Google's first-amendment right to adjust rankings or remove pages from its search results.

Garnet's thought: A developed a million page internet directory that rivalled the size of Looksmart. I was one guy and algorithms against a 250 editors at Looksmart. I ended up with 75000 visitors a day, pre-2000! I said for a long time, the biggest challenge was how to add the human element back into the algorithm.

Looks like Google has finally figured that out. A blog post by Matt Cutts

Marissa Mayer : "Up to today we have relied on automation, but I believe the future will be a blend of both, combing the scale of automation and human intelligence."
Larry Page: "Larry says search is finding content... and that Wikipedia found a better way to organize information. he seems to like the model of using humans and process and machines."
Matt Cutts: "So I think too many people get hung up on "Google having algorithms." They miss the larger picture, which (to me) is to pursue approaches that are scalable and robust, even if that implies a human side. There's nothing inherently wrong with using contributions from people-you just have to bear in mind the limitations of that data."
Udi Manber said: "If somebody says they don't like a particular search result that's a signal. So, we've been using that for a long time, and we are working on new ways of using it."