Evidence reveals MSN using editors to maintain quality in its search results,
Written by power
@ 11:03 AM
permanent link on
Tuesday, January 24, 2006
| Post a Comment |
Some of our referral logs show visits from this URLhttp://64.4.8.28/hrsv3/Judging.aspx , at first look it seems like a spam site but it is not the case,
We see a login screen when we visit that URL mshrs.search.msn-int.com/hrsv3/Login.aspx it looks like human review area for MSN search results,
Google has been doing human review using eval.google.com for a longtime,
What googleguy said about eval.google.com
"walkman, your comment illustrates a misconception that I've seen in a couple places. The system that was up at eval.google.com was a console to evaluate quality passively, not to tweak our results actively. But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all.
I have serious reservations about Henk van Ess taking information from one of his own students (who presumably signed a non-disclosure agreement when the student agreed to help rate the quality of our results) and posting that information online. I also believe these web pages said things like "Google Proprietary and Confidential," but it appears that the screenshots have been cropped to exclude that information. Those are the two things that really made me sad, not the "breaking news" the Google evaluates its own results quality. It shouldn't be a surprise that Google evaluates the quality of its results in lots of ways--the fact is that every major search engine evaluates its relevance in many ways. "
I said
But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all. and you replied
Google Guy, do I read between the lines that you think my postings are irrelevant and misleading? That would be a shame.
I don't believe they're irrelevant, but yes: I do believe that the assertions you've made are misleading. In my original post, I was replying to walkman, who asked "ok, so how do you know you've been manually hit by this?" which implies that walkman thought that eval.google.com was responsible for sites being hit. Likewise, I have a ton of respect for Tara Calashain at ResearchBuzz. But her post about your site says "Basically what Henk seems to have found is a part of Google that allows humans to tweak search results to ostensibly get rid of spam and let the most contextually-relevant search results rise to the top." Again, Tara wonders whether your posts said that results were being directly tweaked. Then there are assertions from your site like "The Google testers are paid $10 - $20 for each hour they filter the results of Google." "Filter" again makes it sound like an active process. And your self-submission to Slashdot ("Real people, from all over the world, are paid to finetune the index of Google"), which also gives the impression that people used eval.google.com to change our search results.
So yes, I looked at the wording from when you submitted your own site to Slashdot, plus the use of active verbs such as "filter" on your own site, plus the comments of smart people such as Tara and walkman and how they interpreted what you wrote, and in my opinion your posts have been misleading. Again, this was not a console in which people could directly fine-tune, tweak, filter, or otherwise modify our search results. eval.google.com was for "eval," i.e. passive evaluation.
Your follow-up question was "Why pay them for something if it has no effect om the index? Must be charity then." Why are you surprised that we would pay people to rate search results? The job posting has been public, after all. We do provide ways for people to volunteer to help Google (e.g. see our translation console at http://services.google.com/tc/Welcome.html ), but to rate search results consistently and well takes time and training. I think it's perfectly normal to pay people for their time.
When you quoted me on your site, you said "Google Guy: I've serious reservations about Henk van Ess" and in your post you said "Google's spokesmen Google Guy, who I love to read, has serious reservations about me." Just to be clear, that's not accurate: I don't have reservations about you personally, Henk. I think I stated clearly that I have serious reservations about two of your actions. I mentioned those two specific things in my first post, and I'll reiterate them: you took information from one of your students, and you posted information that (in my opinion) was clearly proprietary/confidential. Regarding the first, I believe you wrote in a comment on your own site that this information came from a student of yours? Regarding the second, I'm quite surprised that you assert "I'm not aware of restrictions." Besides the copyright symbol that you mentioned earlier, the very first picture you posted has a link "An NDA Reminder..." on the left in the Important Announcements section, where NDA stands for non-disclosure agreement. Are you honestly saying that if you had realized there were restrictions, you wouldn't have done five blog posts (so far), posted screenshots, posted employee's real names on the web without consulting them, and posted two training documents? In that case, I'll ask politely. Henk, this information was for ratings training. It's copyrighted, and I'm sure that the evaluation group considers it proprietary/confidential. I'd appreciate it if you would stop posting these documents.
By the way, I apologize in advance if this post comes across as strident. I hate he-said-she-said stuff, and normally I try not to post when I'm at ruffled at all. But I do think that things like posting an innocent employee's name from internal training documents is rude and unnecessary. Henk, feel free to include this entry on your blog, but if you do, I'd appreciate if you'd quote the entire post.
then we have yahoo's human review of search results, we can see referrals from corp.yahoo domain
now we have MSN human review of search, I think its mostly for quality control purpose? anyway good to see they are hand reviewing search, their results are spammed a lot by search engine spammers,
We see a login screen when we visit that URL mshrs.search.msn-int.com/hrsv3/Login.aspx it looks like human review area for MSN search results,
Google has been doing human review using eval.google.com for a longtime,
What googleguy said about eval.google.com
"walkman, your comment illustrates a misconception that I've seen in a couple places. The system that was up at eval.google.com was a console to evaluate quality passively, not to tweak our results actively. But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all.
I have serious reservations about Henk van Ess taking information from one of his own students (who presumably signed a non-disclosure agreement when the student agreed to help rate the quality of our results) and posting that information online. I also believe these web pages said things like "Google Proprietary and Confidential," but it appears that the screenshots have been cropped to exclude that information. Those are the two things that really made me sad, not the "breaking news" the Google evaluates its own results quality. It shouldn't be a surprise that Google evaluates the quality of its results in lots of ways--the fact is that every major search engine evaluates its relevance in many ways. "
I said
But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all. and you replied
Google Guy, do I read between the lines that you think my postings are irrelevant and misleading? That would be a shame.
I don't believe they're irrelevant, but yes: I do believe that the assertions you've made are misleading. In my original post, I was replying to walkman, who asked "ok, so how do you know you've been manually hit by this?" which implies that walkman thought that eval.google.com was responsible for sites being hit. Likewise, I have a ton of respect for Tara Calashain at ResearchBuzz. But her post about your site says "Basically what Henk seems to have found is a part of Google that allows humans to tweak search results to ostensibly get rid of spam and let the most contextually-relevant search results rise to the top." Again, Tara wonders whether your posts said that results were being directly tweaked. Then there are assertions from your site like "The Google testers are paid $10 - $20 for each hour they filter the results of Google." "Filter" again makes it sound like an active process. And your self-submission to Slashdot ("Real people, from all over the world, are paid to finetune the index of Google"), which also gives the impression that people used eval.google.com to change our search results.
So yes, I looked at the wording from when you submitted your own site to Slashdot, plus the use of active verbs such as "filter" on your own site, plus the comments of smart people such as Tara and walkman and how they interpreted what you wrote, and in my opinion your posts have been misleading. Again, this was not a console in which people could directly fine-tune, tweak, filter, or otherwise modify our search results. eval.google.com was for "eval," i.e. passive evaluation.
Your follow-up question was "Why pay them for something if it has no effect om the index? Must be charity then." Why are you surprised that we would pay people to rate search results? The job posting has been public, after all. We do provide ways for people to volunteer to help Google (e.g. see our translation console at http://services.google.com/tc/Welcome.html ), but to rate search results consistently and well takes time and training. I think it's perfectly normal to pay people for their time.
When you quoted me on your site, you said "Google Guy: I've serious reservations about Henk van Ess" and in your post you said "Google's spokesmen Google Guy, who I love to read, has serious reservations about me." Just to be clear, that's not accurate: I don't have reservations about you personally, Henk. I think I stated clearly that I have serious reservations about two of your actions. I mentioned those two specific things in my first post, and I'll reiterate them: you took information from one of your students, and you posted information that (in my opinion) was clearly proprietary/confidential. Regarding the first, I believe you wrote in a comment on your own site that this information came from a student of yours? Regarding the second, I'm quite surprised that you assert "I'm not aware of restrictions." Besides the copyright symbol that you mentioned earlier, the very first picture you posted has a link "An NDA Reminder..." on the left in the Important Announcements section, where NDA stands for non-disclosure agreement. Are you honestly saying that if you had realized there were restrictions, you wouldn't have done five blog posts (so far), posted screenshots, posted employee's real names on the web without consulting them, and posted two training documents? In that case, I'll ask politely. Henk, this information was for ratings training. It's copyrighted, and I'm sure that the evaluation group considers it proprietary/confidential. I'd appreciate it if you would stop posting these documents.
By the way, I apologize in advance if this post comes across as strident. I hate he-said-she-said stuff, and normally I try not to post when I'm at ruffled at all. But I do think that things like posting an innocent employee's name from internal training documents is rude and unnecessary. Henk, feel free to include this entry on your blog, but if you do, I'd appreciate if you'd quote the entire post.
then we have yahoo's human review of search results, we can see referrals from corp.yahoo domain
now we have MSN human review of search, I think its mostly for quality control purpose? anyway good to see they are hand reviewing search, their results are spammed a lot by search engine spammers,
0 Comments:
Post a Comment
Links to this post:
Create a Link
<< SEO Blog Home