Advanced search

Message boards : News : Validator issues

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 22
Images Observed: 774

              
Message 714 - Posted: 21 Jul 2013, 21:26:54 UTC

Hi Everyone,

It looks like there have been some issues with the validator and videos being repeatedly shown to people when they really shouldn't. I should be implementing a few fixes for this tuesday (I have an important proposal to finish up for tomorrow). Here's the current plan:

1. I'm going to change how validation works. FIrst, videos will be shown to a maximum of 3 people for validation. If there is no perfect agreement by at least 2 of the three, then I'll have a webpage for Susan, her students and myself to look at to select what the correct observations are for that video and award credit accordingly. I think I'll also make a webpage to display these videos so people can get a better understanding of the tricky cases and why the observations were what they were.

2. I'm pretty sure I will implement a negative credit penalty for invalid observations The point of this is to make people more careful in video watching since there have been a few complaints about this. However, you'll be able to get this credit back, as in the next point.

3. I'm going to make a webpage where you can go over videos you've already watched that were marked invalid; which will let you update your observations to remove the credit penalty and to get partial (or maybe full?) credit back if they were previously marked invalid.

I'll leave this open for comments and suggestions, before I start implementing things tuesday.

cheers,
--Travis

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 716 - Posted: 21 Jul 2013, 22:49:09 UTC - in response to Message 714.
Last modified: 21 Jul 2013, 23:40:20 UTC

The main issue I have with giving back full credit for correcting mistakes is that it removes the necessity to try to do a good job the first time. Someone could fly through a bunch of videos knowing they will be able to correct their mistakes and still get just as much credit.

Will the correction page show what the validated result was? Or will the user have to watch the video again in an attempt to get the right result?

Will there be a mechanism for the viewer to make the claim that his initial marking was actually correct and it was the wingmen that got it wrong?

Edit: Will the penalties be retroactive?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 22
Images Observed: 774

              
Message 717 - Posted: 22 Jul 2013, 1:40:56 UTC - in response to Message 716.

The main issue I have with giving back full credit for correcting mistakes is that it removes the necessity to try to do a good job the first time. Someone could fly through a bunch of videos knowing they will be able to correct their mistakes and still get just as much credit.


True, maybe fixing the error should just get rid of the negative credit penalty, but not provide credit.

Will the correction page show what the validated result was? Or will the user have to watch the video again in an attempt to get the right result?


Not sure, open to suggestions here.

Will there be a mechanism for the viewer to make the claim that his initial marking was actually correct and it was the wingmen that got it wrong?


I could add a report this validation button to the page that shows up when a video observation is made, which would flag those observations for review as well -- even if the wingmen validated it incorrectly. Or our review page would show all the videos with any conflicting observations.


Will the penalties be retroactive?


No, too much work for not much benefit, especially as the project is still in pretty alpha stages.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 718 - Posted: 22 Jul 2013, 2:15:09 UTC - in response to Message 717.

The main issue I have with giving back full credit for correcting mistakes is that it removes the necessity to try to do a good job the first time. Someone could fly through a bunch of videos knowing they will be able to correct their mistakes and still get just as much credit.


True, maybe fixing the error should just get rid of the negative credit penalty, but not provide credit.


I guess I'm not seeing much disincentive here to prevent someone from just marking what they see in the initial frame and submitting. They get a penalty when the video is finally validated (presuming no one else is doing the same thing), but can get the credits back. Even taking into account the time it would take to make the correction, someone who takes such a shortcut could probably still go through three times as many videos as someone who watches the whole video.

In other words, how would this be different from what we currently have?

Will the correction page show what the validated result was? Or will the user have to watch the video again in an attempt to get the right result?


Not sure, open to suggestions here.


I would opt for not showing the validated result, but showing the invalid result that the viewer originally submitted. The viewer would have to watch the entire video, whereupon he could make a claim that his original marking was indeed correct, or mark the corrections as appropriate.

Profile DoctorNow
Avatar
Send message
Joined: 31 Jan 12
Posts: 328
Combined Credit: 2,333,912
DNA@Home: 94,617
SubsetSum@Home: 162,940
Wildlife@Home: 2,076,355
Wildlife@Home Watched: 1,977,485s
Wildlife@Home Events: 589
Climate Tweets: 128
Images Observed: 2,678

              
Message 719 - Posted: 22 Jul 2013, 3:42:20 UTC
Last modified: 22 Jul 2013, 3:47:44 UTC

No objections from me so far, we'll see how and if this works. ;-)

Edit:
Newbies may complain about getting minus credits when their videos are not right.
This may disencourage them to continue watching, even when the penalty can be reversed.
____________
Wildlife & SubsetSum Badges-creator ;-)
Mod/Guru @ the german team BOINC@Heidelberg
My BOINC-Stats / My Badges

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 720 - Posted: 22 Jul 2013, 3:53:40 UTC - in response to Message 719.

How about a graded penalty adjustment based on user credits? A viewer with under 10k credits gets 100% penalty adjustment for making corrections. A viewer with 10k-100k credits gets 75% penalty adjustment, from 100k to 1000k gets 50% adjustment, and over 1000k gets no adjustment.

Profile barblovesroses
Send message
Joined: 22 Feb 13
Posts: 41
Combined Credit: 7,345
DNA@Home: 331
SubsetSum@Home: 7,014
Wildlife@Home: 0
Wildlife@Home Watched: 87,622s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

  
Message 721 - Posted: 22 Jul 2013, 5:57:35 UTC - in response to Message 718.

I think this is hard to justify at all because viewers don't see that their result was questionable. We don't know that someone else disagreed with our result. Its only subsequent viewers who see that a result differed from our result.

The only way that something like this would be fair is if our own result came back to us once a viewing happened that differed from our own result so we could review it once more to see if we were willing to change our original result to agree with the differed result. Its not always our result that is wrong. It may be the result from the 2nd or third viewer who is wrong. If we aren't willing to change our result, then the 2nd or third viewer needs an opportunity to review their result as well until all viewers are in agreement.

This could be a nightmare to manage in the system so your proposal of a review panel to oversee these video's that are in question is probably the best way to handle them.

As to adjusting credit, reversing out and giving negative credit doesn't seem fair, but maybe reducing credit by a percentage. The person still gave the same effort in watching the video and making a report of their observation. Reducing by maybe 25% seems fair but any more than that seems too much considering the same effort went into watching the video and making a report.

Profile Eric_Kaiser
Send message
Joined: 12 Jun 13
Posts: 11
Combined Credit: 7,445
DNA@Home: 0
SubsetSum@Home: 7,445
Wildlife@Home: 0
Wildlife@Home Watched: 939,570s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

  
Message 722 - Posted: 22 Jul 2013, 15:49:05 UTC - in response to Message 721.

O.k. only click unsure-option in the future...
I wrote my thoughts of penalty in the other thread.
Go your way. I'm thinking about mine.
I'm not sure if I will give it a try.

Profile STE\/E
Avatar
Send message
Joined: 5 Apr 13
Posts: 416
Combined Credit: 29,783,819
DNA@Home: 2,634,206
SubsetSum@Home: 735,231
Wildlife@Home: 26,414,382
Wildlife@Home Watched: 53,380,530s
Wildlife@Home Events: 9,349
Climate Tweets: 0
Images Observed: 0

          
Message 723 - Posted: 22 Jul 2013, 18:29:17 UTC - in response to Message 720.

How about a graded penalty adjustment based on user credits? A viewer with under 10k credits gets 100% penalty adjustment for making corrections. A viewer with 10k-100k credits gets 75% penalty adjustment, from 100k to 1000k gets 50% adjustment, and over 1000k gets no adjustment.


Sounds good to me, I know it took me awhile to figure things out, if I would have been Penalized at the start I would have just dropped the Project, but now I make so few errors a loss of credit now & then won't bother me ...
____________

Profile barblovesroses
Send message
Joined: 22 Feb 13
Posts: 41
Combined Credit: 7,345
DNA@Home: 331
SubsetSum@Home: 7,014
Wildlife@Home: 0
Wildlife@Home Watched: 87,622s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

  
Message 725 - Posted: 22 Jul 2013, 21:45:26 UTC - in response to Message 714.

Hi Travis,

" I think I'll also make a webpage to display these videos so people can get a better understanding of the tricky cases and why the observations were what they were."

Why not just add a separate section to this message board for problem videos and corrections and then post them in here so that the people who are interested in learning can come in here to review the videos that were problems and learn from them. Going to a separate web page sounds like a real hassle to me.


" I'm going to make a webpage where you can go over videos you've already watched that were marked invalid; which will let you update your observations to remove the credit penalty and to get partial (or maybe full?) credit back if they were previously marked invalid."

How do you plan to notify people that they have videos to be reviewed in order to have future credit given or penalized?

Again, a separate webpage??? There has to be some way to keep it part of this already existing site even with all its limitations.

Does this mean that you are thinking against having Susan and her students not review the videos in question, or are you still planning to have them do so and then still resubmit them to the original viewers for a revision to their original submission? I'm just trying to understand what your plan is here.




Post to thread

Message boards : News : Validator issues