Advanced search

Message boards : News : updating observations with new validator, and team stats

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1673 - Posted: 22 Sep 2013, 18:52:15 UTC

Hi Everyone,

I'll be running the new validator over everything today. I'll be re-generating credit (although I don't expect anything to be very different) as well as re-calculating accuracy. Part of the reason I'm doing this is it will also make it significantly easier to generate all the team statistics (as I should have the top lists for teams done by the end of the day as well).

Here's a run down of the main changes. Validation will work as folllows:

1. I calculate the totals for each yes/no/unsure for each observation marking (eg., bird leave, bird return, etc) for a video that has enough for validation.

2. from these totals, i calculate what we would expect to have as the canonical result (if it is possible to have a canonical result). Basically, for each observaton marking (yes, no, unsure) the canonical one would be the the total which is >= 2, and also is greater than the other two, so, if there are 3 yes, 2 no, 1 unsure, then the expected canonical marking is yes. If there are 2 yes, 2 no, 1 unsure, then there's no consensus so there's no possible canonical marking yet.

3. If there is an observation which matches the expected canonical marks, then this is the canonical result. All marks that match it exactly are valid and an accuracy rating of 1.0, others get assigned an accuracy rating equal to what % of the marks matched the canonical one. Unsure matches yes/no for half credit/accuracy here.

4. if there is no observation which matches the expected canonical marks, or it wasn't possible to get a set of expected canonical marks yet, the video will be shown to someone else (with a maximum of 5 views) to try and get a canonical result.

The other big change here is accuracy isn't binary, and valid/invalid doesn't matter as much. Your accuracy rating will be calculated as the average accuracy rating over all your observations.

--Travis

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 1674 - Posted: 22 Sep 2013, 19:08:47 UTC - in response to Message 1673.

2. from these totals, i calculate what we would expect to have as the canonical result (if it is possible to have a canonical result). Basically, for each observaton marking (yes, no, unsure) the canonical one would be the the total which is >= 2, and also is greater than the other two, so, if there are 3 yes, 2 no, 1 unsure, then the expected canonical marking is yes. If there are 2 yes, 2 no, 1 unsure, then there's no consensus so there's no possible canonical marking yet.

If I understand you correctly, the canonical marking could indicate Unsure for something like Parent Present. Does that mean both Yes and No will then be considered Valid markings for Parent Present? Or would they both be consider Invalid markings?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1675 - Posted: 22 Sep 2013, 19:23:39 UTC - in response to Message 1674.

2. from these totals, i calculate what we would expect to have as the canonical result (if it is possible to have a canonical result). Basically, for each observaton marking (yes, no, unsure) the canonical one would be the the total which is >= 2, and also is greater than the other two, so, if there are 3 yes, 2 no, 1 unsure, then the expected canonical marking is yes. If there are 2 yes, 2 no, 1 unsure, then there's no consensus so there's no possible canonical marking yet.

If I understand you correctly, the canonical marking could indicate Unsure for something like Parent Present. Does that mean both Yes and No will then be considered Valid markings for Parent Present? Or would they both be consider Invalid markings?


If the canonical marking has an unsure in it, yes and no will count as half correct.

If the canonical marking has a yes/no, then unsures will count as half correct.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 1676 - Posted: 22 Sep 2013, 19:25:20 UTC - in response to Message 1675.

If the canonical marking has an unsure in it, yes and no will count as half correct.

If the canonical marking has a yes/no, then unsures will count as half correct.

Okay. When I go through my Watched Videos, will it be under my Valid videos or my Invalid videos?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1677 - Posted: 22 Sep 2013, 20:06:31 UTC - in response to Message 1676.

If the canonical marking has an unsure in it, yes and no will count as half correct.

If the canonical marking has a yes/no, then unsures will count as half correct.

Okay. When I go through my Watched Videos, will it be under my Valid videos or my Invalid videos?


Valid will be all the videos that matched the canonical 100%. Invalid will be everything else.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 1679 - Posted: 22 Sep 2013, 20:22:56 UTC - in response to Message 1678.

? What your saying then is you can be absolutely correct with your Marking's for the Video but if they don't match the Canonicals Marking's your going to be Tagged Invalid ? Or at the very least you'll lose Credits & or Accuracy if you don't Match the Canonicals Marking even though you may be the right & the Canonical wrong ...

How can the Canonical be wrong? If the Canonical really is wrong, that would be reason to Report the video.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1681 - Posted: 22 Sep 2013, 20:31:51 UTC - in response to Message 1678.

Valid will be all the videos that matched the canonical 100%. Invalid will be everything else.


? What your saying then is you can be absolutely correct with your Marking's for the Video but if they don't match the Canonicals Marking's your going to be Tagged Invalid ? Or at the very least you'll lose Credits & or Accuracy if you don't Match the Canonicals Marking even though you may be the right & the Canonical wrong ...


No, not at all.

Say the canonical is:

Bird Leave - Yes
Bird Return - Yes
Bird Presence - Yes
Bird Absence - Yes
Predator Presence - No
Chick Presence - No
Nest Defense - No
Nest Success - No

And your mark is:

Bird Leave - No
Bird Return - No
Bird Presence - Yes
Bird Absence - No
Predator Presence - No
Chick Presence - No
Nest Defense - No
Nest Success - No

Then in that case you'll get 5/8 accuracy, and 5/8 the credit of the canonical video (because you matched 5/8ths of the markings). Currently, you'd get 0 accuracy for that video.

If your marks were:

Bird Leave - unsure
Bird Return - unsure
Bird Presence - unsure
Bird Absence - unsure
Predator Presence - No
Chick Presence - No
Nest Defense - No
Nest Success - No

Then in that case you'd get 4/8 accuracy + 2/8 accuracy (as the unsures count for half), so 6/8 accuracy; and you'd also get 6/8ths of the credit as the canonical observations for the same reason.

Does that make sense? Basically you get partial credit and partial accuracy for everything that matches the canonical result, and unsures count for half (instead of nothing).

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1682 - Posted: 22 Sep 2013, 20:32:46 UTC - in response to Message 1680.

? What your saying then is you can be absolutely correct with your Marking's for the Video but if they don't match the Canonicals Marking's your going to be Tagged Invalid ? Or at the very least you'll lose Credits & or Accuracy if you don't Match the Canonicals Marking even though you may be the right & the Canonical wrong ...

How can the Canonical be wrong? If the Canonical really is wrong, that would be reason to Report the video.


The Canonical can be wrong if the person it's being Canonical against is wrong too, I've seen that on some Video's, I have a few reported myself where they were both wrong & I was right.


If the canonical is wrong, because most observers had those wrong observations -- that's exactly the point behind the report button; so we have another layer of robustness for misclassifications.

On another note, the validator should also now be re-validating things when they have been reported and an expert has reviewed the video.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1685 - Posted: 22 Sep 2013, 20:48:17 UTC - in response to Message 1684.

Yeah, the validator is at 10k of 70k observations or so. Probably be done in an hour.

GLeeM
Send message
Joined: 1 Jul 13
Posts: 118
Combined Credit: 47,541,025
DNA@Home: 28,994
SubsetSum@Home: 231,079
Wildlife@Home: 47,280,952
Wildlife@Home Watched: 3,888,714s
Wildlife@Home Events: 628
Climate Tweets: 0
Images Observed: 0

          
Message 1697 - Posted: 23 Sep 2013, 13:54:45 UTC

Basically you get partial credit and partial accuracy for everything that matches the canonical result, and unsures count for half (instead of nothing).

When two people say Unsure they will Validate - what happens to the video, does it go to experts to decide? And if the expert decides there is a bird present or absent, does the Unsure then become Invalid?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1701 - Posted: 23 Sep 2013, 17:07:33 UTC - in response to Message 1697.

Basically you get partial credit and partial accuracy for everything that matches the canonical result, and unsures count for half (instead of nothing).

When two people say Unsure they will Validate - what happens to the video, does it go to experts to decide? And if the expert decides there is a bird present or absent, does the Unsure then become Invalid?


If the canonical marks are unsure, that means that chances are the video was too dark or had some issue. If that's not correct (and your yes or no doesn't match the unsure) you can use the report button to get an expert to look at it.

If the expert marks it with a yes or no, then those unsures will become not correct, as the expert observation will count as the new canonical one.


Post to thread

Message boards : News : updating observations with new validator, and team stats