Advanced search

Message boards : News : testing new validator

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1346 - Posted: 27 Aug 2013, 0:24:43 UTC

Hi Everyone,

I think I've tweaked the validator to get things working the way they should be now. It's going through all the observations and remarking everything.


I haven't touched your previous accuracy and credit --> the validator is updating new calculations for these in different tables so in case we don't like it I can always go back and nothing is lost. You can check your new credit calculating (note they're being updated as we speak so it'll take awhile for the validator to get through everything) here:

http://volunteer.cs.und.edu/wildlife/top_bossa_users.php?sort_by=bossa_total_credit&offset=0

Also, you can check to see how much credit was awarded to individual observations on your watched videos page. This will let you know how partial credit is being awarded, and will help me find any inconsistencies with the new approach.

So in a nutshell, the new validation scheme does:

1. gives the most priority (especially to canonical results) for observations with no unsures.
2. gives less priority to observations marked 'unsure'.
3. gives even less priority to 'video issue' markings.

Partial credit will go to observations marked invalid, based on how many of the markings matched the canonical result. If the canonical result was a video issue, unsures will count as a match to it for partial credit.

Please let me know if anything isn't working right. Once I have this debugged I'll update the validator to update things for reviewed videos.

Senilix
Send message
Joined: 2 Feb 12
Posts: 96
Combined Credit: 8,073,607
DNA@Home: 106,112
SubsetSum@Home: 114,729
Wildlife@Home: 7,852,766
Wildlife@Home Watched: 7,118,967s
Wildlife@Home Events: 1,563
Climate Tweets: 5
Images Observed: 26

            
Message 1399 - Posted: 27 Aug 2013, 20:00:18 UTC - in response to Message 1346.

The validator seems to judge a bit asymmetrically.

I've seen a couple of cases where there have been 3 observers with
#1 marking YES
#2 marking UNSURE
#3 marking NO
or
#1 marking NO
#2 marking UNSURE
#3 marking YES
The result of observer #1 always got validated (CANONICAL). Why? IMO the validator should send the video to a 4th observer.

Here are some examples:
http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N1031_24LED_cam/7-31-12_N1031/CH00_20120730_061921MN_CHILD2.mp4
http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N1025/7-16-2012_N1025/CH00_20120712_102116MN_CHILD64.mp4
http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N334/8-03-2012_N334/CH00_20120729_183558MN_CHILD8.mp4

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1404 - Posted: 27 Aug 2013, 21:22:43 UTC - in response to Message 1399.

The validator seems to judge a bit asymmetrically.

I've seen a couple of cases where there have been 3 observers with
#1 marking YES
#2 marking UNSURE
#3 marking NO
or
#1 marking NO
#2 marking UNSURE
#3 marking YES
The result of observer #1 always got validated (CANONICAL). Why? IMO the validator should send the video to a 4th observer.

http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N1031_24LED_cam/7-31-12_N1031/CH00_20120730_061921MN_CHILD2.mp4
http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N1025/7-16-2012_N1025/CH00_20120712_102116MN_CHILD64.mp4
http://wildlife.und.edu//share/wildlife/streaming_2/missouri_river_project/2012/piping_plover/N334/8-03-2012_N334/CH00_20120729_183558MN_CHILD8.mp4



You're definitely right. I'll get that fixed.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 1519 - Posted: 4 Sep 2013, 17:59:10 UTC - in response to Message 1404.

For the record, I am still getting videos where I am Unvalidated while someone else is Canonical.

The specific case where I see this happening is when someone else has already watched the video, while I and another person both watch at the same time but the other person submits before I do, making me the third person. In each instance, we all three have had the same marking.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1579 - Posted: 10 Sep 2013, 3:14:49 UTC - in response to Message 1525.

This is messed up, I was the only one out of 3 to make the decision the Parent was Present & I get Reduced Points for it while the 2 that were Unsure get the full 180 Points, what a messed up Validation system ...

Dario666 CANONICAL x x x ? ? x x x x x 180
TBirdTheYuri VALID x x x ? ? x x x x x 180
DoA VALID x x x ✓ x x x x x x 135




That is really strange. I still haven't had a spare block of a couple hours to update the validator (damn the beginning of the semester rush!) but I should hopefully get everything working correctly this weekend. If you see any other examples like this please let me know and post them in here. It will help the debugging process.

thanks!
--Travis

GLeeM
Send message
Joined: 1 Jul 13
Posts: 118
Combined Credit: 47,541,025
DNA@Home: 28,994
SubsetSum@Home: 231,079
Wildlife@Home: 47,280,952
Wildlife@Home Watched: 3,888,714s
Wildlife@Home Events: 628
Climate Tweets: 0
Images Observed: 0

          
Message 1586 - Posted: 10 Sep 2013, 13:04:14 UTC - in response to Message 1579.



Lemon CANONICAL x x x ✓ x x x x x x 180

GLeeM INVALID x ? ? ? ? ? ? ? ? ✓ 0

Senilix VALID x x x ? ? x x x x x 135

This one should not have validated yet.
And it shows Video Problem as all Unsures like you wanted, but Invalid?



Lemon INVALID x x ✓ ✓ ✓ x x x x x 135

STE\/E CANONICAL x x x x ✓ x x x x x 180

GLeeM INVALID x x x ✓ x x x x x x 135

Dario666 VALID x x x ? ? x x x x x 135

Here is another that should not have validated yet and the only correctly marked is Lemon.

Profile branjo
Send message
Joined: 7 Sep 13
Posts: 12
Combined Credit: 0
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 0
Wildlife@Home Watched: 390,391s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

  
Message 1651 - Posted: 18 Sep 2013, 21:37:49 UTC - in response to Message 1650.

I don't know why, but my video stopped around 2:55 - 2:56, so I saw parent just entering video, not the last 4 - 5 seconds when it returned to the nest.
____________
For Science and Shrubberies! Since January 13 2000.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 1668 - Posted: 21 Sep 2013, 19:27:39 UTC - in response to Message 1652.
Last modified: 21 Sep 2013, 19:28:02 UTC

I've been working on updating the validator, and I'm testing the new version right now. Pretty close to having something solid. Should be in place by the end of the day.


Post to thread

Message boards : News : testing new validator