Advanced search

Message boards : News : validator working?

1 · 2 · Next
Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 970 - Posted: 14 Aug 2013, 22:15:54 UTC

Hi Everyone,

I think I have the validator working correctly now. If you could check your watched videos and see if there's anything weird going on I'd appreciate it. From my own it looks like everything checks out correctly.

If everything is looking right I'll have the validator go through and re-award credit with the new validations.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 971 - Posted: 14 Aug 2013, 22:22:45 UTC - in response to Message 970.

What is considered validated? Three viewers in agreement? I see valids with just two in agreement. But I'm pulling up videos to watch that already have two in agreement.

What should we be looking for?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 972 - Posted: 14 Aug 2013, 22:41:44 UTC - in response to Message 971.

What is considered validated? Three viewers in agreement? I see valids with just two in agreement. But I'm pulling up videos to watch that already have two in agreement.

What should we be looking for?


Right now it's validating with two in agreement (I should up that to three though I think). Things aren't going through the validator until there are 3 observations however, so that would account for the discrepancy.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 973 - Posted: 14 Aug 2013, 22:44:03 UTC - in response to Message 972.

Revalidating everything right now requiring a minimum of 3 observations in agreement. Things won't get sent to the validator until there's 3 observations either.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 974 - Posted: 14 Aug 2013, 23:00:13 UTC - in response to Message 973.

How are you handling validation when one or more fields are marked Unsure?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 975 - Posted: 14 Aug 2013, 23:13:06 UTC - in response to Message 974.
Last modified: 14 Aug 2013, 23:13:30 UTC

How are you handling validation when one or more fields are marked Unsure?


Observations with unsure will mark valid against other observations (so long as they agree with all non-unsure fields); however they won't count as a canonical observation.

So if 3 observations match, but have unsures -- they won't validate the video or create a canonical observation.

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 976 - Posted: 14 Aug 2013, 23:16:45 UTC - in response to Message 975.

Do regular observations validate against Too Dark or Corrupt? Or does the same principle apply regarding how Unsures are handled?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 977 - Posted: 15 Aug 2013, 1:05:09 UTC - in response to Message 976.

Do regular observations validate against Too Dark or Corrupt? Or does the same principle apply regarding how Unsures are handled?


Regular observations won't validate against too dark or corrupt.

Profile DoctorNow
Avatar
Send message
Joined: 31 Jan 12
Posts: 328
Combined Credit: 2,333,912
DNA@Home: 94,617
SubsetSum@Home: 162,940
Wildlife@Home: 2,076,355
Wildlife@Home Watched: 1,977,485s
Wildlife@Home Events: 589
Climate Tweets: 128
Images Observed: 2,678

              
Message 978 - Posted: 15 Aug 2013, 4:44:45 UTC - in response to Message 975.
Last modified: 15 Aug 2013, 4:47:13 UTC

How are you handling validation when one or more fields are marked Unsure?

So if 3 observations match, but have unsures -- they won't validate the video or create a canonical observation.

I already wondered what would happen when there are only "unsure"s.
I think that could be a problem. There are videos where you can't really decide what's going on (for example if a bug sits on the lens and covers the entire nest - or, lately, in one of the grouse nests leafs covering the nest). If it happens, that three people mark such cases the same way with "unsure" then they won't get credit for it? A bit unfair...
____________
Wildlife & SubsetSum Badges-creator ;-)
Mod/Guru @ the german team BOINC@Heidelberg
My BOINC-Stats / My Badges

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 980 - Posted: 15 Aug 2013, 5:53:23 UTC - in response to Message 978.

How are you handling validation when one or more fields are marked Unsure?

So if 3 observations match, but have unsures -- they won't validate the video or create a canonical observation.

I already wondered what would happen when there are only "unsure"s.
I think that could be a problem. There are videos where you can't really decide what's going on (for example if a bug sits on the lens and covers the entire nest - or, lately, in one of the grouse nests leafs covering the nest). If it happens, that three people mark such cases the same way with "unsure" then they won't get credit for it? A bit unfair...


Observations with unsures get credit if their non-unsures match the canonical result.

So, if an observation has 2/8 marked unsure, and it validates, then it will get 6/8ths the credit. So they get credit, just not as much as if they were completely correct. I think this is pretty fair.

If a video doesn't validate because of unsures or none of the observations agreeing, then it will be automatically reported for Susan and her students to watch to determine what's right -- and then credit will be awarded based on that.

So basically, if you're unsure about something and mark it unsure -- it won't hurt you any. You'll still get credit for your non-unsure markings (assuming they're right).

Profile DoctorNow
Avatar
Send message
Joined: 31 Jan 12
Posts: 328
Combined Credit: 2,333,912
DNA@Home: 94,617
SubsetSum@Home: 162,940
Wildlife@Home: 2,076,355
Wildlife@Home Watched: 1,977,485s
Wildlife@Home Events: 589
Climate Tweets: 128
Images Observed: 2,678

              
Message 981 - Posted: 15 Aug 2013, 6:58:16 UTC - in response to Message 980.

If a video doesn't validate because of unsures or none of the observations agreeing, then it will be automatically reported for Susan and her students to watch to determine what's right -- and then credit will be awarded based on that.

Ah, I see. Forgot about that... ;-)
____________
Wildlife & SubsetSum Badges-creator ;-)
Mod/Guru @ the german team BOINC@Heidelberg
My BOINC-Stats / My Badges

Profile STE\/E
Avatar
Send message
Joined: 5 Apr 13
Posts: 416
Combined Credit: 29,783,819
DNA@Home: 2,634,206
SubsetSum@Home: 735,231
Wildlife@Home: 26,414,382
Wildlife@Home Watched: 53,380,530s
Wildlife@Home Events: 9,349
Climate Tweets: 0
Images Observed: 0

          
Message 982 - Posted: 15 Aug 2013, 12:49:57 UTC - in response to Message 970.

Hi Everyone,

I think I have the validator working correctly now. If you could check your watched videos and see if there's anything weird going on I'd appreciate it. From my own it looks like everything checks out correctly.

If everything is looking right I'll have the validator go through and re-award credit with the new validations.


I don't see where much of anything is being Validated whether it's a 2 or 3 Quorum. I've watched a Ton of Videos over the last week or more & my BOSSA Credit has barely moved at all ...

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 983 - Posted: 15 Aug 2013, 13:38:25 UTC - in response to Message 982.

I don't see where much of anything is being Validated whether it's a 2 or 3 Quorum. I've watched a Ton of Videos over the last week or more & my BOSSA Credit has barely moved at all ...

Now that it takes 3 viewers to agree on a video for validation, don't expect your bossa credits to climb much at all. I doubt we'll see more than ten hours of validated video a day, and even if you've watched over twenty hours, your credits will only increase by 36000. The good news is that your credits will continue to increase for months after you stop watching.

Also, if Travis adjusts credits based on the current validation, expect your credits to take a huge nosedive. From what I've seen, my credits will drop to about 1/3 of what they currently are.

Profile DoctorNow
Avatar
Send message
Joined: 31 Jan 12
Posts: 328
Combined Credit: 2,333,912
DNA@Home: 94,617
SubsetSum@Home: 162,940
Wildlife@Home: 2,076,355
Wildlife@Home Watched: 1,977,485s
Wildlife@Home Events: 589
Climate Tweets: 128
Images Observed: 2,678

              
Message 984 - Posted: 15 Aug 2013, 14:27:48 UTC - in response to Message 983.
Last modified: 15 Aug 2013, 14:29:20 UTC

I don't see where much of anything is being Validated whether it's a 2 or 3 Quorum. I've watched a Ton of Videos over the last week or more & my BOSSA Credit has barely moved at all ...

Now that it takes 3 viewers to agree on a video for validation, don't expect your bossa credits to climb much at all.
...

Plus the validator isn't constantly turned on yet. Probably Travis did run it only a few times while testing.
I were the third wingman for a lot of videos in the last days - today also - but my credits haven't changed much also. ;-)

Btw: the server seems to be a bit slow here with the forum and loading the videos. Is there a little bottleneck somewhere or are you working on something, Travis?
____________
Wildlife & SubsetSum Badges-creator ;-)
Mod/Guru @ the german team BOINC@Heidelberg
My BOINC-Stats / My Badges

Profile STE\/E
Avatar
Send message
Joined: 5 Apr 13
Posts: 416
Combined Credit: 29,783,819
DNA@Home: 2,634,206
SubsetSum@Home: 735,231
Wildlife@Home: 26,414,382
Wildlife@Home Watched: 53,380,530s
Wildlife@Home Events: 9,349
Climate Tweets: 0
Images Observed: 0

          
Message 985 - Posted: 15 Aug 2013, 15:14:45 UTC - in response to Message 983.

Also, if Travis adjusts credits based on the current validation, expect your credits to take a huge nosedive. From what I've seen, my credits will drop to about 1/3 of what they currently are.


Why would that be ??, from what I've seen 99%+ of your Videos are accurate, mine would be less mostly because of Mental errors but I didn't/don't get the Credit in the first place because they shouldn't have Validated to start with ... IMO
____________

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 986 - Posted: 15 Aug 2013, 16:20:09 UTC - in response to Message 985.

Why would that be ??

As it now takes 3 markings in agreement for a video to be validated, the number of validated videos I have has dropped by 2/3.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 987 - Posted: 15 Aug 2013, 17:47:24 UTC - in response to Message 984.

I don't see where much of anything is being Validated whether it's a 2 or 3 Quorum. I've watched a Ton of Videos over the last week or more & my BOSSA Credit has barely moved at all ...

Now that it takes 3 viewers to agree on a video for validation, don't expect your bossa credits to climb much at all.
...

Plus the validator isn't constantly turned on yet. Probably Travis did run it only a few times while testing.
I were the third wingman for a lot of videos in the last days - today also - but my credits haven't changed much also. ;-)

Btw: the server seems to be a bit slow here with the forum and loading the videos. Is there a little bottleneck somewhere or are you working on something, Travis?


I've been running some things testing the validator (and resetting the validation to rerun the validator) which could be why things were running a bit slow.

I've turned the validator on now, because it seems like it's working correctly. If you guys notice anything weird with videos getting validated incorrectly, let me know so I can take a look at them.

Profile STE\/E
Avatar
Send message
Joined: 5 Apr 13
Posts: 416
Combined Credit: 29,783,819
DNA@Home: 2,634,206
SubsetSum@Home: 735,231
Wildlife@Home: 26,414,382
Wildlife@Home Watched: 53,380,530s
Wildlife@Home Events: 9,349
Climate Tweets: 0
Images Observed: 0

          
Message 989 - Posted: 15 Aug 2013, 21:19:44 UTC - in response to Message 986.

Why would that be ??

As it now takes 3 markings in agreement for a video to be validated, the number of validated videos I have has dropped by 2/3.


I haven't seen your BOSSA Credit drop nor mine either, both have gone up a little if fact. I don't think it's Travis's intention to have all the Videos already Validated by 2 People re-watched by another person to make it a 3 people Quorum, but then I could be wrong on that too ... Going forward yes it'll take 3 people to Validate ...
____________

Profile Lemon
Send message
Joined: 10 May 13
Posts: 229
Combined Credit: 476,659
DNA@Home: 190,781
SubsetSum@Home: 225,957
Wildlife@Home: 59,921
Wildlife@Home Watched: 11,190,214s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

        
Message 990 - Posted: 15 Aug 2013, 22:32:04 UTC - in response to Message 989.

I haven't seen your BOSSA Credit drop nor mine either, both have gone up a little if fact.

The credit hasn't dropped yet, and it isn't clear that Travis will reset the credits to match the current validations, but I don't see why he shouldn't. If he does, I expect my credit to be 1/3 of what it currently is.

I don't think it's Travis's intention to have all the Videos already Validated by 2 People re-watched by another person to make it a 3 people Quorum, but then I could be wrong on that too ... Going forward yes it'll take 3 people to Validate ...

I don't know about that. I've been watching hundreds of videos since yesterday that already have two identical responses.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 774

              
Message 991 - Posted: 15 Aug 2013, 22:50:21 UTC - in response to Message 990.

I haven't seen your BOSSA Credit drop nor mine either, both have gone up a little if fact.

The credit hasn't dropped yet, and it isn't clear that Travis will reset the credits to match the current validations, but I don't see why he shouldn't. If he does, I expect my credit to be 1/3 of what it currently is.

I don't think it's Travis's intention to have all the Videos already Validated by 2 People re-watched by another person to make it a 3 people Quorum, but then I could be wrong on that too ... Going forward yes it'll take 3 people to Validate ...

I don't know about that. I've been watching hundreds of videos since yesterday that already have two identical responses.


Right now it's set as requiring 3 for a quorum to validate.

I ran the numbers and there would be quite a bit of a credit drop if I redid everyones credit (it seems like credits were getting awarded multiple times before). I'm okay with leaving it the way it is, and going forward with the new validator; unless the majority of people want me to redo everyone's credit. The code's all in place to go either way, I just don't want to bug anyone by causing a large credit drop for everyone. Since the project is still really alpha I have no problems with you guys getting some bonus credits for putting up with all the bugs and design changes, etc. :)

1 · 2 · Next
Post to thread

Message boards : News : validator working?