Advanced search

Message boards : News : [climate] validating tweets - part 4

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1812
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 755

              
Message 5942 - Posted: 23 Oct 2015, 6:09:43 UTC
Last modified: 23 Oct 2015, 6:10:28 UTC

So getting the validation working is turning out to be a fun little problem. I've come up with quality scores for classification matches:

1. full match: attitude is the same, all categories are the same
2. close match: attitudes can differ by one, more selected categories match than mismatch
3. far match: attitudes can differ by one, at least 1 selected category matches

Given these different kinds of matches, classifications with full matches will be awarded 1 point, classifications with close matches will be awarded .75 points, and classifications with far matches will be awarded .5 points. Others will be awarded 0.

Thoughts? Alternately, any of the 3 matches could just be 1 point, and others invalid. The classifications are pretty subjective, so I don't want to have the validation be too strict.

At any rate, the code for the validator is up and ready to go, I just haven't been committing anything to the database. Just wanted to gather some feedback before I started going live with it.

Profile JumpinJohnny
Avatar
Send message
Joined: 24 Sep 13
Posts: 237
Combined Credit: 10,275,610
DNA@Home: 192,548
SubsetSum@Home: 201,740
Wildlife@Home: 9,881,323
Wildlife@Home Watched: 55,997,833s
Wildlife@Home Events: 15,584
Climate Tweets: 327
Images Observed: 351

              
Message 5943 - Posted: 23 Oct 2015, 17:41:19 UTC - in response to Message 5942.

My 2 cents:

Keep it as "loose" as possible due to subjectivity of the classifications and the limitations of tweet characters,, (and apparent limitations of tweeters).
In order to get some decent data from this project, I would sugest NOT allowing major choice changes in the "Review" pages and somewhat ignoring user arguements for validation... just let the results stand.
It's not rock science after all and some of the tweeters are not the brightest bulbs in the knife drawer.
____________

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1812
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 755

              
Message 5944 - Posted: 23 Oct 2015, 18:57:57 UTC - in response to Message 5943.

My 2 cents:

Keep it as "loose" as possible due to subjectivity of the classifications and the limitations of tweet characters,, (and apparent limitations of tweeters).
In order to get some decent data from this project, I would sugest NOT allowing major choice changes in the "Review" pages and somewhat ignoring user arguements for validation... just let the results stand.
It's not rock science after all and some of the tweeters are not the brightest bulbs in the knife drawer.


I don't think I'm going to allow choice changes here as it's so quick and easy to classify a tweet. Definitely want to keep it as loose as possible - I want as many people getting credit for what their opinions are (even if they aren't the exact same as everyone else's).

Profile Skivelitis2
Avatar
Send message
Joined: 16 May 15
Posts: 60
Combined Credit: 11,066,329
DNA@Home: 19,068
SubsetSum@Home: 575,552
Wildlife@Home: 10,471,708
Wildlife@Home Watched: 9,158s
Wildlife@Home Events: 4
Climate Tweets: 387
Images Observed: 52

            
Message 5947 - Posted: 24 Oct 2015, 12:43:20 UTC - in response to Message 5944.
Last modified: 24 Oct 2015, 12:43:57 UTC

My 2 cents:

Keep it as "loose" as possible due to subjectivity of the classifications and the limitations of tweet characters,, (and apparent limitations of tweeters).
In order to get some decent data from this project, I would sugest NOT allowing major choice changes in the "Review" pages and somewhat ignoring user arguements for validation... just let the results stand.
It's not rock science after all and some of the tweeters are not the brightest bulbs in the knife drawer.


I don't think I'm going to allow choice changes here as it's so quick and easy to classify a tweet. Definitely want to keep it as loose as possible - I want as many people getting credit for what their opinions are (even if they aren't the exact same as everyone else's).

All sounds good. In total agreement with letting results stand unless a major problem becomes systemic. It would be more than a full time job if someone had to respond to every user complaint for every invalid tweet.


Post to thread

Message boards : News : [climate] validating tweets - part 4