Advanced search

Message boards : News : [wildlife] updated workunit deadlines

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6924 - Posted: 19 Apr 2017, 21:27:45 UTC

So I'm now dynamically generating workunit deadlines based on how much work they'll require (so deadlines are based on credit). Right now I'm setting the deadline to 1 day per 500 credit awarded -- let me know if this seems correct of if I should extend it further. I've also stopped the one search which should have been generating the really large workunits, so I imagine newly generated workunits should not have the excessively long multi-day runtimes you've been seeing.

Dirk Broer
Send message
Joined: 7 Aug 14
Posts: 48
Combined Credit: 4,606,510
DNA@Home: 285,955
SubsetSum@Home: 1,566,395
Wildlife@Home: 2,754,160
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 230
Images Observed: 0

        
Message 6925 - Posted: 20 Apr 2017, 0:52:23 UTC - in response to Message 6924.

Can't get any new WU at all
____________

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6926 - Posted: 20 Apr 2017, 4:49:36 UTC - in response to Message 6925.

Can't get any new WU at all


Still? It looks like work is available...

Profile Conan
Avatar
Send message
Joined: 13 Apr 12
Posts: 141
Combined Credit: 28,057,247
DNA@Home: 399,792
SubsetSum@Home: 1,448,876
Wildlife@Home: 26,208,580
Wildlife@Home Watched: 70,910s
Wildlife@Home Events: 0
Climate Tweets: 393
Images Observed: 0

          
Message 6928 - Posted: 20 Apr 2017, 12:41:14 UTC - in response to Message 6924.
Last modified: 20 Apr 2017, 12:44:51 UTC

So I'm now dynamically generating workunit deadlines based on how much work they'll require (so deadlines are based on credit). Right now I'm setting the deadline to 1 day per 500 credit awarded -- let me know if this seems correct of if I should extend it further. I've also stopped the one search which should have been generating the really large workunits, so I imagine newly generated workunits should not have the excessively long multi-day runtimes you've been seeing.


This may work, but on a few sent out to me they started with run times estimated at a bit over 1 day each, the due dates have turned out to be too short.
As they progressed and 1 day had passed all the estimated times jumped by days.

So the 3 sent with 3 day deadlines (sent the 17/4/17 due 20/4/17 or 21/4/17) they have all been running for 2 days and 8 hours now, 1 is due to finish in 9 hours so should make deadline, but the other 2 have had 6 days and 8 days added to there estimated run times and so have no hope of making the 21st deadline, they need another week.

One other sent to me with a longer deadline of 25/4/17 after 1 day running estimated run time jumped to another 10 days, and after 2 days 8 hours running still needs another 10days 15 hours to finish, again will not meet deadline.

(Oh and the one I mentioned in a different thread about long running work units, sent to me on the 9th has now clocked up 9 days 14 hours running with 2 days 4 hours to go, not expecting much credit for these long runners as others do them in much shorter times, so good for clocking up hours but not for clocking up points.)

Conan
____________

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6929 - Posted: 20 Apr 2017, 14:30:28 UTC - in response to Message 6928.

As they progressed and 1 day had passed all the estimated times jumped by days.


Trying to figure out what exactly is going on here as I can't replicate it on my end... so do they start progressing somewhat as expected, then all of the sudden the estimated time increases significantly?

Does the progress drop back down? I mean, will it get up to say 10% then drop down to 0.5% or something like that?

Dirk Broer
Send message
Joined: 7 Aug 14
Posts: 48
Combined Credit: 4,606,510
DNA@Home: 285,955
SubsetSum@Home: 1,566,395
Wildlife@Home: 2,754,160
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 230
Images Observed: 0

        
Message 6930 - Posted: 20 Apr 2017, 19:38:56 UTC - in response to Message 6926.

Can't get any new WU at all


Still? It looks like work is available...


For some reason only DNA was selected in my preferences -but I checked the box 'accept other applications when there is no work for your chosen applications'
____________

mmonnin
Send message
Joined: 31 May 16
Posts: 25
Combined Credit: 19,831,095
DNA@Home: 0
SubsetSum@Home: 1,023,200
Wildlife@Home: 18,807,895
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 54
Images Observed: 0

      
Message 6931 - Posted: 20 Apr 2017, 22:01:09 UTC
Last modified: 20 Apr 2017, 22:01:49 UTC

I think the deadlines should be several times the expected run times.

This WU is on my dedicated 2670v1 2P machine that has a 0.25 day queue so it shouldn't take long from download to start.

exact_genome_1492132398_5_11058_3

Download time: 4/17/2017 at 11:14:12 PM

It has been running for: 02d,12:10:58
% Complete: 93.38
Estimated Run Time Left: 03:43:18
Deadline: 4/20/2017 10:03:31 PM

Local time is 5:53PM so with almost 4 hours left it might just barely make the deadline if the ETA is spot on.

BM has it in high priority mode since its taking so long.

There are 4 units on that computer with a higher % complete (94-98% complete) with 5,6,7,7(WU size) and run times of 20h, 9.25h, 14.3h and 12.75h, respectively. That middle number indicating the WU size doesn't indicate the run time at all.

Profile Conan
Avatar
Send message
Joined: 13 Apr 12
Posts: 141
Combined Credit: 28,057,247
DNA@Home: 399,792
SubsetSum@Home: 1,448,876
Wildlife@Home: 26,208,580
Wildlife@Home Watched: 70,910s
Wildlife@Home Events: 0
Climate Tweets: 393
Images Observed: 0

          
Message 6932 - Posted: 20 Apr 2017, 22:15:31 UTC - in response to Message 6929.
Last modified: 20 Apr 2017, 22:16:33 UTC

As they progressed and 1 day had passed all the estimated times jumped by days.


Trying to figure out what exactly is going on here as I can't replicate it on my end... so do they start progressing somewhat as expected, then all of the sudden the estimated time increases significantly?

Does the progress drop back down? I mean, will it get up to say 10% then drop down to 0.5% or something like that?


No the percentage done keeps going as it should, just that the initial estimate has been way off.
One has already gone past it's 3 day deadline (even though it has only been running 2 days 17 hours) and still has 5 days 23 hours to go, but I am not sure if I will get credit for it as two other results are already back.
The other 3 day one will time out in a few hours, it still has 8 days 6 hours to go.

I have noticed that they run for a long time with no change to the percentage done and then the percentage done goes up and time to go goes down by a good bit, then it sits there again, however the time remaining keeps counting back up in 4 second jumps in between the percentage moves.

Probably why it takes so long to come down (an hour has been added whilst I was writing this, so the drops are large but so are the gains, meaning small drops overall).

Conan
____________

mmonnin
Send message
Joined: 31 May 16
Posts: 25
Combined Credit: 19,831,095
DNA@Home: 0
SubsetSum@Home: 1,023,200
Wildlife@Home: 18,807,895
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 54
Images Observed: 0

      
Message 6935 - Posted: 21 Apr 2017, 2:51:45 UTC - in response to Message 6931.

I think the deadlines should be several times the expected run times.

This WU is on my dedicated 2670v1 2P machine that has a 0.25 day queue so it shouldn't take long from download to start.

exact_genome_1492132398_5_11058_3

Download time: 4/17/2017 at 11:14:12 PM

It has been running for: 02d,12:10:58
% Complete: 93.38
Estimated Run Time Left: 03:43:18
Deadline: 4/20/2017 10:03:31 PM

Local time is 5:53PM so with almost 4 hours left it might just barely make the deadline if the ETA is spot on.

BM has it in high priority mode since its taking so long.

There are 4 units on that computer with a higher % complete (94-98% complete) with 5,6,7,7(WU size) and run times of 20h, 9.25h, 14.3h and 12.75h, respectively. That middle number indicating the WU size doesn't indicate the run time at all.


2.5 days down the crapper.
http://csgrid.org/csg/workunit.php?wuid=1017415

mmonnin
Send message
Joined: 31 May 16
Posts: 25
Combined Credit: 19,831,095
DNA@Home: 0
SubsetSum@Home: 1,023,200
Wildlife@Home: 18,807,895
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 54
Images Observed: 0

      
Message 6936 - Posted: 21 Apr 2017, 2:59:36 UTC - in response to Message 6931.
Last modified: 21 Apr 2017, 3:00:03 UTC

Duplicate.

Profile Conan
Avatar
Send message
Joined: 13 Apr 12
Posts: 141
Combined Credit: 28,057,247
DNA@Home: 399,792
SubsetSum@Home: 1,448,876
Wildlife@Home: 26,208,580
Wildlife@Home Watched: 70,910s
Wildlife@Home Events: 0
Climate Tweets: 393
Images Observed: 0

          
Message 6938 - Posted: 21 Apr 2017, 9:26:19 UTC - in response to Message 6932.

As they progressed and 1 day had passed all the estimated times jumped by days.


Trying to figure out what exactly is going on here as I can't replicate it on my end... so do they start progressing somewhat as expected, then all of the sudden the estimated time increases significantly?

Does the progress drop back down? I mean, will it get up to say 10% then drop down to 0.5% or something like that?


No the percentage done keeps going as it should, just that the initial estimate has been way off.
One has already gone past it's 3 day deadline (even though it has only been running 2 days 17 hours) and still has 5 days 23 hours to go, but I am not sure if I will get credit for it as two other results are already back.
The other 3 day one will time out in a few hours, it still has 8 days 6 hours to go.

I have noticed that they run for a long time with no change to the percentage done and then the percentage done goes up and time to go goes down by a good bit, then it sits there again, however the time remaining keeps counting back up in 4 second jumps in between the percentage moves.

Probably why it takes so long to come down (an hour has been added whilst I was writing this, so the drops are large but so are the gains, meaning small drops overall).

Conan


I have aborted the two I had due on the 21st as they still had between 5 and 8 days to go but both have had two valid results already returned so I am wasting my time with these ones.

Conan
____________

Dr Who Fan
Avatar
Send message
Joined: 25 May 15
Posts: 17
Combined Credit: 1,341,448
DNA@Home: 14,404
SubsetSum@Home: 70,037
Wildlife@Home: 1,257,008
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

      
Message 6939 - Posted: 21 Apr 2017, 17:11:15 UTC

Another MONSTER TASK running WAY OVER ORIGINAL ESTIMATE. Original estimate was around 8.5 hours to complete. Actually took 86.310 HOURS. That is slightly more than 10 times the original target.

Completed, too late to validate
Work was DONE & RETURNED. NOT MY FAULT ORIGINAL ESTIMATES WERE BAD, I SHOULD RECEIVE CREDIT.

exact_genome_1492132398_5_12056 http://csgrid.org/csg/workunit.php?wuid=1020459

CPU TIME:310,716.30 // CREDIT: 0.00 / APP NAME: EXACT MNIST Convolutional Neural Network Trainer v0.20
____________

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6940 - Posted: 21 Apr 2017, 19:30:27 UTC - in response to Message 6939.

Another MONSTER TASK running WAY OVER ORIGINAL ESTIMATE. Original estimate was around 8.5 hours to complete. Actually took 86.310 HOURS. That is slightly more than 10 times the original target.

Completed, too late to validate
Work was DONE & RETURNED. NOT MY FAULT ORIGINAL ESTIMATES WERE BAD, I SHOULD RECEIVE CREDIT.

exact_genome_1492132398_5_12056 http://csgrid.org/csg/workunit.php?wuid=1020459

CPU TIME:310,716.30 // CREDIT: 0.00 / APP NAME: EXACT MNIST Convolutional Neural Network Trainer v0.20


Working on something to award credit for workunits that have come in too late to validate.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6944 - Posted: 22 Apr 2017, 1:03:08 UTC - in response to Message 6940.

So I've added a setting that will keep WUs around for 4 days after the deadline so that they can still be validated if reported late.

Hoping to find some time this weekend to award credit for the ones that already came in and were too late to validate and didn't receive credit.

Profile Conan
Avatar
Send message
Joined: 13 Apr 12
Posts: 141
Combined Credit: 28,057,247
DNA@Home: 399,792
SubsetSum@Home: 1,448,876
Wildlife@Home: 26,208,580
Wildlife@Home Watched: 70,910s
Wildlife@Home Events: 0
Climate Tweets: 393
Images Observed: 0

          
Message 6947 - Posted: 23 Apr 2017, 12:06:36 UTC
Last modified: 23 Apr 2017, 12:07:55 UTC

Well I have finally finished that work unit sent to me on the 9th This WU
It has taken over 1,000,000 seconds (1,018,303 seconds) or 282.86 Hours
Awarded 2,816.32 credits
For just 9.96 credits per hour.

The wingman only took 173,048 seconds (just 48.07 hours) for 58.59 cr/h

This WU took 11.75 days to run but was still validated.

I think this is a new record for run time.

I have another long runner at 5 days, 30.463% done and 11 days 9 hours still to go, if it goes this long it will be my longest running WU by a large margin.
So I hope I can still get credit for it.

Conan
____________

Zir Madmax
Send message
Joined: 17 Feb 17
Posts: 11
Combined Credit: 467,831
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 467,831
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 0

  
Message 6948 - Posted: 24 Apr 2017, 5:06:01 UTC

exact_genome _1492919102_6_5678 go switch after 13,40 houers to be over 4 day.. and still rising 1houer every 5 minuts. And the "progess" drops from 1,800 to 0,700% /houer and still faling. Time limit 27/4 . Befor the time jump it say 18 houers.
____________


Post to thread

Message boards : News : [wildlife] updated workunit deadlines