Hey Travis. Got a couple of MNIST units that are running really long, well outside the normal time frames I get. (normally no more than ~30 hours)

exact_genome_1490477651_4_1417_0 -> currently at 46 hrs+ and 87% complete
and
exact_genome_1490477651_3_1411_1 -> 52 hrs and 64% complete (wingman complete at 68 hours).

Should be interesting to look at.
____________

exact_genome_1490477651_3_1411_1 -> 52 hrs and 64% complete (wingman complete at 68 hours).

Should be interesting to look at.

exact_genome_1490477651_3_1411_1 -> 52 hrs and 64% complete (wingman complete at 68 hours).

Should be interesting to look at.

Well .. this one finished and it is interesting, but not quite what I hoped. This 96 hour work unit yielded only 747.82 points ( I, of course, was expecting a veritable bonanza of over 10K points!). I know that points are not based on a straight line basis of time, but this is seriously out of whack no matter what the algorithm. It obviously was not doing the kind of work needed to accrue points.

So could you investigate. If there something hinky with my machine, then I need to fix it (though other units appear to be proceeding normally). Somewhere, this one went off the rails.

I also have seen these.
Two of them I aborted after 30 hours.
I let a few run only to get 700 credit for 400,000+ cpu sec.
I am aborting another.
I wish I could tell which of these are going to do this in advance of letting a core waste an entire day on 30% completion only to be aborted.
The slowness of the WU's does not seem to reflect OS type or CPU speed.
Some wingmen are doing them in comparable long times some are alot quicker.
------------------------------------------------------------------------------
Even the rest of the results are really just all over the place.
25,604.48 sec = 1,351.24 credits
68,537.76 = 1,258.08

36,625.87 = 2,257.77
32,292.02 = 880.93
The credits algorithm seems to have taken a turn to the dark side.

So this seems like something weird may be afoot. Any chance you could link me the work units you aborted? Or if you see these again let me know the work unit so I can take a deeper look and see what's going on? There may be some kind of bug going on that's making them run significantly longer than they should.

So this seems like something weird may be afoot. Any chance you could link me the work units you aborted? Or if you see these again let me know the work unit so I can take a deeper look and see what's going on? There may be some kind of bug going on that's making them run significantly longer than they should.

So this seems like something weird may be afoot. Any chance you could link me the work units you aborted? Or if you see these again let me know the work unit so I can take a deeper look and see what's going on? There may be some kind of bug going on that's making them run significantly longer than they should.

Here's one of many:

http://csgrid.org/csg/workunit.php?wuid=980866

My theory is that the apps are now being compiled with the wrong switches for general use.

This is really really odd. I'm looking into it. Both of those are running Windows 7, so I don't know why one would be significantly slower depending on how they were compiled.

What's even weirder, is that the one it's running slower on has more cache and memory, so if anything it should be running quite a bit faster.

....
The slowness of the WU's does not seem to reflect OS type or CPU speed.
Some wingmen are doing them in comparable long times some are alot quicker.
------------------------------------------------------------------------------
Even the rest of the results are really just all over the place.
25,604.48 sec = 1,351.24 credits
68,537.76 = 1,258.08

36,625.87 = 2,257.77
32,292.02 = 880.93
The credits algorithm seems to have taken a turn to the dark side.

....
The slowness of the WU's does not seem to reflect OS type or CPU speed.
Some wingmen are doing them in comparable long times some are alot quicker.
------------------------------------------------------------------------------
Even the rest of the results are really just all over the place.
25,604.48 sec = 1,351.24 credits
68,537.76 = 1,258.08

36,625.87 = 2,257.77
32,292.02 = 880.93
The credits algorithm seems to have taken a turn to the dark side.

So .... Not just a few of the extra long ones are running wierdly, Plenty of other examples of inconsistent times compared to credits,,, which also do NOT always match up with a CPU type or OS.

Just making sure -- in the first example, you have two different work units giving the same amount of credit but with significantly different runtimes. And in the second one you have two different workunits with similar runtimes giving significantly different credit. Are these from the same system?

If that's the case I need to update the credit calculation. Inconsistent runtime to credit is probably a problem with how I'm calculating credit as the convolutional neural networks get larger. This is an easier fix and not a problem with the application at any rate.

I'll make it a priority and try and get it resolved by the end of the week (unfortunately I'm stuck in some 8 hours of meetings tomorrow which will slow me down).

Are these from the same system?

If that's the case I need to update the credit calculation. Inconsistent runtime to credit is probably a problem with how I'm calculating credit as the convolutional neural networks get larger. This is an easier fix and not a problem with the application at any rate.

I'll make it a priority and try and get it resolved by the end of the week (unfortunately I'm stuck in some 8 hours of meetings tomorrow which will slow me down).

I've updated the my algorithm for calculating credit which I think should fix the credit inconsistencies. If you're still seeing this with newly generated workunits let me know.

Hello Travis,

I have now a Problem with a Monster-WU too, I crunch now 4,5 Days on this WU and I wait some Hours for the okay of my Wingman. Now I see I become only 1092 Credits, ist very crazy Travis.
If you want check my WU, so I can give you this Informations from me. :

2136329 979728 32305 2 Apr 2017, 6:12:10 UTC 6 Apr 2017, 9:41:13 UTC Fertig und Bestätigt 358,028.01 286,172.30 1,091.76 EXACT MNIST Convolutional Neural Network Trainer v0.20

Many Greetz SEARCHER
____________
Member of CHARITY TEAM