Nameexact_genome_1542531604_173_6239_3
Workunit4097019
Created21 Dec 2018, 10:32:50 UTC
Sent21 Dec 2018, 10:32:52 UTC
Report deadline23 Jan 2019, 10:31:38 UTC
Received22 Dec 2018, 4:50:00 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84153
Run time15 hours 41 min 2 sec
CPU time14 hours 18 min 45 sec
Validate stateValid
Credit8,249.79
Device peak FLOPS4.74 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size578.45 MB
Peak swap size582.09 MB
Peak disk usage8.53 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_173_6239.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_173_6239_3_r87248460_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542531604_173_6239.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.551607
read mu: 0.551607
read mu_delta: 0.97181
read initial_learning_rate: 0.00158498
read learning_rate: 0.00158498
read learning_rate_delta: 0.913003
read initial_weight_decay: 0.000318933
read weight_decay: 0.000318933
read weight_decay_delta: 0.913809
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.132621
read input_dropout_probability: 0.0016875
read hidden_dropout_probability: 0.0669987
read velocity_reset: 1806
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6239
read normal distribution: '1 0 0'
generator_str: '1446846939'
read generator: 1446846939
read generated_by_map:
1 disable_node 1
reading 97 nodes.
reading 402 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1199168792
post shuffle 1: 1300481511
epoch time: 2045.79553222656s, input_fired_time: 22.3053131103516, output_fired_time: 9.93956756591797, propagate_forward_time: 732.051635742188, propagate_backward_time: 1255.431640625, weight_update_time: 0.606128096580505, other_time: 26.0673828125
epoch time: 155.087295532227s, input_fired_time: 1.70854926109314, output_fired_time: 0, propagate_forward_time: 148.371398925781, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.00735473632812
validation[          , genome  6239] predictions:    5974/  10000 (59.74%), best:    5974/10000 (59.74%), error:     11640.46875, best error:     11640.46875 on epoch:     0, epoch:    0/25, mu: 0.5516073704, learning_rate: 0.0015849795, weight_decay: 0.0003189330

epoch time: 2093.4196777344s, input_fired_time: 22.6592311859, output_fired_time: 9.8518295288, propagate_forward_time: 746.0244140625, propagate_backward_time: 1291.5706787109, weight_update_time: 0.6141747832, other_time: 23.3135986328
epoch time: 147.9876556396s, input_fired_time: 1.5472067595, output_fired_time: 0.0000000000, propagate_forward_time: 141.7134552002, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7269897461
validation[          , genome  6239] predictions:    5942/  10000 (59.42%), best:    5974/10000 (59.74%), error:     11662.79590, best error:     11640.46875 on epoch:     0, epoch:    1/25, mu: 0.5639654994, learning_rate: 0.0014470906, weight_decay: 0.0002914439

epoch time: 2046.0660400391s, input_fired_time: 23.3047370911, output_fired_time: 14.0325202942, propagate_forward_time: 732.0809936523, propagate_backward_time: 1253.2631835938, weight_update_time: 0.6168311834, other_time: 23.3847656250
epoch time: 152.6782379150s, input_fired_time: 2.6696741581, output_fired_time: 0.0000000000, propagate_forward_time: 146.0132141113, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9953460693
validation[          , genome  6239] predictions:    5771/  10000 (57.71%), best:    5974/10000 (59.74%), error:     12672.13867, best error:     11640.46875 on epoch:     0, epoch:    2/25, mu: 0.5759752393, learning_rate: 0.0013211978, weight_decay: 0.0002663241

epoch time: 2043.3081054688s, input_fired_time: 23.2486972809, output_fired_time: 9.8137731552, propagate_forward_time: 732.5242919922, propagate_backward_time: 1255.7436523438, weight_update_time: 0.6017586589, other_time: 21.9777832031
epoch time: 153.5908966064s, input_fired_time: 1.6778931618, output_fired_time: 0.0000000000, propagate_forward_time: 147.0462188721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8667907715
validation[          , genome  6239] predictions:    6119/  10000 (61.19%), best:    6119/10000 (61.19%), error:     11208.86426, best error:     11208.86426 on epoch:     3, epoch:    3/25, mu: 0.5876464844, learning_rate: 0.0012062573, weight_decay: 0.0002433693

epoch time: 2035.3876953125s, input_fired_time: 26.3273963928, output_fired_time: 11.8272390366, propagate_forward_time: 727.5789794922, propagate_backward_time: 1245.7009277344, weight_update_time: 0.6059163213, other_time: 23.9531250000
epoch time: 152.6039886475s, input_fired_time: 1.6855512857, output_fired_time: 0.0000000000, propagate_forward_time: 146.9231414795, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9953002930
validation[          , genome  6239] predictions:    6022/  10000 (60.22%), best:    6119/10000 (61.19%), error:     11407.21094, best error:     11208.86426 on epoch:     3, epoch:    4/25, mu: 0.5989887118, learning_rate: 0.0011013163, weight_decay: 0.0002223931

epoch time: 2047.1395263672s, input_fired_time: 22.4644584656, output_fired_time: 10.8936023712, propagate_forward_time: 719.9423828125, propagate_backward_time: 1268.0386962891, weight_update_time: 0.6028176546, other_time: 25.8004150391
epoch time: 153.0562744141s, input_fired_time: 2.6988770962, output_fired_time: 0.0000000000, propagate_forward_time: 146.3678588867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9895324707
validation[          , genome  6239] predictions:    5955/  10000 (59.55%), best:    6119/10000 (61.19%), error:     11654.91113, best error:     11208.86426 on epoch:     3, epoch:    5/25, mu: 0.6100111604, learning_rate: 0.0010055049, weight_decay: 0.0002032248

epoch time: 2041.1035156250s, input_fired_time: 23.3760452271, output_fired_time: 9.8113098145, propagate_forward_time: 721.2341308594, propagate_backward_time: 1263.8721923828, weight_update_time: 0.5931379199, other_time: 22.8098144531
epoch time: 153.4633331299s, input_fired_time: 1.6894795895, output_fired_time: 0.0000000000, propagate_forward_time: 147.8088684082, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9649810791
validation[          , genome  6239] predictions:    6033/  10000 (60.33%), best:    6119/10000 (61.19%), error:     11461.50098, best error:     11208.86426 on epoch:     3, epoch:    6/25, mu: 0.6207228899, learning_rate: 0.0009180288, weight_decay: 0.0001857087

epoch time: 2044.1242675781s, input_fired_time: 22.3819828033, output_fired_time: 10.7609148026, propagate_forward_time: 728.3217163086, propagate_backward_time: 1257.7666015625, weight_update_time: 0.6167236567, other_time: 24.8930664062
epoch time: 150.8216247559s, input_fired_time: 2.5840005875, output_fired_time: 0.0000000000, propagate_forward_time: 144.5144042969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7232208252
validation[          , genome  6239] predictions:    6081/  10000 (60.81%), best:    6119/10000 (61.19%), error:     11293.28711, best error:     11208.86426 on epoch:     3, epoch:    7/25, mu: 0.6311326623, learning_rate: 0.0008381628, weight_decay: 0.0001697023

epoch time: 2044.3483886719s, input_fired_time: 22.4554119110, output_fired_time: 11.7983322144, propagate_forward_time: 734.1410522461, propagate_backward_time: 1254.2235107422, weight_update_time: 0.6005315185, other_time: 21.7301025391
epoch time: 150.4341888428s, input_fired_time: 1.5978716612, output_fired_time: 0.0000000000, propagate_forward_time: 145.0520019531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7843170166
validation[          , genome  6239] predictions:    6149/  10000 (61.49%), best:    6149/10000 (61.49%), error:     11073.27148, best error:     11073.27148 on epoch:     8, epoch:    8/25, mu: 0.6412490010, learning_rate: 0.0007652450, weight_decay: 0.0001550755

epoch time: 2049.3151855469s, input_fired_time: 25.4663219452, output_fired_time: 12.0255603790, propagate_forward_time: 725.2368164062, propagate_backward_time: 1262.3146972656, weight_update_time: 0.6125752926, other_time: 24.2718505859
epoch time: 153.0594482422s, input_fired_time: 1.7091886997, output_fired_time: 0.0000000000, propagate_forward_time: 147.4316253662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9186401367
validation[          , genome  6239] predictions:    6201/  10000 (62.01%), best:    6201/10000 (62.01%), error:     10834.67871, best error:     10834.67871 on epoch:     9, epoch:    9/25, mu: 0.6510801911, learning_rate: 0.0006986708, weight_decay: 0.0001417094

epoch time: 2021.9114990234s, input_fired_time: 24.1774940491, output_fired_time: 9.7096977234, propagate_forward_time: 728.8041992188, propagate_backward_time: 1233.9670410156, weight_update_time: 0.6027952433, other_time: 25.2530517578
epoch time: 156.1589355469s, input_fired_time: 2.6938636303, output_fired_time: 0.0000000000, propagate_forward_time: 148.4648590088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0002136230
validation[          , genome  6239] predictions:    6187/  10000 (61.87%), best:    6201/10000 (62.01%), error:     10936.66211, best error:     10834.67871 on epoch:     9, epoch:   10/25, mu: 0.6606342196, learning_rate: 0.0006378884, weight_decay: 0.0001294953

epoch time: 2048.7661132812s, input_fired_time: 25.5557842255, output_fired_time: 10.8421325684, propagate_forward_time: 722.5664062500, propagate_backward_time: 1264.0422363281, weight_update_time: 1.6038103104, other_time: 25.7595214844
epoch time: 157.9468536377s, input_fired_time: 1.7073132992, output_fired_time: 0.0000000000, propagate_forward_time: 151.9937286377, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2458190918
validation[          , genome  6239] predictions:    6220/  10000 (62.20%), best:    6201/10000 (62.01%), error:     10908.47656, best error:     10834.67871 on epoch:     9, epoch:   11/25, mu: 0.6699189544, learning_rate: 0.0005823939, weight_decay: 0.0001183340

epoch time: 2044.5675048828s, input_fired_time: 22.2661151886, output_fired_time: 10.7366428375, propagate_forward_time: 730.9339599609, propagate_backward_time: 1256.7719726562, weight_update_time: 0.6022235751, other_time: 23.8587646484
epoch time: 150.1316528320s, input_fired_time: 1.6737151146, output_fired_time: 0.0000000000, propagate_forward_time: 144.4531097412, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0048217773
validation[          , genome  6239] predictions:    6293/  10000 (62.93%), best:    6293/10000 (62.93%), error:     10671.71387, best error:     10671.71387 on epoch:    12, epoch:   12/25, mu: 0.6789419055, learning_rate: 0.0005317273, weight_decay: 0.0001081347

epoch time: 2029.2520751953s, input_fired_time: 22.3566722870, output_fired_time: 9.7714557648, propagate_forward_time: 724.8410034180, propagate_backward_time: 1248.3701171875, weight_update_time: 1.6034233570, other_time: 23.9128417969
epoch time: 147.9117736816s, input_fired_time: 1.5532590151, output_fired_time: 0.0000000000, propagate_forward_time: 142.5500183105, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8085021973
validation[          , genome  6239] predictions:    6245/  10000 (62.45%), best:    6293/10000 (62.93%), error:     10807.97754, best error:     10671.71387 on epoch:    12, epoch:   13/25, mu: 0.6877105236, learning_rate: 0.0004854685, weight_decay: 0.0000988144

epoch time: 2047.0336914062s, input_fired_time: 22.4515380859, output_fired_time: 11.2443847656, propagate_forward_time: 715.7886352539, propagate_backward_time: 1273.4898681641, weight_update_time: 0.6024541259, other_time: 24.0592041016
epoch time: 151.3936767578s, input_fired_time: 2.7030065060, output_fired_time: 0.0000000000, propagate_forward_time: 143.8109893799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8796844482
validation[          , genome  6239] predictions:    6177/  10000 (61.77%), best:    6293/10000 (62.93%), error:     11024.84082, best error:     10671.71387 on epoch:    12, epoch:   14/25, mu: 0.6962319613, learning_rate: 0.0004432341, weight_decay: 0.0000902975

epoch time: 2033.9517822266s, input_fired_time: 26.3147830963, output_fired_time: 11.8432760239, propagate_forward_time: 727.9036865234, propagate_backward_time: 1245.6370849609, weight_update_time: 0.6310161352, other_time: 22.2529296875
epoch time: 154.1817474365s, input_fired_time: 1.7248815298, output_fired_time: 0.0000000000, propagate_forward_time: 148.4916534424, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9652099609
validation[          , genome  6239] predictions:    6224/  10000 (62.24%), best:    6293/10000 (62.93%), error:     10874.97949, best error:     10671.71387 on epoch:    12, epoch:   15/25, mu: 0.7045131922, learning_rate: 0.0004046739, weight_decay: 0.0000825147

epoch time: 2656.7106933594s, input_fired_time: 30.9369831085, output_fired_time: 12.7375612259, propagate_forward_time: 981.6090698242, propagate_backward_time: 1601.4676513672, weight_update_time: 0.7164797187, other_time: 29.9593505859
epoch time: 245.4304962158s, input_fired_time: 3.3055016994, output_fired_time: 0.0000000000, propagate_forward_time: 235.9183197021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2066802979
validation[          , genome  6239] predictions:    6266/  10000 (62.66%), best:    6293/10000 (62.93%), error:     10740.39551, best error:     10671.71387 on epoch:    12, epoch:   16/25, mu: 0.7125609517, learning_rate: 0.0003694684, weight_decay: 0.0000754027

epoch time: 3350.5344238281s, input_fired_time: 38.8028488159, output_fired_time: 16.6811714172, propagate_forward_time: 1276.9913330078, propagate_backward_time: 1981.7318115234, weight_update_time: 0.8033128381, other_time: 36.3273925781
epoch time: 241.5549774170s, input_fired_time: 2.1092138290, output_fired_time: 0.0000000000, propagate_forward_time: 232.1427459717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3030242920
validation[          , genome  6239] predictions:    6229/  10000 (62.29%), best:    6293/10000 (62.93%), error:     10719.77637, best error:     10671.71387 on epoch:    12, epoch:   17/25, mu: 0.7203818560, learning_rate: 0.0003373257, weight_decay: 0.0000689037

epoch time: 3277.3220214844s, input_fired_time: 38.7081794739, output_fired_time: 16.5090370178, propagate_forward_time: 1239.6273193359, propagate_backward_time: 1944.3668212891, weight_update_time: 0.8129606247, other_time: 38.1105957031
epoch time: 154.9330444336s, input_fired_time: 1.7207120657, output_fired_time: 0.0000000000, propagate_forward_time: 147.2497558594, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9625701904
validation[          , genome  6239] predictions:    6313/  10000 (63.13%), best:    6313/10000 (63.13%), error:     10488.46973, best error:     10488.46973 on epoch:    18, epoch:   18/25, mu: 0.7279822826, learning_rate: 0.0003079793, weight_decay: 0.0000629648

epoch time: 2033.0423583984s, input_fired_time: 23.2833290100, output_fired_time: 9.8254079819, propagate_forward_time: 728.9420776367, propagate_backward_time: 1248.1414794922, weight_update_time: 0.5937289000, other_time: 22.8499755859
epoch time: 148.7172393799s, input_fired_time: 1.6137844324, output_fired_time: 0.0000000000, propagate_forward_time: 143.3692169189, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7342376709
validation[          , genome  6239] predictions:    6377/  10000 (63.77%), best:    6377/10000 (63.77%), error:     10372.78223, best error:     10372.78223 on epoch:    19, epoch:   19/25, mu: 0.7353684902, learning_rate: 0.0002811860, weight_decay: 0.0000575378

epoch time: 2051.4228515625s, input_fired_time: 22.5303268433, output_fired_time: 11.9712457657, propagate_forward_time: 743.0255737305, propagate_backward_time: 1250.4255371094, weight_update_time: 0.6189351082, other_time: 23.4702148438
epoch time: 155.0269927979s, input_fired_time: 2.6953394413, output_fired_time: 0.0000000000, propagate_forward_time: 148.3906860352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9409637451
validation[          , genome  6239] predictions:    6241/  10000 (62.41%), best:    6377/10000 (63.77%), error:     10675.23242, best error:     10372.78223 on epoch:    19, epoch:   20/25, mu: 0.7425464392, learning_rate: 0.0002567236, weight_decay: 0.0000525786

epoch time: 2039.2814941406s, input_fired_time: 24.5635967255, output_fired_time: 10.9132308960, propagate_forward_time: 726.0986938477, propagate_backward_time: 1253.8251953125, weight_update_time: 0.6079341769, other_time: 23.8808593750
epoch time: 154.5150451660s, input_fired_time: 1.7702888250, output_fired_time: 0.0000000000, propagate_forward_time: 148.7570495605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9877014160
validation[          , genome  6239] predictions:    6391/  10000 (63.91%), best:    6391/10000 (63.91%), error:     10302.59473, best error:     10302.59473 on epoch:    21, epoch:   21/25, mu: 0.7495220900, learning_rate: 0.0002343893, weight_decay: 0.0000480468

epoch time: 2046.5418701172s, input_fired_time: 22.5305862427, output_fired_time: 10.7660360336, propagate_forward_time: 732.0535888672, propagate_backward_time: 1257.2528076172, weight_update_time: 0.6157021523, other_time: 23.9388427734
epoch time: 152.6606597900s, input_fired_time: 1.7348773479, output_fired_time: 0.0000000000, propagate_forward_time: 145.9561004639, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9696807861
validation[          , genome  6239] predictions:    6468/  10000 (64.68%), best:    6468/10000 (64.68%), error:     10169.77539, best error:     10169.77539 on epoch:    22, epoch:   22/25, mu: 0.7563011050, learning_rate: 0.0002139981, weight_decay: 0.0000439056

epoch time: 2051.5100097656s, input_fired_time: 23.5618553162, output_fired_time: 10.9844064713, propagate_forward_time: 730.0297851562, propagate_backward_time: 1262.0323486328, weight_update_time: 0.6113889813, other_time: 24.9016113281
epoch time: 151.1606750488s, input_fired_time: 1.6357719898, output_fired_time: 0.0000000000, propagate_forward_time: 145.6870574951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8378448486
validation[          , genome  6239] predictions:    6397/  10000 (63.97%), best:    6468/10000 (64.68%), error:     10293.17676, best error:     10169.77539 on epoch:    22, epoch:   23/25, mu: 0.7628889680, learning_rate: 0.0001953809, weight_decay: 0.0000401213

epoch time: 2037.6499023438s, input_fired_time: 23.5953216553, output_fired_time: 10.7907714844, propagate_forward_time: 721.5773315430, propagate_backward_time: 1258.4935302734, weight_update_time: 0.6181542873, other_time: 23.1929931641
epoch time: 153.3249206543s, input_fired_time: 1.7171251774, output_fired_time: 0.0000000000, propagate_forward_time: 147.6277313232, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9800567627
validation[          , genome  6239] predictions:    6426/  10000 (64.26%), best:    6468/10000 (64.68%), error:     10215.15332, best error:     10169.77539 on epoch:    22, epoch:   24/25, mu: 0.7692911625, learning_rate: 0.0001783833, weight_decay: 0.0000366632

epoch time: 2036.3336181641s, input_fired_time: 21.5577449799, output_fired_time: 11.0254745483, propagate_forward_time: 721.6077880859, propagate_backward_time: 1257.4322509766, weight_update_time: 0.6057549119, other_time: 24.7103271484
epoch time: 150.4985046387s, input_fired_time: 1.7199581861, output_fired_time: 0.0000000000, propagate_forward_time: 144.8831787109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8953704834
validation[          , genome  6239] predictions:    6382/  10000 (63.82%), best:    6468/10000 (64.68%), error:     10306.58398, best error:     10169.77539 on epoch:    22, epoch:   25/25, mu: 0.7755128741, learning_rate: 0.0001628644, weight_decay: 0.0000335032

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 755.2797241211s, input_fired_time: 8.2361698151, output_fired_time: 0.0000000000, propagate_forward_time: 726.8302001953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 20.2133789062
best training[          , genome  6239] predictions:   35673/  50000 (71.35%), best:    6468/10000 (64.68%), error:     42537.87891, best error:     10169.77539 on epoch:    22, epoch:   26/25, mu: 0.7815591693, learning_rate: 0.0001486957, weight_decay: 0.0000306155
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 156.9210052490s, input_fired_time: 1.7281820774, output_fired_time: 0.0000000000, propagate_forward_time: 151.2081146240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9847106934
   testing[          , genome  6239] predictions:    6468/  10000 (64.68%), best:    6468/10000 (64.68%), error:     10169.77539, best error:     10169.77539 on epoch:    22, epoch:   26/25, mu: 0.7815591693, learning_rate: 0.0001486957, weight_decay: 0.0000306155
backpropagation finished successfully!
22:49:28 (11351): called boinc_finish(0)

</stderr_txt>
]]>