Nameexact_genome_1542414603_170_6123_0
Workunit4094406
Created17 Nov 2018, 2:37:31 UTC
Sent17 Nov 2018, 3:06:28 UTC
Report deadline11 Dec 2018, 23:13:44 UTC
Received18 Nov 2018, 1:11:03 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID55826
Run time17 hours 20 min 17 sec
CPU time17 hours 5 min 23 sec
Validate stateValid
Credit6,209.60
Device peak FLOPS2.13 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size555.02 MB
Peak swap size556.27 MB
Peak disk usage6.96 MB

Stderr output

<core_client_version>7.6.31</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542414603_170_6123.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542414603_170_6123_0_r1428186028_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542414603_170_6123.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.456834
read mu: 0.456834
read mu_delta: 0.935303
read initial_learning_rate: 0.00137202
read learning_rate: 0.00137202
read learning_rate_delta: 0.903675
read initial_weight_decay: 0.000174862
read weight_decay: 0.000174862
read weight_decay_delta: 0.922257
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0491454
read input_dropout_probability: 0.00163287
read hidden_dropout_probability: 0.0794559
read velocity_reset: 1377
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6123
read normal distribution: '1 0.290435612201691 -0.831787467002869'
generator_str: '1725429695'
read generator: 1725429695
read generated_by_map:
1 add_edge 1
reading 95 nodes.
reading 342 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1825198424
post shuffle 1: 1137305266
epoch time: 2184.49072265625s, input_fired_time: 34.5559959411621, output_fired_time: 11.2312793731689, propagate_forward_time: 837.452880859375, propagate_backward_time: 1160.244140625, weight_update_time: 0.519638776779175, other_time: 141.006591796875
epoch time: 199.135681152344s, input_fired_time: 1.90513575077057, output_fired_time: 0, propagate_forward_time: 169.778884887695, propagate_backward_time: 0, weight_update_time: 0, other_time: 27.45166015625
validation[          , genome  6123] predictions:    6046/  10000 (60.46%), best:    6046/10000 (60.46%), error:     11289.81641, best error:     11289.81641 on epoch:     0, epoch:    0/25, mu: 0.4568337500, learning_rate: 0.0013720192, weight_decay: 0.0001748619

epoch time: 2181.4768066406s, input_fired_time: 34.3637466431, output_fired_time: 10.6216974258, propagate_forward_time: 838.3789672852, propagate_backward_time: 1158.9902343750, weight_update_time: 0.4616175592, other_time: 139.1223144531
epoch time: 199.7034606934s, input_fired_time: 1.9469578266, output_fired_time: 0.0000000000, propagate_forward_time: 170.2019653320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.5545349121
validation[          , genome  6123] predictions:    6178/  10000 (61.78%), best:    6178/10000 (61.78%), error:     11050.64258, best error:     11050.64258 on epoch:     1, epoch:    1/25, mu: 0.4913280308, learning_rate: 0.0012398595, weight_decay: 0.0001612676

epoch time: 2180.8942871094s, input_fired_time: 34.2348632812, output_fired_time: 10.4633121490, propagate_forward_time: 838.0138549805, propagate_backward_time: 1159.6363525391, weight_update_time: 0.4718229175, other_time: 138.5457763672
epoch time: 197.2696228027s, input_fired_time: 1.8567513227, output_fired_time: 0.0000000000, propagate_forward_time: 168.2785034180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.1343688965
validation[          , genome  6123] predictions:    6067/  10000 (60.67%), best:    6178/10000 (61.78%), error:     11213.46387, best error:     11050.64258 on epoch:     1, epoch:    2/25, mu: 0.5235906243, learning_rate: 0.0011204301, weight_decay: 0.0001487301

epoch time: 2180.2597656250s, input_fired_time: 34.1710929871, output_fired_time: 10.5076961517, propagate_forward_time: 839.1697998047, propagate_backward_time: 1158.1469726562, weight_update_time: 0.4658936858, other_time: 138.2640380859
epoch time: 198.5382080078s, input_fired_time: 1.9492334127, output_fired_time: 0.0000000000, propagate_forward_time: 169.1078491211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.4811248779
validation[          , genome  6123] predictions:    6121/  10000 (61.21%), best:    6178/10000 (61.78%), error:     11094.73633, best error:     11050.64258 on epoch:     1, epoch:    3/25, mu: 0.5537659526, learning_rate: 0.0010125047, weight_decay: 0.0001371673

epoch time: 2179.9379882812s, input_fired_time: 34.2590141296, output_fired_time: 10.4731798172, propagate_forward_time: 838.2815551758, propagate_backward_time: 1158.3096923828, weight_update_time: 0.4703333080, other_time: 138.6146240234
epoch time: 198.6906738281s, input_fired_time: 1.9002212286, output_fired_time: 0.0000000000, propagate_forward_time: 168.8159790039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.9744720459
validation[          , genome  6123] predictions:    6103/  10000 (61.03%), best:    6178/10000 (61.78%), error:     11166.81738, best error:     11050.64258 on epoch:     1, epoch:    4/25, mu: 0.5819889903, learning_rate: 0.0009149752, weight_decay: 0.0001265035

epoch time: 2182.9721679688s, input_fired_time: 34.3291702271, output_fired_time: 10.4985733032, propagate_forward_time: 838.7514648438, propagate_backward_time: 1160.7518310547, weight_update_time: 0.4880438149, other_time: 138.6412353516
epoch time: 199.4431610107s, input_fired_time: 1.9084854126, output_fired_time: 0.0000000000, propagate_forward_time: 170.0411682129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.4934997559
validation[          , genome  6123] predictions:    6153/  10000 (61.53%), best:    6153/10000 (61.53%), error:     10952.75293, best error:     10952.75293 on epoch:     5, epoch:    5/25, mu: 0.6083860993, learning_rate: 0.0008268402, weight_decay: 0.0001166686

epoch time: 2182.6601562500s, input_fired_time: 34.4185142517, output_fired_time: 10.4770908356, propagate_forward_time: 840.4766235352, propagate_backward_time: 1158.3571777344, weight_update_time: 0.4920035005, other_time: 138.9309082031
epoch time: 198.4554138184s, input_fired_time: 1.9153442383, output_fired_time: 0.0000000000, propagate_forward_time: 169.3926849365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.1473846436
validation[          , genome  6123] predictions:    6170/  10000 (61.70%), best:    6153/10000 (61.53%), error:     10983.21777, best error:     10952.75293 on epoch:     5, epoch:    6/25, mu: 0.6330753565, learning_rate: 0.0007471949, weight_decay: 0.0001075984

epoch time: 2183.1826171875s, input_fired_time: 34.1898422241, output_fired_time: 10.4121980667, propagate_forward_time: 839.8945312500, propagate_backward_time: 1159.8563232422, weight_update_time: 0.4646678865, other_time: 138.8297119141
epoch time: 199.9326019287s, input_fired_time: 1.9756616354, output_fired_time: 0.0000000000, propagate_forward_time: 170.2083282471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.7486114502
validation[          , genome  6123] predictions:    6220/  10000 (62.20%), best:    6220/10000 (62.20%), error:     10796.77441, best error:     10796.77441 on epoch:     7, epoch:    7/25, mu: 0.6561673284, learning_rate: 0.0006752213, weight_decay: 0.0000992334

epoch time: 2182.2458496094s, input_fired_time: 34.3870201111, output_fired_time: 10.4815559387, propagate_forward_time: 837.6733398438, propagate_backward_time: 1160.8052978516, weight_update_time: 0.4695820808, other_time: 138.8988037109
epoch time: 199.8123168945s, input_fired_time: 1.9627616405, output_fired_time: 0.0000000000, propagate_forward_time: 170.3183593750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.5311889648
validation[          , genome  6123] predictions:    6298/  10000 (62.98%), best:    6298/10000 (62.98%), error:     10724.22949, best error:     10724.22949 on epoch:     8, epoch:    8/25, mu: 0.6777653098, learning_rate: 0.0006101807, weight_decay: 0.0000915186

epoch time: 2176.7351074219s, input_fired_time: 34.3489990234, output_fired_time: 10.4143848419, propagate_forward_time: 836.2749633789, propagate_backward_time: 1157.6436767578, weight_update_time: 0.4638620615, other_time: 138.0533447266
epoch time: 196.8684234619s, input_fired_time: 1.8762441874, output_fired_time: 0.0000000000, propagate_forward_time: 168.0078277588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.9843444824
validation[          , genome  6123] predictions:    6223/  10000 (62.23%), best:    6298/10000 (62.98%), error:     10780.47949, best error:     10724.22949 on epoch:     8, epoch:    9/25, mu: 0.6979659796, learning_rate: 0.0005514050, weight_decay: 0.0000844037

epoch time: 2173.8610839844s, input_fired_time: 34.3045463562, output_fired_time: 10.5423135757, propagate_forward_time: 836.3538818359, propagate_backward_time: 1154.0965576172, weight_update_time: 0.4655163586, other_time: 138.5639648438
epoch time: 197.8181457520s, input_fired_time: 1.9905029535, output_fired_time: 0.0000000000, propagate_forward_time: 168.5995330811, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.2281036377
validation[          , genome  6123] predictions:    6316/  10000 (63.16%), best:    6316/10000 (63.16%), error:     10539.28613, best error:     10539.28613 on epoch:    10, epoch:   10/25, mu: 0.7168596983, learning_rate: 0.0004982909, weight_decay: 0.0000778419

epoch time: 2171.9274902344s, input_fired_time: 34.2032775879, output_fired_time: 10.3667716980, propagate_forward_time: 834.9331054688, propagate_backward_time: 1154.4459228516, weight_update_time: 0.4627482295, other_time: 137.9783935547
epoch time: 197.7076416016s, input_fired_time: 1.9458067417, output_fired_time: 0.0000000000, propagate_forward_time: 168.6149139404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.1469268799
validation[          , genome  6123] predictions:    6073/  10000 (60.73%), best:    6316/10000 (63.16%), error:     11211.78809, best error:     10539.28613 on epoch:    10, epoch:   11/25, mu: 0.7345310450, learning_rate: 0.0004502931, weight_decay: 0.0000717902

epoch time: 2170.7265625000s, input_fired_time: 34.3077507019, output_fired_time: 10.2696819305, propagate_forward_time: 835.5653076172, propagate_backward_time: 1152.9798583984, weight_update_time: 0.4656563103, other_time: 137.6037597656
epoch time: 197.3314361572s, input_fired_time: 1.9082386494, output_fired_time: 0.0000000000, propagate_forward_time: 168.3060302734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.1171722412
validation[          , genome  6123] predictions:    6233/  10000 (62.33%), best:    6316/10000 (63.16%), error:     10630.68945, best error:     10539.28613 on epoch:    10, epoch:   12/25, mu: 0.7510591149, learning_rate: 0.0004069186, weight_decay: 0.0000662090

epoch time: 2172.1757812500s, input_fired_time: 34.2497062683, output_fired_time: 10.3624515533, propagate_forward_time: 834.5790405273, propagate_backward_time: 1154.5590820312, weight_update_time: 0.4611701667, other_time: 138.4252929688
epoch time: 199.1311950684s, input_fired_time: 1.9683393240, output_fired_time: 0.0000000000, propagate_forward_time: 169.7657775879, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.3970794678
validation[          , genome  6123] predictions:    6268/  10000 (62.68%), best:    6316/10000 (63.16%), error:     10631.29395, best error:     10539.28613 on epoch:    10, epoch:   13/25, mu: 0.7665178776, learning_rate: 0.0003677222, weight_decay: 0.0000610617

epoch time: 2173.3342285156s, input_fired_time: 34.3590545654, output_fired_time: 10.4478454590, propagate_forward_time: 836.1927490234, propagate_backward_time: 1154.7623291016, weight_update_time: 0.4666999280, other_time: 137.5722656250
epoch time: 197.7845916748s, input_fired_time: 1.9121396542, output_fired_time: 0.0000000000, propagate_forward_time: 169.0251617432, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.8472900391
validation[          , genome  6123] predictions:    6248/  10000 (62.48%), best:    6316/10000 (63.16%), error:     10707.08691, best error:     10539.28613 on epoch:    10, epoch:   14/25, mu: 0.7809765339, learning_rate: 0.0003323013, weight_decay: 0.0000563145

epoch time: 2172.8271484375s, input_fired_time: 34.2538070679, output_fired_time: 10.6458816528, propagate_forward_time: 835.5981445312, propagate_backward_time: 1154.0002441406, weight_update_time: 0.4684520364, other_time: 138.3288574219
epoch time: 196.9136962891s, input_fired_time: 1.8973159790, output_fired_time: 0.0000000000, propagate_forward_time: 167.9772796631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.0391082764
validation[          , genome  6123] predictions:    6256/  10000 (62.56%), best:    6316/10000 (63.16%), error:     10755.55273, best error:     10539.28613 on epoch:    10, epoch:   15/25, mu: 0.7944997549, learning_rate: 0.0003002924, weight_decay: 0.0000519364

epoch time: 2174.0275878906s, input_fired_time: 34.2377700806, output_fired_time: 10.3680582047, propagate_forward_time: 835.2105712891, propagate_backward_time: 1156.3044433594, weight_update_time: 0.4943711758, other_time: 137.9066162109
epoch time: 198.1075897217s, input_fired_time: 1.8971066475, output_fired_time: 0.0000000000, propagate_forward_time: 168.6011505127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.6093292236
validation[          , genome  6123] predictions:    6283/  10000 (62.83%), best:    6316/10000 (63.16%), error:     10549.98926, best error:     10539.28613 on epoch:    10, epoch:   16/25, mu: 0.8071480393, learning_rate: 0.0002713668, weight_decay: 0.0000478987

epoch time: 2164.8967285156s, input_fired_time: 34.2283859253, output_fired_time: 10.4058542252, propagate_forward_time: 831.4602050781, propagate_backward_time: 1150.6010742188, weight_update_time: 0.4690842628, other_time: 138.2014160156
epoch time: 197.0069427490s, input_fired_time: 1.9204069376, output_fired_time: 0.0000000000, propagate_forward_time: 168.0348815918, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 27.0516510010
validation[          , genome  6123] predictions:    6341/  10000 (63.41%), best:    6341/10000 (63.41%), error:     10476.23730, best error:     10476.23730 on epoch:    17, epoch:   17/25, mu: 0.8189780116, learning_rate: 0.0002452274, weight_decay: 0.0000441749

epoch time: 2166.7128906250s, input_fired_time: 34.2204895020, output_fired_time: 10.3721551895, propagate_forward_time: 832.8841552734, propagate_backward_time: 1151.5557861328, weight_update_time: 0.4707432687, other_time: 137.6804199219
epoch time: 196.5283508301s, input_fired_time: 1.9699288607, output_fired_time: 0.0000000000, propagate_forward_time: 167.7834930420, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.7749328613
validation[          , genome  6123] predictions:    6302/  10000 (63.02%), best:    6341/10000 (63.41%), error:     10625.18750, best error:     10476.23730 on epoch:    17, epoch:   18/25, mu: 0.8300426006, learning_rate: 0.0002216058, weight_decay: 0.0000407406

epoch time: 2161.3916015625s, input_fired_time: 34.1879653931, output_fired_time: 10.3132877350, propagate_forward_time: 830.0473022461, propagate_backward_time: 1149.8334960938, weight_update_time: 0.4692329168, other_time: 137.0095214844
epoch time: 194.1947479248s, input_fired_time: 1.9085634947, output_fired_time: 0.0000000000, propagate_forward_time: 165.4183807373, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.8677978516
validation[          , genome  6123] predictions:    6340/  10000 (63.40%), best:    6341/10000 (63.41%), error:     10496.16895, best error:     10476.23730 on epoch:    17, epoch:   19/25, mu: 0.8403913975, learning_rate: 0.0002002597, weight_decay: 0.0000375733

epoch time: 2139.0710449219s, input_fired_time: 33.9326133728, output_fired_time: 10.2800598145, propagate_forward_time: 820.4566650391, propagate_backward_time: 1138.8703613281, weight_update_time: 0.4689534307, other_time: 135.5313720703
epoch time: 194.6790466309s, input_fired_time: 1.8984918594, output_fired_time: 0.0000000000, propagate_forward_time: 165.9728240967, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.8077239990
validation[          , genome  6123] predictions:    6263/  10000 (62.63%), best:    6341/10000 (63.41%), error:     10618.97070, best error:     10476.23730 on epoch:    17, epoch:   20/25, mu: 0.8500706553, learning_rate: 0.0001809697, weight_decay: 0.0000346522

epoch time: 2144.4277343750s, input_fired_time: 33.9026069641, output_fired_time: 10.1703424454, propagate_forward_time: 822.9182128906, propagate_backward_time: 1142.0208740234, weight_update_time: 0.4627173543, other_time: 135.4156494141
epoch time: 195.0510406494s, input_fired_time: 1.9318047762, output_fired_time: 0.0000000000, propagate_forward_time: 166.5417785645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.5774536133
validation[          , genome  6123] predictions:    6443/  10000 (64.43%), best:    6443/10000 (64.43%), error:     10310.89551, best error:     10310.89551 on epoch:    21, epoch:   21/25, mu: 0.8591236472, learning_rate: 0.0001635378, weight_decay: 0.0000319583

epoch time: 2141.9375000000s, input_fired_time: 33.8043670654, output_fired_time: 10.2429924011, propagate_forward_time: 824.0294189453, propagate_backward_time: 1139.0311279297, weight_update_time: 0.4629139900, other_time: 134.8295898438
epoch time: 195.5064849854s, input_fired_time: 1.9299957752, output_fired_time: 0.0000000000, propagate_forward_time: 167.0639801025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.5125122070
validation[          , genome  6123] predictions:    6413/  10000 (64.13%), best:    6443/10000 (64.43%), error:     10345.84961, best error:     10310.89551 on epoch:    21, epoch:   22/25, mu: 0.8675909638, learning_rate: 0.0001477850, weight_decay: 0.0000294737

epoch time: 2135.1293945312s, input_fired_time: 33.7140197754, output_fired_time: 10.2434549332, propagate_forward_time: 817.7565307617, propagate_backward_time: 1138.1602783203, weight_update_time: 0.4644332230, other_time: 135.2550048828
epoch time: 186.6634979248s, input_fired_time: 1.8721776009, output_fired_time: 0.0000000000, propagate_forward_time: 158.6881866455, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.1031341553
validation[          , genome  6123] predictions:    6359/  10000 (63.59%), best:    6443/10000 (64.43%), error:     10459.39844, best error:     10310.89551 on epoch:    21, epoch:   23/25, mu: 0.8755104542, learning_rate: 0.0001335496, weight_decay: 0.0000271823

epoch time: 2070.1767578125s, input_fired_time: 33.3798408508, output_fired_time: 9.8128490448, propagate_forward_time: 781.0060424805, propagate_backward_time: 1112.7000732422, weight_update_time: 0.4530774653, other_time: 133.2779541016
epoch time: 184.3127136230s, input_fired_time: 1.8657793999, output_fired_time: 0.0000000000, propagate_forward_time: 156.2159729004, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 26.2309570312
validation[          , genome  6123] predictions:    6408/  10000 (64.08%), best:    6443/10000 (64.43%), error:     10412.46289, best error:     10310.89551 on epoch:    21, epoch:   24/25, mu: 0.8829175830, learning_rate: 0.0001206854, weight_decay: 0.0000250691

epoch time: 2057.3576660156s, input_fired_time: 33.1321067810, output_fired_time: 9.7324218750, propagate_forward_time: 773.3120727539, propagate_backward_time: 1109.1057128906, weight_update_time: 0.4514457285, other_time: 132.0754394531
epoch time: 183.6738891602s, input_fired_time: 1.8428580761, output_fired_time: 0.0000000000, propagate_forward_time: 156.1920166016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 25.6390075684
validation[          , genome  6123] predictions:    6441/  10000 (64.41%), best:    6441/10000 (64.41%), error:     10301.50586, best error:     10301.50586 on epoch:    25, epoch:   25/25, mu: 0.8898454905, learning_rate: 0.0001090604, weight_decay: 0.0000231201

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 910.6086425781s, input_fired_time: 9.2026786804, output_fired_time: 0.0000000000, propagate_forward_time: 773.7157592773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 127.6901855469
best training[          , genome  6123] predictions:   36017/  50000 (72.03%), best:    6441/10000 (64.41%), error:     41934.30078, best error:     10301.50586 on epoch:    25, epoch:   26/25, mu: 0.8963251710, learning_rate: 0.0000985552, weight_decay: 0.0000213227
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 184.3845214844s, input_fired_time: 1.8621932268, output_fired_time: 0.0000000000, propagate_forward_time: 156.7144927979, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 25.8078308105
   testing[          , genome  6123] predictions:    6441/  10000 (64.41%), best:    6441/10000 (64.41%), error:     10301.50586, best error:     10301.50586 on epoch:    25, epoch:   26/25, mu: 0.8963251710, learning_rate: 0.0000985552, weight_decay: 0.0000213227
backpropagation finished successfully!
01:10:50 (32555): called boinc_finish(0)

</stderr_txt>
]]>