Nameexact_genome_1542383404_181_6096_2
Workunit4093502
Created21 Nov 2018, 18:13:01 UTC
Sent21 Nov 2018, 18:13:07 UTC
Report deadline11 Jan 2019, 18:18:52 UTC
Received23 Nov 2018, 20:36:25 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time23 hours 23 min 12 sec
CPU time23 hours 23 min 12 sec
Validate stateValid
Credit12,751.00
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size545.00 MB
Peak swap size543.77 MB
Peak disk usage11.17 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_181_6096.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_181_6096_2_r621229481_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_181_6096.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.476453
read mu: 0.476453
read mu_delta: 0.945496
read initial_learning_rate: 0.00155683
read learning_rate: 0.00155683
read learning_rate_delta: 0.921162
read initial_weight_decay: 0.00046869
read weight_decay: 0.00046869
read weight_decay_delta: 0.933077
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0682982
read input_dropout_probability: 0.00127031
read hidden_dropout_probability: 0.107182
read velocity_reset: 1291
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6096
read normal distribution: '1 0 0'
generator_str: '2055848533'
read generator: 2055848533
read generated_by_map:
1 disable_node 1
reading 90 nodes.
reading 336 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1781897548
post shuffle 1: 708153234
epoch time: 1664.81701660156s, input_fired_time: 18.1173534393311, output_fired_time: 9.3438663482666, propagate_forward_time: 671.38037109375, propagate_backward_time: 950.403930664063, weight_update_time: 0.539373934268951, other_time: 15.571533203125
epoch time: 123.47615814209s, input_fired_time: 1.30535328388214, output_fired_time: 0, propagate_forward_time: 119.73722076416, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.43358612060547
validation[          , genome  6096] predictions:    5533/  10000 (55.33%), best:    5533/10000 (55.33%), error:     13187.29492, best error:     13187.29492 on epoch:     0, epoch:    0/50, mu: 0.4764530659, learning_rate: 0.0015568265, weight_decay: 0.0004686896

epoch time: 1691.2025146484s, input_fired_time: 17.9218578339, output_fired_time: 9.3823957443, propagate_forward_time: 683.0838012695, propagate_backward_time: 965.9702758789, weight_update_time: 0.5293113589, other_time: 14.8441162109
epoch time: 97.5803527832s, input_fired_time: 1.1028876305, output_fired_time: 0.0000000000, propagate_forward_time: 94.2406463623, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2368164063
validation[          , genome  6096] predictions:    5692/  10000 (56.92%), best:    5692/10000 (56.92%), error:     12488.38574, best error:     12488.38574 on epoch:     1, epoch:    1/50, mu: 0.5044436455, learning_rate: 0.0014340900, weight_decay: 0.0004373237

epoch time: 1389.0045166016s, input_fired_time: 16.6092338562, output_fired_time: 7.7985024452, propagate_forward_time: 553.3540039063, propagate_backward_time: 797.3408203125, weight_update_time: 0.4539082348, other_time: 13.9019775391
epoch time: 162.4400329590s, input_fired_time: 1.5966788530, output_fired_time: 0.0000000000, propagate_forward_time: 158.0833282471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7600250244
validation[          , genome  6096] predictions:    5451/  10000 (54.51%), best:    5692/10000 (56.92%), error:     13298.97363, best error:     12488.38574 on epoch:     1, epoch:    2/50, mu: 0.5309085846, learning_rate: 0.0013210296, weight_decay: 0.0004080569

epoch time: 1698.0274658203s, input_fired_time: 17.9163932800, output_fired_time: 9.4162034988, propagate_forward_time: 686.9702148438, propagate_backward_time: 968.7471923828, weight_update_time: 0.5328476429, other_time: 14.9774169922
epoch time: 146.8088836670s, input_fired_time: 1.4572256804, output_fired_time: 0.0000000000, propagate_forward_time: 142.7067260742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6449279785
validation[          , genome  6096] predictions:    5414/  10000 (54.14%), best:    5692/10000 (56.92%), error:     13249.44922, best error:     12488.38574 on epoch:     1, epoch:    3/50, mu: 0.5559310913, learning_rate: 0.0012168827, weight_decay: 0.0003807487

epoch time: 1430.9825439453s, input_fired_time: 16.7186241150, output_fired_time: 7.9653997421, propagate_forward_time: 570.9292602539, propagate_backward_time: 821.2471313477, weight_update_time: 0.4670225084, other_time: 14.1220703125
epoch time: 92.1493911743s, input_fired_time: 1.0879585743, output_fired_time: 0.0000000000, propagate_forward_time: 88.7527313232, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3087005615
validation[          , genome  6096] predictions:    5607/  10000 (56.07%), best:    5692/10000 (56.92%), error:     12577.42090, best error:     12488.38574 on epoch:     1, epoch:    4/50, mu: 0.5795897841, learning_rate: 0.0011209465, weight_decay: 0.0003552680

epoch time: 1611.8106689453s, input_fired_time: 17.4998569489, output_fired_time: 8.9701414108, propagate_forward_time: 650.5955810547, propagate_backward_time: 920.0278320313, weight_update_time: 0.5110757351, other_time: 14.7172851563
epoch time: 86.0017547607s, input_fired_time: 1.0021194220, output_fired_time: 0.0000000000, propagate_forward_time: 82.7683868408, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2312469482
validation[          , genome  6096] predictions:    5675/  10000 (56.75%), best:    5675/10000 (56.75%), error:     12433.43945, best error:     12433.43945 on epoch:     5, epoch:    5/50, mu: 0.6019589305, learning_rate: 0.0010325736, weight_decay: 0.0003314925

epoch time: 1339.6961669922s, input_fired_time: 16.3138847351, output_fired_time: 7.4503450394, propagate_forward_time: 531.0576171875, propagate_backward_time: 770.9862060547, weight_update_time: 0.4451664984, other_time: 13.8881835938
epoch time: 111.7867813110s, input_fired_time: 1.2212426662, output_fired_time: 0.0000000000, propagate_forward_time: 108.0844116211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4811248779
validation[          , genome  6096] predictions:    5673/  10000 (56.73%), best:    5673/10000 (56.73%), error:     12402.38477, best error:     12402.38477 on epoch:     6, epoch:    6/50, mu: 0.6231088638, learning_rate: 0.0009511679, weight_decay: 0.0003093082

epoch time: 1640.1619873047s, input_fired_time: 17.6050834656, output_fired_time: 9.1390953064, propagate_forward_time: 662.2595825195, propagate_backward_time: 936.3648071289, weight_update_time: 0.5175108314, other_time: 14.7934570313
epoch time: 142.5754241943s, input_fired_time: 1.4249161482, output_fired_time: 0.0000000000, propagate_forward_time: 138.5451812744, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6053314209
validation[          , genome  6096] predictions:    5442/  10000 (54.42%), best:    5673/10000 (56.73%), error:     12973.41895, best error:     12402.38477 on epoch:     6, epoch:    7/50, mu: 0.6431060433, learning_rate: 0.0008761800, weight_decay: 0.0002886085

epoch time: 1495.8295898438s, input_fired_time: 17.1714572906, output_fired_time: 8.3814001083, propagate_forward_time: 600.3022460938, propagate_backward_time: 855.1448974609, weight_update_time: 0.4979948103, other_time: 14.8295898438
epoch time: 97.0633468628s, input_fired_time: 1.1161152124, output_fired_time: 0.0000000000, propagate_forward_time: 93.5847244263, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3625106812
validation[          , genome  6096] predictions:    5501/  10000 (55.01%), best:    5673/10000 (56.73%), error:     13012.68457, best error:     12402.38477 on epoch:     6, epoch:    8/50, mu: 0.6620132923, learning_rate: 0.0008071040, weight_decay: 0.0002692941

epoch time: 1535.2441406250s, input_fired_time: 17.2936229706, output_fired_time: 8.5554771423, propagate_forward_time: 616.1750488281, propagate_backward_time: 878.4055786133, weight_update_time: 0.5042141080, other_time: 14.8145141602
epoch time: 156.6593170166s, input_fired_time: 1.5251601934, output_fired_time: 0.0000000000, propagate_forward_time: 152.4331359863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7010192871
validation[          , genome  6096] predictions:    5675/  10000 (56.75%), best:    5673/10000 (56.73%), error:     12456.22656, best error:     12402.38477 on epoch:     6, epoch:    9/50, mu: 0.6798900366, learning_rate: 0.0007434738, weight_decay: 0.0002512723

epoch time: 1796.7886962891s, input_fired_time: 18.3178710938, output_fired_time: 10.0153217316, propagate_forward_time: 729.1686401367, propagate_backward_time: 1023.8515014648, weight_update_time: 0.5577819943, other_time: 15.4353637695
epoch time: 141.2573089600s, input_fired_time: 1.4522978067, output_fired_time: 0.0000000000, propagate_forward_time: 137.0957794189, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7092285156
validation[          , genome  6096] predictions:    5572/  10000 (55.72%), best:    5673/10000 (56.73%), error:     12653.79395, best error:     12402.38477 on epoch:     6, epoch:   10/50, mu: 0.6967923641, learning_rate: 0.0006848600, weight_decay: 0.0002344565

epoch time: 1250.9848632813s, input_fired_time: 15.9141168594, output_fired_time: 6.9579677582, propagate_forward_time: 495.3207397461, propagate_backward_time: 719.2558593750, weight_update_time: 0.4200299382, other_time: 13.5361938477
epoch time: 144.7768096924s, input_fired_time: 1.4547786713, output_fired_time: 0.0000000000, propagate_forward_time: 140.6275634766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6944732666
validation[          , genome  6096] predictions:    5071/  10000 (50.71%), best:    5673/10000 (56.73%), error:     15777.77344, best error:     12402.38477 on epoch:     6, epoch:   11/50, mu: 0.7127734423, learning_rate: 0.0006308672, weight_decay: 0.0002187660

epoch time: 1689.8204345703s, input_fired_time: 17.8715438843, output_fired_time: 9.3896350861, propagate_forward_time: 684.0706787109, propagate_backward_time: 963.3520507813, weight_update_time: 0.5352122784, other_time: 15.1364746094
epoch time: 132.1895294189s, input_fired_time: 1.3386415243, output_fired_time: 0.0000000000, propagate_forward_time: 128.3211822510, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5297088623
validation[          , genome  6096] predictions:    5754/  10000 (57.54%), best:    5754/10000 (57.54%), error:     12185.85645, best error:     12185.85645 on epoch:    12, epoch:   12/50, mu: 0.7278835177, learning_rate: 0.0005811311, weight_decay: 0.0002041257

epoch time: 1366.1436767578s, input_fired_time: 16.3979091644, output_fired_time: 7.5889706612, propagate_forward_time: 543.9700927734, propagate_backward_time: 784.2904663086, weight_update_time: 0.4504179060, other_time: 13.8961791992
epoch time: 130.1497955322s, input_fired_time: 1.3618738651, output_fired_time: 0.0000000000, propagate_forward_time: 126.1667022705, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6212158203
validation[          , genome  6096] predictions:    5812/  10000 (58.12%), best:    5812/10000 (58.12%), error:     11948.06836, best error:     11948.06836 on epoch:    13, epoch:   13/50, mu: 0.7421700358, learning_rate: 0.0005353161, weight_decay: 0.0001904651

epoch time: 1570.2550048828s, input_fired_time: 17.3695850372, output_fired_time: 8.7500085831, propagate_forward_time: 630.8060913086, propagate_backward_time: 898.6030273438, weight_update_time: 0.5050143600, other_time: 14.7262573242
epoch time: 96.6641006470s, input_fired_time: 1.1083111763, output_fired_time: 0.0000000000, propagate_forward_time: 93.2091827393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3466033936
validation[          , genome  6096] predictions:    5790/  10000 (57.90%), best:    5812/10000 (58.12%), error:     12062.74707, best error:     11948.06836 on epoch:    13, epoch:   14/50, mu: 0.7556778789, learning_rate: 0.0004931130, weight_decay: 0.0001777187

epoch time: 1578.9891357422s, input_fired_time: 17.4226436615, output_fired_time: 8.8036003113, propagate_forward_time: 635.5815429688, propagate_backward_time: 902.4221191406, weight_update_time: 0.5068618655, other_time: 14.7592773438
epoch time: 153.0258789063s, input_fired_time: 1.5096309185, output_fired_time: 0.0000000000, propagate_forward_time: 148.8311462402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6851043701
validation[          , genome  6096] predictions:    5860/  10000 (58.60%), best:    5860/10000 (58.60%), error:     11818.25781, best error:     11818.25781 on epoch:    15, epoch:   15/50, mu: 0.7684494853, learning_rate: 0.0004542371, weight_decay: 0.0001658253

epoch time: 1559.8239746094s, input_fired_time: 17.4215450287, output_fired_time: 8.7682991028, propagate_forward_time: 626.5256347656, propagate_backward_time: 892.2416992188, weight_update_time: 0.5065145493, other_time: 14.8668212891
epoch time: 87.2526626587s, input_fired_time: 1.0593683720, output_fired_time: 0.0000000000, propagate_forward_time: 83.8031845093, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3901062012
validation[          , genome  6096] predictions:    5919/  10000 (59.19%), best:    5919/10000 (59.19%), error:     11720.03027, best error:     11720.03027 on epoch:    16, epoch:   16/50, mu: 0.7805249691, learning_rate: 0.0004184261, weight_decay: 0.0001547278

epoch time: 1352.0150146484s, input_fired_time: 16.3544521332, output_fired_time: 7.5164337158, propagate_forward_time: 537.1800537109, propagate_backward_time: 777.2009277344, weight_update_time: 0.4460772872, other_time: 13.7630615234
epoch time: 129.2618865967s, input_fired_time: 1.3325148821, output_fired_time: 0.0000000000, propagate_forward_time: 125.4185104370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5108642578
validation[          , genome  6096] predictions:    5874/  10000 (58.74%), best:    5919/10000 (59.19%), error:     11765.91895, best error:     11720.03027 on epoch:    16, epoch:   17/50, mu: 0.7919422984, learning_rate: 0.0003854384, weight_decay: 0.0001443730

epoch time: 1459.7932128906s, input_fired_time: 17.1284523010, output_fired_time: 8.2077264786, propagate_forward_time: 583.3303222656, propagate_backward_time: 836.0797119141, weight_update_time: 0.4919019341, other_time: 15.0469970703
epoch time: 162.5964660645s, input_fired_time: 1.6336760521, output_fired_time: 0.0000000000, propagate_forward_time: 158.0676116943, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8951721191
validation[          , genome  6096] predictions:    5834/  10000 (58.34%), best:    5919/10000 (59.19%), error:     11843.68457, best error:     11720.03027 on epoch:    16, epoch:   18/50, mu: 0.8027372956, learning_rate: 0.0003550513, weight_decay: 0.0001347112

epoch time: 1446.3615722656s, input_fired_time: 16.7636222839, output_fired_time: 8.0135707855, propagate_forward_time: 579.5291748047, propagate_backward_time: 828.0021362305, weight_update_time: 0.4699273109, other_time: 14.0530395508
epoch time: 153.0477142334s, input_fired_time: 1.5057488680, output_fired_time: 0.0000000000, propagate_forward_time: 148.8695831299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6723785400
validation[          , genome  6096] predictions:    5973/  10000 (59.73%), best:    5973/10000 (59.73%), error:     11590.29590, best error:     11590.29590 on epoch:    19, epoch:   19/50, mu: 0.8129439354, learning_rate: 0.0003270599, weight_decay: 0.0001256960

epoch time: 1427.0318603516s, input_fired_time: 16.7786884308, output_fired_time: 7.9700622559, propagate_forward_time: 568.3099975586, propagate_backward_time: 819.3386230469, weight_update_time: 0.4685555995, other_time: 14.6344604492
epoch time: 129.9315032959s, input_fired_time: 1.3367326260, output_fired_time: 0.0000000000, propagate_forward_time: 125.9375839233, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6571884155
validation[          , genome  6096] predictions:    5981/  10000 (59.81%), best:    5981/10000 (59.81%), error:     11535.19922, best error:     11535.19922 on epoch:    20, epoch:   20/50, mu: 0.8225942850, learning_rate: 0.0003012752, weight_decay: 0.0001172841

epoch time: 1557.4398193359s, input_fired_time: 17.4680709839, output_fired_time: 8.7362565994, propagate_forward_time: 625.3084106445, propagate_backward_time: 890.5479125977, weight_update_time: 0.5074819922, other_time: 15.3792724609
epoch time: 128.4876708984s, input_fired_time: 1.3804961443, output_fired_time: 0.0000000000, propagate_forward_time: 123.8459777832, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2611999512
validation[          , genome  6096] predictions:    6027/  10000 (60.27%), best:    5981/10000 (59.81%), error:     11563.86816, best error:     11535.19922 on epoch:    20, epoch:   21/50, mu: 0.8317186236, learning_rate: 0.0002775234, weight_decay: 0.0001094352

epoch time: 1362.5587158203s, input_fired_time: 16.7243347168, output_fired_time: 7.7180957794, propagate_forward_time: 538.2245483398, propagate_backward_time: 784.2766723633, weight_update_time: 0.4635872841, other_time: 15.6149902344
epoch time: 128.1431579590s, input_fired_time: 1.3390203714, output_fired_time: 0.0000000000, propagate_forward_time: 124.0653228760, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7388153076
validation[          , genome  6096] predictions:    5984/  10000 (59.84%), best:    5981/10000 (59.81%), error:     11648.71387, best error:     11535.19922 on epoch:    20, epoch:   22/50, mu: 0.8403456807, learning_rate: 0.0002556441, weight_decay: 0.0001021115

epoch time: 1558.6679687500s, input_fired_time: 17.5029277802, output_fired_time: 8.7920217514, propagate_forward_time: 622.4524536133, propagate_backward_time: 894.0933227539, weight_update_time: 0.5094963908, other_time: 15.8272705078
epoch time: 104.5419540405s, input_fired_time: 1.1535928249, output_fired_time: 0.0000000000, propagate_forward_time: 101.0835037231, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3048553467
validation[          , genome  6096] predictions:    6080/  10000 (60.80%), best:    6080/10000 (60.80%), error:     11328.08887, best error:     11328.08887 on epoch:    23, epoch:   23/50, mu: 0.8485025167, learning_rate: 0.0002354897, weight_decay: 0.0000952779

epoch time: 1464.1950683594s, input_fired_time: 16.9747562408, output_fired_time: 8.1919345856, propagate_forward_time: 583.6010131836, propagate_backward_time: 840.6513061523, weight_update_time: 0.4774588645, other_time: 14.7761230469
epoch time: 137.2235260010s, input_fired_time: 1.4021106958, output_fired_time: 0.0000000000, propagate_forward_time: 133.2514343262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5699768066
validation[          , genome  6096] predictions:    5929/  10000 (59.29%), best:    6080/10000 (60.80%), error:     11672.51074, best error:     11328.08887 on epoch:    23, epoch:   24/50, mu: 0.8562147617, learning_rate: 0.0002169242, weight_decay: 0.0000889017

epoch time: 1424.9442138672s, input_fired_time: 16.7658252716, output_fired_time: 7.9566040039, propagate_forward_time: 568.3939819336, propagate_backward_time: 817.4821166992, weight_update_time: 0.4644913673, other_time: 14.3455810547
epoch time: 129.4645080566s, input_fired_time: 1.3477232456, output_fired_time: 0.0000000000, propagate_forward_time: 125.0822753906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0345153809
validation[          , genome  6096] predictions:    5992/  10000 (59.92%), best:    6080/10000 (60.80%), error:     11561.11621, best error:     11328.08887 on epoch:    23, epoch:   25/50, mu: 0.8635066748, learning_rate: 0.0001998224, weight_decay: 0.0000829522

epoch time: 1501.2331542969s, input_fired_time: 17.1134719849, output_fired_time: 8.3934535980, propagate_forward_time: 598.0395507813, propagate_backward_time: 862.6362915039, weight_update_time: 0.4885516763, other_time: 15.0503540039
epoch time: 131.7012329102s, input_fired_time: 1.3702300787, output_fired_time: 0.0000000000, propagate_forward_time: 127.8035354614, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5274734497
validation[          , genome  6096] predictions:    5872/  10000 (58.72%), best:    6080/10000 (60.80%), error:     11915.36133, best error:     11328.08887 on epoch:    23, epoch:   26/50, mu: 0.8704011440, learning_rate: 0.0001840689, weight_decay: 0.0000774008

epoch time: 1441.0684814453s, input_fired_time: 16.8312683105, output_fired_time: 8.0556507111, propagate_forward_time: 572.9320678711, propagate_backward_time: 828.5449218750, weight_update_time: 0.4756546915, other_time: 14.7045288086
epoch time: 129.6315612793s, input_fired_time: 1.3674149513, output_fired_time: 0.0000000000, propagate_forward_time: 124.8944015503, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3697433472
validation[          , genome  6096] predictions:    5779/  10000 (57.79%), best:    6080/10000 (60.80%), error:     12020.26563, best error:     11328.08887 on epoch:    23, epoch:   27/50, mu: 0.8769198060, learning_rate: 0.0001695573, weight_decay: 0.0000722209

epoch time: 1493.8912353516s, input_fired_time: 17.1650066376, output_fired_time: 8.3939609528, propagate_forward_time: 594.2369995117, propagate_backward_time: 858.7619628906, weight_update_time: 0.4916075468, other_time: 15.3333129883
epoch time: 143.3785705566s, input_fired_time: 1.4616476297, output_fired_time: 0.0000000000, propagate_forward_time: 139.1718139648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7451019287
validation[          , genome  6096] predictions:    6082/  10000 (60.82%), best:    6082/10000 (60.82%), error:     11262.55469, best error:     11262.55469 on epoch:    28, epoch:   28/50, mu: 0.8830831647, learning_rate: 0.0001561898, weight_decay: 0.0000673877

epoch time: 1553.5270996094s, input_fired_time: 17.4474239349, output_fired_time: 8.7561025620, propagate_forward_time: 618.1047363281, propagate_backward_time: 893.1479492188, weight_update_time: 0.5074748993, other_time: 16.0709228516
epoch time: 105.4691848755s, input_fired_time: 1.2363477945, output_fired_time: 0.0000000000, propagate_forward_time: 101.3046951294, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9281387329
validation[          , genome  6096] predictions:    6004/  10000 (60.04%), best:    6082/10000 (60.82%), error:     11456.91309, best error:     11262.55469 on epoch:    28, epoch:   29/50, mu: 0.8889105916, learning_rate: 0.0001438762, weight_decay: 0.0000628780

epoch time: 1523.8690185547s, input_fired_time: 17.8525104523, output_fired_time: 8.6968984604, propagate_forward_time: 605.5147094727, propagate_backward_time: 874.1739501953, weight_update_time: 0.5188332200, other_time: 17.6309204102
epoch time: 115.1177520752s, input_fired_time: 1.2905009985, output_fired_time: 0.0000000000, propagate_forward_time: 111.2895965576, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5376510620
validation[          , genome  6096] predictions:    5854/  10000 (58.54%), best:    6082/10000 (60.82%), error:     12066.21680, best error:     11262.55469 on epoch:    28, epoch:   30/50, mu: 0.8944203854, learning_rate: 0.0001325333, weight_decay: 0.0000586700

epoch time: 1461.6604003906s, input_fired_time: 17.3352375031, output_fired_time: 8.3157491684, propagate_forward_time: 581.5933837891, propagate_backward_time: 838.2332153320, weight_update_time: 0.4982834160, other_time: 16.1828002930
epoch time: 109.0875701904s, input_fired_time: 1.2406954765, output_fired_time: 0.0000000000, propagate_forward_time: 105.2864532471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5604248047
validation[          , genome  6096] predictions:    6045/  10000 (60.45%), best:    6082/10000 (60.82%), error:     11437.73828, best error:     11262.55469 on epoch:    28, epoch:   31/50, mu: 0.8996298909, learning_rate: 0.0001220847, weight_decay: 0.0000547437

epoch time: 1435.6514892578s, input_fired_time: 17.3733005524, output_fired_time: 8.2542676926, propagate_forward_time: 568.4143066406, propagate_backward_time: 824.7446289063, weight_update_time: 0.4936043024, other_time: 16.8649902344
epoch time: 91.2931518555s, input_fired_time: 1.1179305315, output_fired_time: 0.0000000000, propagate_forward_time: 87.6148300171, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5603942871
validation[          , genome  6096] predictions:    6082/  10000 (60.82%), best:    6082/10000 (60.82%), error:     11192.83594, best error:     11192.83594 on epoch:    32, epoch:   32/50, mu: 0.9045554399, learning_rate: 0.0001124598, weight_decay: 0.0000510801

epoch time: 1737.7509765625s, input_fired_time: 18.3102798462, output_fired_time: 9.8067560196, propagate_forward_time: 699.2412109375, propagate_backward_time: 993.5645141602, weight_update_time: 0.5553730726, other_time: 16.8281860352
epoch time: 148.8260498047s, input_fired_time: 1.5050936937, output_fired_time: 0.0000000000, propagate_forward_time: 143.4467010498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8742523193
validation[          , genome  6096] predictions:    6057/  10000 (60.57%), best:    6082/10000 (60.82%), error:     11298.80664, best error:     11192.83594 on epoch:    32, epoch:   33/50, mu: 0.9092125297, learning_rate: 0.0001035937, weight_decay: 0.0000476617

epoch time: 1615.3211669922s, input_fired_time: 17.8676223755, output_fired_time: 9.1323299408, propagate_forward_time: 645.2642211914, propagate_backward_time: 925.3512573242, weight_update_time: 0.5315635800, other_time: 17.7056884766
epoch time: 149.5895080566s, input_fired_time: 1.5504900217, output_fired_time: 0.0000000000, propagate_forward_time: 144.8687133789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1703033447
validation[          , genome  6096] predictions:    6049/  10000 (60.49%), best:    6082/10000 (60.82%), error:     11242.21387, best error:     11192.83594 on epoch:    32, epoch:   34/50, mu: 0.9136158228, learning_rate: 0.0000954267, weight_decay: 0.0000444720

epoch time: 1488.3948974609s, input_fired_time: 17.5424365997, output_fired_time: 8.4687232971, propagate_forward_time: 589.4131469727, propagate_backward_time: 855.6942138672, weight_update_time: 0.5014442205, other_time: 17.2763061523
epoch time: 148.4875640869s, input_fired_time: 1.5050158501, output_fired_time: 0.0000000000, propagate_forward_time: 144.0669555664, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9155883789
validation[          , genome  6096] predictions:    6077/  10000 (60.77%), best:    6082/10000 (60.82%), error:     11273.95215, best error:     11192.83594 on epoch:    32, epoch:   35/50, mu: 0.9177790880, learning_rate: 0.0000879034, weight_decay: 0.0000414958

epoch time: 1579.2047119141s, input_fired_time: 17.6025714874, output_fired_time: 8.8981714249, propagate_forward_time: 629.9429931641, propagate_backward_time: 906.4396362305, weight_update_time: 0.5135787725, other_time: 16.3213500977
epoch time: 137.9204406738s, input_fired_time: 1.4282125235, output_fired_time: 0.0000000000, propagate_forward_time: 132.8257598877, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6664733887
validation[          , genome  6096] predictions:    6158/  10000 (61.58%), best:    6158/10000 (61.58%), error:     11081.97754, best error:     11081.97754 on epoch:    36, epoch:   36/50, mu: 0.9217154384, learning_rate: 0.0000809733, weight_decay: 0.0000387188

epoch time: 1526.6368408203s, input_fired_time: 17.3505592346, output_fired_time: 8.5530147552, propagate_forward_time: 610.5495605469, propagate_backward_time: 874.9987792969, weight_update_time: 0.4960469007, other_time: 15.1849365234
epoch time: 102.8049011230s, input_fired_time: 1.2177178860, output_fired_time: 0.0000000000, propagate_forward_time: 98.8902893066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6968917847
validation[          , genome  6096] predictions:    6161/  10000 (61.61%), best:    6161/10000 (61.61%), error:     11065.68164, best error:     11065.68164 on epoch:    37, epoch:   37/50, mu: 0.9254372716, learning_rate: 0.0000745896, weight_decay: 0.0000361277

epoch time: 1453.4165039063s, input_fired_time: 17.0730304718, output_fired_time: 8.3203268051, propagate_forward_time: 573.4904174805, propagate_backward_time: 838.0990600586, weight_update_time: 0.4866093993, other_time: 16.4337158203
epoch time: 128.0035247803s, input_fired_time: 1.3600455523, output_fired_time: 0.0000000000, propagate_forward_time: 124.0930557251, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5504226685
validation[          , genome  6096] predictions:    6154/  10000 (61.54%), best:    6161/10000 (61.61%), error:     11171.11133, best error:     11065.68164 on epoch:    37, epoch:   38/50, mu: 0.9289562106, learning_rate: 0.0000687091, weight_decay: 0.0000337099

epoch time: 1477.7501220703s, input_fired_time: 17.1696166992, output_fired_time: 8.3712253571, propagate_forward_time: 585.7956542969, propagate_backward_time: 849.6972656250, weight_update_time: 0.4938870668, other_time: 16.7164306641
epoch time: 128.5218200684s, input_fired_time: 1.3697015047, output_fired_time: 0.0000000000, propagate_forward_time: 123.5556945801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5964202881
validation[          , genome  6096] predictions:    6183/  10000 (61.83%), best:    6161/10000 (61.61%), error:     11117.87402, best error:     11065.68164 on epoch:    37, epoch:   39/50, mu: 0.9322833419, learning_rate: 0.0000632922, weight_decay: 0.0000314540

epoch time: 1566.6981201172s, input_fired_time: 17.6360244751, output_fired_time: 9.0137844086, propagate_forward_time: 619.5751342773, propagate_backward_time: 902.1272583008, weight_update_time: 0.5260532498, other_time: 18.3459472656
epoch time: 138.7790374756s, input_fired_time: 1.4389722347, output_fired_time: 0.0000000000, propagate_forward_time: 134.1502990723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1897735596
validation[          , genome  6096] predictions:    6141/  10000 (61.41%), best:    6161/10000 (61.61%), error:     11129.55176, best error:     11065.68164 on epoch:    37, epoch:   40/50, mu: 0.9354291558, learning_rate: 0.0000583024, weight_decay: 0.0000293490

epoch time: 1524.6435546875s, input_fired_time: 17.4130420685, output_fired_time: 8.6366395950, propagate_forward_time: 605.0802612305, propagate_backward_time: 876.7518920898, weight_update_time: 0.5019590855, other_time: 16.7617187500
epoch time: 116.1899108887s, input_fired_time: 1.2710610628, output_fired_time: 0.0000000000, propagate_forward_time: 112.3006057739, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6182403564
validation[          , genome  6096] predictions:    6081/  10000 (60.81%), best:    6161/10000 (61.61%), error:     11183.84668, best error:     11065.68164 on epoch:    37, epoch:   41/50, mu: 0.9384034872, learning_rate: 0.0000537060, weight_decay: 0.0000273849

epoch time: 1449.1981201172s, input_fired_time: 17.0552864075, output_fired_time: 8.2255039215, propagate_forward_time: 571.4188232422, propagate_backward_time: 836.1824951172, weight_update_time: 0.4898554385, other_time: 16.3160400391
epoch time: 151.1031799316s, input_fired_time: 1.5929923058, output_fired_time: 0.0000000000, propagate_forward_time: 145.3436737061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1665191650
validation[          , genome  6096] predictions:    6144/  10000 (61.44%), best:    6161/10000 (61.61%), error:     11112.77637, best error:     11065.68164 on epoch:    37, epoch:   42/50, mu: 0.9412157536, learning_rate: 0.0000494719, weight_decay: 0.0000255522

epoch time: 1512.1247558594s, input_fired_time: 17.6064605713, output_fired_time: 8.6644325256, propagate_forward_time: 598.3387451172, propagate_backward_time: 869.8676147461, weight_update_time: 0.5102733970, other_time: 17.6475219727
epoch time: 95.2072296143s, input_fired_time: 1.1735785007, output_fired_time: 0.0000000000, propagate_forward_time: 91.5439071655, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4897460938
validation[          , genome  6096] predictions:    6204/  10000 (62.04%), best:    6204/10000 (62.04%), error:     10925.53809, best error:     10925.53809 on epoch:    43, epoch:   43/50, mu: 0.9438747168, learning_rate: 0.0000455717, weight_decay: 0.0000238422

epoch time: 1573.2989501953s, input_fired_time: 18.0116615295, output_fired_time: 8.9440383911, propagate_forward_time: 627.4926757813, propagate_backward_time: 901.4229736328, weight_update_time: 0.5278365612, other_time: 17.4274902344
epoch time: 117.7183532715s, input_fired_time: 1.2842543125, output_fired_time: 0.0000000000, propagate_forward_time: 113.3782653809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0558319092
validation[          , genome  6096] predictions:    6198/  10000 (61.98%), best:    6198/10000 (61.98%), error:     10893.08105, best error:     10893.08105 on epoch:    44, epoch:   44/50, mu: 0.9463887215, learning_rate: 0.0000419789, weight_decay: 0.0000222466

epoch time: 1529.1616210938s, input_fired_time: 17.7317962646, output_fired_time: 8.8508462906, propagate_forward_time: 600.6923217773, propagate_backward_time: 880.6022338867, weight_update_time: 0.5266500115, other_time: 21.2844238281
epoch time: 157.6709594727s, input_fired_time: 1.6017178297, output_fired_time: 0.0000000000, propagate_forward_time: 152.8070983887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2621459961
validation[          , genome  6096] predictions:    6132/  10000 (61.32%), best:    6198/10000 (61.98%), error:     11096.31250, best error:     10893.08105 on epoch:    44, epoch:   45/50, mu: 0.9487657547, learning_rate: 0.0000386694, weight_decay: 0.0000207578

epoch time: 1460.5380859375s, input_fired_time: 17.1447563171, output_fired_time: 8.2918004990, propagate_forward_time: 577.1707763672, propagate_backward_time: 841.6243896484, weight_update_time: 0.4898304343, other_time: 16.3063964844
epoch time: 123.2016067505s, input_fired_time: 1.3347959518, output_fired_time: 0.0000000000, propagate_forward_time: 118.7351455688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1316680908
validation[          , genome  6096] predictions:    6238/  10000 (62.38%), best:    6238/10000 (62.38%), error:     10843.75586, best error:     10843.75586 on epoch:    46, epoch:   46/50, mu: 0.9510132074, learning_rate: 0.0000356208, weight_decay: 0.0000193686

epoch time: 1535.1361083984s, input_fired_time: 17.6645164490, output_fired_time: 8.7916774750, propagate_forward_time: 606.2842407227, propagate_backward_time: 883.7899780273, weight_update_time: 0.5174279213, other_time: 18.6057128906
epoch time: 87.3737564087s, input_fired_time: 1.1190723181, output_fired_time: 0.0000000000, propagate_forward_time: 83.8175888062, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4370956421
validation[          , genome  6096] predictions:    6288/  10000 (62.88%), best:    6288/10000 (62.88%), error:     10753.26367, best error:     10753.26367 on epoch:    47, epoch:   47/50, mu: 0.9531381726, learning_rate: 0.0000328125, weight_decay: 0.0000180724

epoch time: 1366.7225341797s, input_fired_time: 17.1128768921, output_fired_time: 7.9037075043, propagate_forward_time: 538.5602416992, propagate_backward_time: 786.2677001953, weight_update_time: 0.4815065265, other_time: 16.8779907227
epoch time: 132.4601287842s, input_fired_time: 1.4549542665, output_fired_time: 0.0000000000, propagate_forward_time: 128.0990447998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9061279297
validation[          , genome  6096] predictions:    6278/  10000 (62.78%), best:    6288/10000 (62.88%), error:     10767.73633, best error:     10753.26367 on epoch:    47, epoch:   48/50, mu: 0.9551473260, learning_rate: 0.0000302257, weight_decay: 0.0000168630

epoch time: 1501.1461181641s, input_fired_time: 17.4835643768, output_fired_time: 8.5203561783, propagate_forward_time: 597.1389770508, propagate_backward_time: 861.4552612305, weight_update_time: 0.5040677190, other_time: 16.5479736328
epoch time: 125.9082107544s, input_fired_time: 1.3573973179, output_fired_time: 0.0000000000, propagate_forward_time: 121.5163879395, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0344238281
validation[          , genome  6096] predictions:    6240/  10000 (62.40%), best:    6288/10000 (62.88%), error:     10771.91699, best error:     10753.26367 on epoch:    47, epoch:   49/50, mu: 0.9570469260, learning_rate: 0.0000278427, weight_decay: 0.0000157345

epoch time: 1389.9165039063s, input_fired_time: 16.8422374725, output_fired_time: 7.8817696571, propagate_forward_time: 550.9332885742, propagate_backward_time: 798.6874389648, weight_update_time: 0.4664620757, other_time: 15.5717773438
epoch time: 111.3336334229s, input_fired_time: 1.2480685711, output_fired_time: 0.0000000000, propagate_forward_time: 107.4874420166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5981216431
validation[          , genome  6096] predictions:    6271/  10000 (62.71%), best:    6288/10000 (62.88%), error:     10765.95898, best error:     10753.26367 on epoch:    47, epoch:   50/50, mu: 0.9588429928, learning_rate: 0.0000256477, weight_decay: 0.0000146815

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 538.5312500000s, input_fired_time: 6.0513362885, output_fired_time: 0.0000000000, propagate_forward_time: 519.3365478516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.1433715820
best training[          , genome  6096] predictions:   34173/  50000 (68.35%), best:    6288/10000 (62.88%), error:     47284.46875, best error:     10753.26367 on epoch:    47, epoch:   51/50, mu: 0.9605411887, learning_rate: 0.0000236257, weight_decay: 0.0000136990
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 136.2872619629s, input_fired_time: 1.4324994087, output_fired_time: 0.0000000000, propagate_forward_time: 131.8903198242, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9644470215
   testing[          , genome  6096] predictions:    6288/  10000 (62.88%), best:    6288/10000 (62.88%), error:     10753.26367, best error:     10753.26367 on epoch:    47, epoch:   51/50, mu: 0.9605411887, learning_rate: 0.0000236257, weight_decay: 0.0000136990
backpropagation finished successfully!
14:33:52 (2696): called boinc_finish(0)

</stderr_txt>
]]>