Nameexact_genome_1542649503_180_6348_2
Workunit4099750
Created22 Nov 2018, 10:33:07 UTC
Sent22 Nov 2018, 10:33:21 UTC
Report deadline29 Jan 2019, 12:39:45 UTC
Received24 Nov 2018, 6:54:54 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time1 days 6 hours 34 min 56 sec
CPU time1 days 6 hours 34 min 56 sec
Validate stateValid
Credit17,021.95
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size574.15 MB
Peak swap size572.90 MB
Peak disk usage12.82 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6348.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6348_2_r858333079_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6348.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486688
read mu: 0.486688
read mu_delta: 0.933427
read initial_learning_rate: 0.000740584
read learning_rate: 0.000740584
read learning_rate_delta: 0.916685
read initial_weight_decay: 0.000507109
read weight_decay: 0.000507109
read weight_decay_delta: 0.939645
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0996183
read input_dropout_probability: 0.00136941
read hidden_dropout_probability: 0.0961017
read velocity_reset: 1660
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6348
read normal distribution: '1 0 0'
generator_str: '729879664'
read generator: 729879664
read generated_by_map:
1 crossover 1
reading 102 nodes.
reading 402 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 660921184
post shuffle 1: 791707485
epoch time: 2089.76245117188s, input_fired_time: 30.4760456085205, output_fired_time: 15.6513671875, propagate_forward_time: 834.922790527344, propagate_backward_time: 1189.34204101563, weight_update_time: 0.6280597448349, other_time: 19.3701171875
epoch time: 212.931533813477s, input_fired_time: 2.90072727203369, output_fired_time: 0, propagate_forward_time: 206.207641601563, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.82316589355469
validation[          , genome  6348] predictions:    6127/  10000 (61.27%), best:    6127/10000 (61.27%), error:     11208.78809, best error:     11208.78809 on epoch:     0, epoch:    0/50, mu: 0.4866880178, learning_rate: 0.0007405839, weight_decay: 0.0005071091

epoch time: 1998.3710937500s, input_fired_time: 29.7012805939, output_fired_time: 15.1037578583, propagate_forward_time: 794.1963500977, propagate_backward_time: 1140.7209472656, weight_update_time: 0.5969889760, other_time: 18.6486816406
epoch time: 146.9011840820s, input_fired_time: 2.1383793354, output_fired_time: 0.0000000000, propagate_forward_time: 141.5880279541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1747741699
validation[          , genome  6348] predictions:    5891/  10000 (58.91%), best:    6127/10000 (61.27%), error:     11834.94238, best error:     11208.78809 on epoch:     0, epoch:    1/50, mu: 0.5201949477, learning_rate: 0.0006788824, weight_decay: 0.0004765026

epoch time: 2063.8076171875s, input_fired_time: 30.0965442657, output_fired_time: 15.5288019180, propagate_forward_time: 824.6243896484, propagate_backward_time: 1174.5682373047, weight_update_time: 0.6158096194, other_time: 18.9896240234
epoch time: 209.9342041016s, input_fired_time: 2.7823355198, output_fired_time: 0.0000000000, propagate_forward_time: 203.5261230469, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6257476807
validation[          , genome  6348] predictions:    6077/  10000 (60.77%), best:    6127/10000 (61.27%), error:     11303.00879, best error:     11208.78809 on epoch:     0, epoch:    2/50, mu: 0.5514712334, learning_rate: 0.0006223215, weight_decay: 0.0004477433

epoch time: 2073.7172851563s, input_fired_time: 30.3801937103, output_fired_time: 15.7125930786, propagate_forward_time: 827.1575317383, propagate_backward_time: 1180.6835937500, weight_update_time: 0.6330468059, other_time: 19.7832031250
epoch time: 125.3027038574s, input_fired_time: 1.9697448015, output_fired_time: 0.0000000000, propagate_forward_time: 119.9960708618, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3368911743
validation[          , genome  6348] predictions:    5976/  10000 (59.76%), best:    6127/10000 (61.27%), error:     11769.17773, best error:     11208.78809 on epoch:     0, epoch:    3/50, mu: 0.5806653500, learning_rate: 0.0005704730, weight_decay: 0.0004207198

epoch time: 1883.6740722656s, input_fired_time: 29.3271484375, output_fired_time: 14.3143510818, propagate_forward_time: 742.4714355469, propagate_backward_time: 1077.9979248047, weight_update_time: 0.5947187543, other_time: 19.5632324219
epoch time: 181.7920837402s, input_fired_time: 2.5158455372, output_fired_time: 0.0000000000, propagate_forward_time: 175.6105041504, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6657409668
validation[          , genome  6348] predictions:    6048/  10000 (60.48%), best:    6127/10000 (61.27%), error:     11508.24121, best error:     11208.78809 on epoch:     0, epoch:    4/50, mu: 0.6079159379, learning_rate: 0.0005229442, weight_decay: 0.0003953273

epoch time: 1843.7089843750s, input_fired_time: 29.0256710052, output_fired_time: 14.0597581863, propagate_forward_time: 727.5426025391, propagate_backward_time: 1053.8178710938, weight_update_time: 0.5782868266, other_time: 19.2630615234
epoch time: 139.0683441162s, input_fired_time: 2.0153241158, output_fired_time: 0.0000000000, propagate_forward_time: 133.9183197021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1347045898
validation[          , genome  6348] predictions:    6014/  10000 (60.14%), best:    6127/10000 (61.27%), error:     11650.91113, best error:     11208.78809 on epoch:     0, epoch:    5/50, mu: 0.6333523393, learning_rate: 0.0004793752, weight_decay: 0.0003714674

epoch time: 1999.6961669922s, input_fired_time: 29.8195781708, output_fired_time: 15.2368440628, propagate_forward_time: 792.8312377930, propagate_backward_time: 1142.1264648438, weight_update_time: 0.6169648170, other_time: 19.6821289063
epoch time: 165.0548400879s, input_fired_time: 2.3711721897, output_fired_time: 0.0000000000, propagate_forward_time: 158.7730712891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9105987549
validation[          , genome  6348] predictions:    5574/  10000 (55.74%), best:    6127/10000 (61.27%), error:     12881.15723, best error:     11208.78809 on epoch:     0, epoch:    6/50, mu: 0.6570953727, learning_rate: 0.0004394362, weight_decay: 0.0003490475

epoch time: 2053.2353515625s, input_fired_time: 30.2345333099, output_fired_time: 15.6075201035, propagate_forward_time: 816.2795410156, propagate_backward_time: 1170.9464111328, weight_update_time: 0.6299523711, other_time: 20.1673583984
epoch time: 148.3436279297s, input_fired_time: 2.1259653568, output_fired_time: 0.0000000000, propagate_forward_time: 142.9635772705, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2540893555
validation[          , genome  6348] predictions:    5835/  10000 (58.35%), best:    6127/10000 (61.27%), error:     12011.82715, best error:     11208.78809 on epoch:     0, epoch:    7/50, mu: 0.6792578101, learning_rate: 0.0004028247, weight_decay: 0.0003279808

epoch time: 2125.2963867188s, input_fired_time: 30.3756446838, output_fired_time: 16.0956230164, propagate_forward_time: 845.7156982422, propagate_backward_time: 1213.0832519531, weight_update_time: 0.6343095899, other_time: 20.0260009766
epoch time: 208.4745483398s, input_fired_time: 2.6978135109, output_fired_time: 0.0000000000, propagate_forward_time: 202.1643524170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6123809814
validation[          , genome  6348] predictions:    6118/  10000 (61.18%), best:    6127/10000 (61.27%), error:     11290.62891, best error:     11208.78809 on epoch:     0, epoch:    8/50, mu: 0.6999447942, learning_rate: 0.0003692635, weight_decay: 0.0003081856

epoch time: 2159.3544921875s, input_fired_time: 30.7430610657, output_fired_time: 16.3836441040, propagate_forward_time: 861.2445068359, propagate_backward_time: 1230.5319824219, weight_update_time: 0.6575867534, other_time: 20.4512939453
epoch time: 143.6439514160s, input_fired_time: 2.1873962879, output_fired_time: 0.0000000000, propagate_forward_time: 137.4166717529, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0398864746
validation[          , genome  6348] predictions:    6020/  10000 (60.20%), best:    6127/10000 (61.27%), error:     11577.45215, best error:     11208.78809 on epoch:     0, epoch:    9/50, mu: 0.7192546129, learning_rate: 0.0003384984, weight_decay: 0.0002895851

epoch time: 1950.2277832031s, input_fired_time: 29.6361751556, output_fired_time: 14.8266649246, propagate_forward_time: 771.5248413086, propagate_backward_time: 1114.2071533203, weight_update_time: 0.6080427766, other_time: 20.0328369141
epoch time: 136.6650695801s, input_fired_time: 1.9982626438, output_fired_time: 0.0000000000, propagate_forward_time: 131.4210815430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2457275391
validation[          , genome  6348] predictions:    6129/  10000 (61.29%), best:    6129/10000 (61.29%), error:     11075.95508, best error:     11075.95508 on epoch:    10, epoch:   10/50, mu: 0.7372788787, learning_rate: 0.0003102965, weight_decay: 0.0002721072

epoch time: 1861.5534667969s, input_fired_time: 28.9969520569, output_fired_time: 14.3239955902, propagate_forward_time: 730.8514404297, propagate_backward_time: 1067.3386230469, weight_update_time: 0.5828332305, other_time: 20.0424804688
epoch time: 134.0510253906s, input_fired_time: 1.9794013500, output_fired_time: 0.0000000000, propagate_forward_time: 128.6150360107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4565887451
validation[          , genome  6348] predictions:    6041/  10000 (60.41%), best:    6129/10000 (61.29%), error:     11470.35840, best error:     11075.95508 on epoch:    10, epoch:   11/50, mu: 0.7541032434, learning_rate: 0.0002844442, weight_decay: 0.0002556842

epoch time: 2044.5388183594s, input_fired_time: 30.5426349640, output_fired_time: 15.7818450928, propagate_forward_time: 807.4420776367, propagate_backward_time: 1168.0587158203, weight_update_time: 0.6480513811, other_time: 22.7135009766
epoch time: 178.8939208984s, input_fired_time: 2.4874029160, output_fired_time: 0.0000000000, propagate_forward_time: 172.8008880615, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6056365967
validation[          , genome  6348] predictions:    6204/  10000 (62.04%), best:    6129/10000 (61.29%), error:     11091.25488, best error:     11075.95508 on epoch:    10, epoch:   12/50, mu: 0.7698075771, learning_rate: 0.0002607458, weight_decay: 0.0002402524

epoch time: 2044.7705078125s, input_fired_time: 30.2114276886, output_fired_time: 15.6393556595, propagate_forward_time: 811.8917236328, propagate_backward_time: 1166.4683837891, weight_update_time: 0.6331542134, other_time: 20.5595703125
epoch time: 165.3316192627s, input_fired_time: 2.3384668827, output_fired_time: 0.0000000000, propagate_forward_time: 159.4749450684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5182037354
validation[          , genome  6348] predictions:    5949/  10000 (59.49%), best:    6129/10000 (61.29%), error:     11501.68359, best error:     11075.95508 on epoch:    10, epoch:   13/50, mu: 0.7844663858, learning_rate: 0.0002390219, weight_decay: 0.0002257520

epoch time: 1752.6463623047s, input_fired_time: 28.6436176300, output_fired_time: 13.5189380646, propagate_forward_time: 684.5864257813, propagate_backward_time: 1005.2887573242, weight_update_time: 0.5737587214, other_time: 20.6085815430
epoch time: 157.9005279541s, input_fired_time: 2.2652559280, output_fired_time: 0.0000000000, propagate_forward_time: 151.7472534180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8880157471
validation[          , genome  6348] predictions:    6234/  10000 (62.34%), best:    6234/10000 (62.34%), error:     10956.11426, best error:     10956.11426 on epoch:    14, epoch:   14/50, mu: 0.7981493473, learning_rate: 0.0002191078, weight_decay: 0.0002121268

epoch time: 1966.8308105469s, input_fired_time: 30.0149154663, output_fired_time: 15.0586032867, propagate_forward_time: 775.8761596680, propagate_backward_time: 1124.3127441406, weight_update_time: 0.6395522952, other_time: 21.5683593750
epoch time: 133.9711608887s, input_fired_time: 2.0228700638, output_fired_time: 0.0000000000, propagate_forward_time: 128.5023345947, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4459533691
validation[          , genome  6348] predictions:    5852/  10000 (58.52%), best:    6234/10000 (62.34%), error:     12070.64941, best error:     10956.11426 on epoch:    14, epoch:   15/50, mu: 0.8109213710, learning_rate: 0.0002008529, weight_decay: 0.0001993239

epoch time: 1902.8961181641s, input_fired_time: 29.4150276184, output_fired_time: 14.5410823822, propagate_forward_time: 750.1878051758, propagate_backward_time: 1088.4392089844, weight_update_time: 0.5991209745, other_time: 20.3129882813
epoch time: 142.8783111572s, input_fired_time: 2.1388850212, output_fired_time: 0.0000000000, propagate_forward_time: 137.2839508057, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4554748535
validation[          , genome  6348] predictions:    6187/  10000 (61.87%), best:    6234/10000 (62.34%), error:     11194.91211, best error:     10956.11426 on epoch:    14, epoch:   16/50, mu: 0.8228431344, learning_rate: 0.0001841189, weight_decay: 0.0001872937

epoch time: 1957.6746826172s, input_fired_time: 30.0638923645, output_fired_time: 15.1175222397, propagate_forward_time: 772.0346679688, propagate_backward_time: 1118.4821777344, weight_update_time: 0.6261053085, other_time: 21.9764404297
epoch time: 210.3938293457s, input_fired_time: 2.8155934811, output_fired_time: 0.0000000000, propagate_forward_time: 203.4332122803, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1450195313
validation[          , genome  6348] predictions:    6237/  10000 (62.37%), best:    6234/10000 (62.34%), error:     10965.62402, best error:     10956.11426 on epoch:    14, epoch:   17/50, mu: 0.8339712620, learning_rate: 0.0001687791, weight_decay: 0.0001759896

epoch time: 2024.9455566406s, input_fired_time: 30.5152626038, output_fired_time: 15.6496696472, propagate_forward_time: 798.7875976563, propagate_backward_time: 1157.1702880859, weight_update_time: 0.6464855075, other_time: 22.8227539063
epoch time: 201.7680206299s, input_fired_time: 2.8012623787, output_fired_time: 0.0000000000, propagate_forward_time: 194.1751556396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7915954590
validation[          , genome  6348] predictions:    6270/  10000 (62.70%), best:    6270/10000 (62.70%), error:     10769.56445, best error:     10769.56445 on epoch:    18, epoch:   18/50, mu: 0.8443585634, learning_rate: 0.0001547173, weight_decay: 0.0001653678

epoch time: 1886.3735351563s, input_fired_time: 28.9736881256, output_fired_time: 14.3744039536, propagate_forward_time: 743.3519287109, propagate_backward_time: 1080.3372802734, weight_update_time: 0.5801181793, other_time: 19.3363037109
epoch time: 158.1139831543s, input_fired_time: 2.2264077663, output_fired_time: 0.0000000000, propagate_forward_time: 152.4813079834, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4062652588
validation[          , genome  6348] predictions:    6361/  10000 (63.61%), best:    6361/10000 (63.61%), error:     10713.93848, best error:     10713.93848 on epoch:    19, epoch:   19/50, mu: 0.8540543318, learning_rate: 0.0001418271, weight_decay: 0.0001553870

epoch time: 1827.0662841797s, input_fired_time: 28.6579856873, output_fired_time: 13.8593959808, propagate_forward_time: 717.8201293945, propagate_backward_time: 1047.5319824219, weight_update_time: 0.5686811209, other_time: 19.1967773438
epoch time: 133.6932678223s, input_fired_time: 1.9708093405, output_fired_time: 0.0000000000, propagate_forward_time: 128.5899047852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1325531006
validation[          , genome  6348] predictions:    6354/  10000 (63.54%), best:    6354/10000 (63.54%), error:     10674.44043, best error:     10674.44043 on epoch:    20, epoch:   20/50, mu: 0.8631046414, learning_rate: 0.0001300108, weight_decay: 0.0001460087

epoch time: 1904.3358154297s, input_fired_time: 29.3190212250, output_fired_time: 14.6379776001, propagate_forward_time: 749.3895874023, propagate_backward_time: 1090.2402343750, weight_update_time: 0.5969042778, other_time: 20.7490234375
epoch time: 132.5777740479s, input_fired_time: 1.9788430929, output_fired_time: 0.0000000000, propagate_forward_time: 127.4155960083, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1833419800
validation[          , genome  6348] predictions:    6410/  10000 (64.10%), best:    6410/10000 (64.10%), error:     10533.16602, best error:     10533.16602 on epoch:    21, epoch:   21/50, mu: 0.8715524077, learning_rate: 0.0001191790, weight_decay: 0.0001371963

epoch time: 1897.1488037109s, input_fired_time: 29.2970485687, output_fired_time: 14.6032123566, propagate_forward_time: 747.3721313477, propagate_backward_time: 1085.6552734375, weight_update_time: 0.5952107906, other_time: 20.2211914063
epoch time: 183.6249694824s, input_fired_time: 2.5709240437, output_fired_time: 0.0000000000, propagate_forward_time: 177.3351135254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7189331055
validation[          , genome  6348] predictions:    6335/  10000 (63.35%), best:    6410/10000 (64.10%), error:     10752.95117, best error:     10533.16602 on epoch:    21, epoch:   22/50, mu: 0.8794378042, learning_rate: 0.0001092496, weight_decay: 0.0001289159

epoch time: 1984.6613769531s, input_fired_time: 30.2972164154, output_fired_time: 15.3004283905, propagate_forward_time: 782.7999267578, propagate_backward_time: 1133.9654541016, weight_update_time: 0.6321560740, other_time: 22.2983398438
epoch time: 117.7220001221s, input_fired_time: 1.8929992914, output_fired_time: 0.0000000000, propagate_forward_time: 112.6452713013, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1837310791
validation[          , genome  6348] predictions:    6267/  10000 (62.67%), best:    6410/10000 (64.10%), error:     10816.58105, best error:     10533.16602 on epoch:    21, epoch:   23/50, mu: 0.8867982626, learning_rate: 0.0001001475, weight_decay: 0.0001211351

epoch time: 2095.1972656250s, input_fired_time: 30.3133029938, output_fired_time: 16.0413341522, propagate_forward_time: 830.2567138672, propagate_backward_time: 1198.0592041016, weight_update_time: 0.6351770759, other_time: 20.5268554688
epoch time: 163.7141876221s, input_fired_time: 2.3037228584, output_fired_time: 0.0000000000, propagate_forward_time: 157.4731445313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9373168945
validation[          , genome  6348] predictions:    6446/  10000 (64.46%), best:    6446/10000 (64.46%), error:     10437.00879, best error:     10437.00879 on epoch:    24, epoch:   24/50, mu: 0.8936687112, learning_rate: 0.0000918038, weight_decay: 0.0001138240

epoch time: 2146.2053222656s, input_fired_time: 30.8929214478, output_fired_time: 17.0275058746, propagate_forward_time: 840.9689331055, propagate_backward_time: 1232.5565185547, weight_update_time: 0.6778548360, other_time: 24.7593994141
epoch time: 171.8493804932s, input_fired_time: 2.3873538971, output_fired_time: 0.0000000000, propagate_forward_time: 165.1945495605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2674713135
validation[          , genome  6348] predictions:    6488/  10000 (64.88%), best:    6488/10000 (64.88%), error:     10396.70703, best error:     10396.70703 on epoch:    25, epoch:   25/50, mu: 0.9000817537, learning_rate: 0.0000841552, weight_decay: 0.0001069542

epoch time: 2082.4863281250s, input_fired_time: 30.3900909424, output_fired_time: 16.1592178345, propagate_forward_time: 820.3554687500, propagate_backward_time: 1193.6604003906, weight_update_time: 0.6426516175, other_time: 21.9211425781
epoch time: 168.0661621094s, input_fired_time: 2.3932363987, output_fired_time: 0.0000000000, propagate_forward_time: 162.1205291748, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5523986816
validation[          , genome  6348] predictions:    6528/  10000 (65.28%), best:    6528/10000 (65.28%), error:     10196.70898, best error:     10196.70898 on epoch:    26, epoch:   26/50, mu: 0.9060678482, learning_rate: 0.0000771438, weight_decay: 0.0001004990

epoch time: 2080.6760253906s, input_fired_time: 30.5388393402, output_fired_time: 16.1640243530, propagate_forward_time: 818.3944702148, propagate_backward_time: 1192.3977050781, weight_update_time: 0.6635947824, other_time: 23.1809082031
epoch time: 158.6789703369s, input_fired_time: 2.2667922974, output_fired_time: 0.0000000000, propagate_forward_time: 152.3694915771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0426788330
validation[          , genome  6348] predictions:    6349/  10000 (63.49%), best:    6528/10000 (65.28%), error:     10542.10156, best error:     10196.70898 on epoch:    26, epoch:   27/50, mu: 0.9116554260, learning_rate: 0.0000707166, weight_decay: 0.0000944334

epoch time: 1962.7550048828s, input_fired_time: 29.6407508850, output_fired_time: 15.0676031113, propagate_forward_time: 770.1016235352, propagate_backward_time: 1126.8476562500, weight_update_time: 0.6093407869, other_time: 21.0974121094
epoch time: 152.1919555664s, input_fired_time: 2.2129735947, output_fired_time: 0.0000000000, propagate_forward_time: 145.8939056396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0850830078
validation[          , genome  6348] predictions:    6465/  10000 (64.65%), best:    6528/10000 (65.28%), error:     10310.77832, best error:     10196.70898 on epoch:    26, epoch:   28/50, mu: 0.9168710709, learning_rate: 0.0000648249, weight_decay: 0.0000887339

epoch time: 1929.0362548828s, input_fired_time: 29.5535087585, output_fired_time: 15.1618041992, propagate_forward_time: 755.1294555664, propagate_backward_time: 1106.9645996094, weight_update_time: 0.6140493155, other_time: 22.2270507813
epoch time: 154.3534088135s, input_fired_time: 2.2277035713, output_fired_time: 0.0000000000, propagate_forward_time: 148.3043212891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8213806152
validation[          , genome  6348] predictions:    6471/  10000 (64.71%), best:    6528/10000 (65.28%), error:     10249.51367, best error:     10196.70898 on epoch:    26, epoch:   29/50, mu: 0.9217394590, learning_rate: 0.0000594240, weight_decay: 0.0000833784

epoch time: 2064.6840820313s, input_fired_time: 30.3291873932, output_fired_time: 16.0393257141, propagate_forward_time: 813.5575561523, propagate_backward_time: 1182.6848144531, weight_update_time: 0.6382800937, other_time: 22.0732421875
epoch time: 168.9488067627s, input_fired_time: 2.3525485992, output_fired_time: 0.0000000000, propagate_forward_time: 162.1755218506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4207305908
validation[          , genome  6348] predictions:    6430/  10000 (64.30%), best:    6528/10000 (65.28%), error:     10422.01855, best error:     10196.70898 on epoch:    26, epoch:   30/50, mu: 0.9262837768, learning_rate: 0.0000544731, weight_decay: 0.0000783461

epoch time: 1935.0943603516s, input_fired_time: 29.5862426758, output_fired_time: 15.1722192764, propagate_forward_time: 758.6102905273, propagate_backward_time: 1109.7067871094, weight_update_time: 0.6144568324, other_time: 22.0187988281
epoch time: 152.2769317627s, input_fired_time: 2.2004947662, output_fired_time: 0.0000000000, propagate_forward_time: 145.9551849365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1212463379
validation[          , genome  6348] predictions:    6389/  10000 (63.89%), best:    6528/10000 (65.28%), error:     10489.79980, best error:     10196.70898 on epoch:    26, epoch:   31/50, mu: 0.9305255413, learning_rate: 0.0000499347, weight_decay: 0.0000736175

epoch time: 1982.2613525391s, input_fired_time: 29.8939437866, output_fired_time: 15.6923856735, propagate_forward_time: 775.4353027344, propagate_backward_time: 1137.7814941406, weight_update_time: 0.6384406686, other_time: 23.4582519531
epoch time: 184.5211791992s, input_fired_time: 2.5370414257, output_fired_time: 0.0000000000, propagate_forward_time: 176.7485351563, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2355957031
validation[          , genome  6348] predictions:    6461/  10000 (64.61%), best:    6528/10000 (65.28%), error:     10331.63965, best error:     10196.70898 on epoch:    26, epoch:   32/50, mu: 0.9344848990, learning_rate: 0.0000457744, weight_decay: 0.0000691743

epoch time: 1971.5737304688s, input_fired_time: 29.8528156281, output_fired_time: 15.5717353821, propagate_forward_time: 772.4552612305, propagate_backward_time: 1131.5634765625, weight_update_time: 0.6240102053, other_time: 22.1303710938
epoch time: 150.0845489502s, input_fired_time: 2.2000060081, output_fired_time: 0.0000000000, propagate_forward_time: 143.8416900635, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0428466797
validation[          , genome  6348] predictions:    6499/  10000 (64.99%), best:    6528/10000 (65.28%), error:     10225.51465, best error:     10196.70898 on epoch:    26, epoch:   33/50, mu: 0.9381806850, learning_rate: 0.0000419607, weight_decay: 0.0000649993

epoch time: 1815.2626953125s, input_fired_time: 28.9061260223, output_fired_time: 14.0201988220, propagate_forward_time: 711.6373291016, propagate_backward_time: 1040.1331787109, weight_update_time: 0.5727007389, other_time: 20.5659179688
epoch time: 114.1216888428s, input_fired_time: 1.8307237625, output_fired_time: 0.0000000000, propagate_forward_time: 109.1877975464, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1031646729
validation[          , genome  6348] predictions:    6525/  10000 (65.25%), best:    6528/10000 (65.28%), error:     10206.50879, best error:     10196.70898 on epoch:    26, epoch:   34/50, mu: 0.9416304231, learning_rate: 0.0000384648, weight_decay: 0.0000610763

epoch time: 1954.8222656250s, input_fired_time: 29.7526550293, output_fired_time: 15.2621450424, propagate_forward_time: 765.7827148438, propagate_backward_time: 1122.4808349609, weight_update_time: 0.6125308275, other_time: 21.5439453125
epoch time: 181.0139160156s, input_fired_time: 2.5013298988, output_fired_time: 0.0000000000, propagate_forward_time: 174.3498840332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1627044678
validation[          , genome  6348] predictions:    6558/  10000 (65.58%), best:    6558/10000 (65.58%), error:     10161.51758, best error:     10161.51758 on epoch:    35, epoch:   35/50, mu: 0.9448505044, learning_rate: 0.0000352601, weight_decay: 0.0000573900

epoch time: 2095.1455078125s, input_fired_time: 30.7617778778, output_fired_time: 16.1999073029, propagate_forward_time: 822.4340209961, propagate_backward_time: 1202.6063232422, weight_update_time: 0.6560228467, other_time: 23.1434326172
epoch time: 156.6661834717s, input_fired_time: 2.3304371834, output_fired_time: 0.0000000000, propagate_forward_time: 150.5293731689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8063659668
validation[          , genome  6348] predictions:    6459/  10000 (64.59%), best:    6558/10000 (65.58%), error:     10325.44043, best error:     10161.51758 on epoch:    35, epoch:   36/50, mu: 0.9478562474, learning_rate: 0.0000323224, weight_decay: 0.0000539263

epoch time: 1910.0339355469s, input_fired_time: 29.5389671326, output_fired_time: 14.9459333420, propagate_forward_time: 746.7983398438, propagate_backward_time: 1096.7702636719, weight_update_time: 0.6063673496, other_time: 21.9804687500
epoch time: 158.3492126465s, input_fired_time: 2.2587351799, output_fired_time: 0.0000000000, propagate_forward_time: 152.6208953857, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4695892334
validation[          , genome  6348] predictions:    6460/  10000 (64.60%), best:    6558/10000 (65.58%), error:     10256.97168, best error:     10161.51758 on epoch:    35, epoch:   37/50, mu: 0.9506618977, learning_rate: 0.0000296295, weight_decay: 0.0000506716

epoch time: 1959.3188476563s, input_fired_time: 29.7971973419, output_fired_time: 15.2922163010, propagate_forward_time: 767.2161254883, propagate_backward_time: 1125.2155761719, weight_update_time: 0.6149309874, other_time: 21.7976074219
epoch time: 154.4099121094s, input_fired_time: 2.2369146347, output_fired_time: 0.0000000000, propagate_forward_time: 148.2073669434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9656372070
validation[          , genome  6348] predictions:    6487/  10000 (64.87%), best:    6558/10000 (65.58%), error:     10185.94824, best error:     10161.51758 on epoch:    35, epoch:   38/50, mu: 0.9532807469, learning_rate: 0.0000271609, weight_decay: 0.0000476133

epoch time: 1943.9577636719s, input_fired_time: 29.6971397400, output_fired_time: 15.2239160538, propagate_forward_time: 758.7823486328, propagate_backward_time: 1117.5181884766, weight_update_time: 0.6185188890, other_time: 22.7362060547
epoch time: 172.7299499512s, input_fired_time: 2.4522593021, output_fired_time: 0.0000000000, propagate_forward_time: 166.6490478516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6286468506
validation[          , genome  6348] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:     10100.16113, best error:     10100.16113 on epoch:    39, epoch:   39/50, mu: 0.9557252526, learning_rate: 0.0000248980, weight_decay: 0.0000447396

epoch time: 2020.7813720703s, input_fired_time: 30.2578983307, output_fired_time: 15.6997213364, propagate_forward_time: 792.7504882813, propagate_backward_time: 1159.6113281250, weight_update_time: 0.6324788928, other_time: 22.4619140625
epoch time: 172.8907012939s, input_fired_time: 2.4838962555, output_fired_time: 0.0000000000, propagate_forward_time: 165.8922119141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5145874023
validation[          , genome  6348] predictions:    6592/  10000 (65.92%), best:    6592/10000 (65.92%), error:     10021.66699, best error:     10021.66699 on epoch:    40, epoch:   40/50, mu: 0.9580070376, learning_rate: 0.0000228236, weight_decay: 0.0000420393

epoch time: 2041.9158935547s, input_fired_time: 30.6755409241, output_fired_time: 15.7793464661, propagate_forward_time: 805.0772094727, propagate_backward_time: 1167.5717773438, weight_update_time: 0.6463626623, other_time: 22.8120117188
epoch time: 147.3755035400s, input_fired_time: 2.2866199017, output_fired_time: 0.0000000000, propagate_forward_time: 140.6283111572, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4605712891
validation[          , genome  6348] predictions:    6508/  10000 (65.08%), best:    6592/10000 (65.92%), error:     10085.17090, best error:     10021.66699 on epoch:    40, epoch:   41/50, mu: 0.9601368904, learning_rate: 0.0000209221, weight_decay: 0.0000395021

epoch time: 1956.4422607422s, input_fired_time: 29.8328895569, output_fired_time: 15.4537181854, propagate_forward_time: 762.8787231445, propagate_backward_time: 1126.2048339844, weight_update_time: 0.6188947558, other_time: 22.0720214844
epoch time: 124.6190261841s, input_fired_time: 1.9451410770, output_fired_time: 0.0000000000, propagate_forward_time: 119.4732208252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2006607056
validation[          , genome  6348] predictions:    6589/  10000 (65.89%), best:    6592/10000 (65.92%), error:     10035.96289, best error:     10021.66699 on epoch:    40, epoch:   42/50, mu: 0.9621249437, learning_rate: 0.0000191790, weight_decay: 0.0000371179

epoch time: 2050.1687011719s, input_fired_time: 30.3227481842, output_fired_time: 15.9873361588, propagate_forward_time: 807.4782104492, propagate_backward_time: 1174.7254638672, weight_update_time: 0.6337546706, other_time: 21.6549072266
epoch time: 131.7978210449s, input_fired_time: 2.0675776005, output_fired_time: 0.0000000000, propagate_forward_time: 126.0001449585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7300949097
validation[          , genome  6348] predictions:    6601/  10000 (66.01%), best:    6601/10000 (66.01%), error:     10001.29980, best error:     10001.29980 on epoch:    43, epoch:   43/50, mu: 0.9639806747, learning_rate: 0.0000175811, weight_decay: 0.0000348777

epoch time: 1980.9074707031s, input_fired_time: 29.8573150635, output_fired_time: 15.3136787415, propagate_forward_time: 775.7518310547, propagate_backward_time: 1138.9591064453, weight_update_time: 0.6133943796, other_time: 21.0255126953
epoch time: 149.2272338867s, input_fired_time: 2.1987559795, output_fired_time: 0.0000000000, propagate_forward_time: 143.0426788330, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9857940674
validation[          , genome  6348] predictions:    6547/  10000 (65.47%), best:    6601/10000 (66.01%), error:     10054.87207, best error:     10001.29980 on epoch:    43, epoch:   44/50, mu: 0.9657128453, learning_rate: 0.0000161163, weight_decay: 0.0000327726

epoch time: 1806.1290283203s, input_fired_time: 29.2889652252, output_fired_time: 14.1098241806, propagate_forward_time: 702.8607177734, propagate_backward_time: 1037.8787841797, weight_update_time: 0.5916006565, other_time: 21.9907226563
epoch time: 156.9644775391s, input_fired_time: 2.4262626171, output_fired_time: 0.0000000000, propagate_forward_time: 149.5849151611, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9532928467
validation[          , genome  6348] predictions:    6577/  10000 (65.77%), best:    6601/10000 (66.01%), error:     10090.77832, best error:     10001.29980 on epoch:    43, epoch:   45/50, mu: 0.9673296809, learning_rate: 0.0000147736, weight_decay: 0.0000307946

epoch time: 1987.5526123047s, input_fired_time: 30.0159816742, output_fired_time: 15.8054370880, propagate_forward_time: 773.2044067383, propagate_backward_time: 1145.1372070313, weight_update_time: 0.6402382851, other_time: 23.3896484375
epoch time: 190.3269805908s, input_fired_time: 2.6395435333, output_fired_time: 0.0000000000, propagate_forward_time: 183.0302734375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6571655273
validation[          , genome  6348] predictions:    6599/  10000 (65.99%), best:    6599/10000 (65.99%), error:      9956.28516, best error:      9956.28516 on epoch:    46, epoch:   46/50, mu: 0.9688389301, learning_rate: 0.0000135427, weight_decay: 0.0000289360

epoch time: 1970.4260253906s, input_fired_time: 30.3770923615, output_fired_time: 15.3067626953, propagate_forward_time: 772.1632080078, propagate_backward_time: 1129.0510253906, weight_update_time: 0.6435800791, other_time: 23.5279541016
epoch time: 155.7735290527s, input_fired_time: 2.3445925713, output_fired_time: 0.0000000000, propagate_forward_time: 149.3081359863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1208038330
validation[          , genome  6348] predictions:    6488/  10000 (64.88%), best:    6599/10000 (65.99%), error:     10229.22070, best error:      9956.28516 on epoch:    46, epoch:   47/50, mu: 0.9702476859, learning_rate: 0.0000124144, weight_decay: 0.0000271896

epoch time: 1901.6437988281s, input_fired_time: 29.9649276733, output_fired_time: 14.9518699646, propagate_forward_time: 743.5890502930, propagate_backward_time: 1089.9930419922, weight_update_time: 0.6200600863, other_time: 23.1448974609
epoch time: 147.1231689453s, input_fired_time: 2.2571322918, output_fired_time: 0.0000000000, propagate_forward_time: 140.8168029785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0492401123
validation[          , genome  6348] predictions:    6524/  10000 (65.24%), best:    6599/10000 (65.99%), error:     10145.47754, best error:      9956.28516 on epoch:    46, epoch:   48/50, mu: 0.9715626240, learning_rate: 0.0000113801, weight_decay: 0.0000255486

epoch time: 1851.9611816406s, input_fired_time: 29.4777526855, output_fired_time: 14.4417686462, propagate_forward_time: 720.7909545898, propagate_backward_time: 1065.3303222656, weight_update_time: 0.6046885252, other_time: 21.9204101563
epoch time: 206.9796295166s, input_fired_time: 2.7806737423, output_fired_time: 0.0000000000, propagate_forward_time: 200.3714752197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8274841309
validation[          , genome  6348] predictions:    6626/  10000 (66.26%), best:    6626/10000 (66.26%), error:      9903.25488, best error:      9903.25488 on epoch:    49, epoch:   49/50, mu: 0.9727900624, learning_rate: 0.0000104320, weight_decay: 0.0000240066

epoch time: 2176.7575683594s, input_fired_time: 31.0607070923, output_fired_time: 16.8324069977, propagate_forward_time: 856.8668212891, propagate_backward_time: 1249.5964355469, weight_update_time: 0.6618607044, other_time: 22.4010009766
epoch time: 199.7826538086s, input_fired_time: 2.7047924995, output_fired_time: 0.0000000000, propagate_forward_time: 192.6374511719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4404144287
validation[          , genome  6348] predictions:    6633/  10000 (66.33%), best:    6626/10000 (66.26%), error:      9919.70703, best error:      9903.25488 on epoch:    49, epoch:   50/50, mu: 0.9739357829, learning_rate: 0.0000095628, weight_decay: 0.0000225577

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 719.8112792969s, input_fired_time: 10.7444095612, output_fired_time: 0.0000000000, propagate_forward_time: 691.0056152344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 18.0612792969
best training[          , genome  6348] predictions:   35260/  50000 (70.52%), best:    6626/10000 (66.26%), error:     44677.07813, best error:      9903.25488 on epoch:    49, epoch:   51/50, mu: 0.9750052094, learning_rate: 0.0000087661, weight_decay: 0.0000211962
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 167.5513000488s, input_fired_time: 2.3864977360, output_fired_time: 0.0000000000, propagate_forward_time: 160.9980621338, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1667327881
   testing[          , genome  6348] predictions:    6626/  10000 (66.26%), best:    6626/10000 (66.26%), error:      9903.25488, best error:      9903.25488 on epoch:    49, epoch:   51/50, mu: 0.9750052094, learning_rate: 0.0000087661, weight_decay: 0.0000211962
backpropagation finished successfully!
00:52:15 (2324): called boinc_finish(0)

</stderr_txt>
]]>