Nameexact_genome_1542727803_172_6422_1
Workunit4101318
Created20 Nov 2018, 20:14:40 UTC
Sent20 Nov 2018, 20:15:27 UTC
Report deadline18 Dec 2018, 17:56:17 UTC
Received21 Nov 2018, 18:22:35 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID83345
Run time10 hours 49 min 30 sec
CPU time10 hours 46 min 33 sec
Validate stateValid
Credit6,975.84
Device peak FLOPS3.66 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size568.92 MB
Peak swap size597.50 MB
Peak disk usage13.21 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422_1_r1451725642_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.478494
read mu: 0.478494
read mu_delta: 0.944476
read initial_learning_rate: 0.000227653
read learning_rate: 0.000227653
read learning_rate_delta: 0.923624
read initial_weight_decay: 0.000213579
read weight_decay: 0.000213579
read weight_decay_delta: 0.945611
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0897042
read input_dropout_probability: 0.00130021
read hidden_dropout_probability: 0.0860186
read velocity_reset: 1771
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '1 0.765655219554901 -2.24495768547058'
generator_str: '1722951300'
read generator: 1722951300
read generated_by_map:
1 merge_node 1
reading 108 nodes.
reading 455 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 973002952
post shuffle 1: 662913874
epoch time: 1357.19030761719s, input_fired_time: 27.6020412445068, output_fired_time: 10.9459314346313, propagate_forward_time: 524.650146484375, propagate_backward_time: 769.717468261719, weight_update_time: 0.596602916717529, other_time: 24.2747192382813
epoch time: 111.976905822754s, input_fired_time: 1.72870481014252, output_fired_time: 0, propagate_forward_time: 105.912124633789, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.33607482910156
validation[          , genome  6422] predictions:    6330/  10000 (63.30%), best:    6330/10000 (63.30%), error:     10811.92871, best error:     10811.92871 on epoch:     0, epoch:    0/25, mu: 0.4784935117, learning_rate: 0.0002276529, weight_decay: 0.0002135793

epoch time: 1359.1677246094s, input_fired_time: 27.7192573547, output_fired_time: 11.0084095001, propagate_forward_time: 525.7435302734, propagate_backward_time: 770.3087158203, weight_update_time: 0.6030583382, other_time: 24.3878173828
epoch time: 111.8415298462s, input_fired_time: 1.7505184412, output_fired_time: 0.0000000000, propagate_forward_time: 105.7766265869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3143844604
validation[          , genome  6422] predictions:    6309/  10000 (63.09%), best:    6330/10000 (63.30%), error:     10865.29395, best error:     10811.92871 on epoch:     0, epoch:    1/25, mu: 0.5068944693, learning_rate: 0.0002102657, weight_decay: 0.0002019630

epoch time: 1377.8658447266s, input_fired_time: 28.1566181183, output_fired_time: 11.1899700165, propagate_forward_time: 532.0773925781, propagate_backward_time: 781.6748657227, weight_update_time: 0.6189972758, other_time: 24.7670288086
epoch time: 112.4913177490s, input_fired_time: 1.7502145767, output_fired_time: 0.0000000000, propagate_forward_time: 106.3786544800, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3624496460
validation[          , genome  6422] predictions:    6274/  10000 (62.74%), best:    6330/10000 (63.30%), error:     10937.30176, best error:     10811.92871 on epoch:     0, epoch:    2/25, mu: 0.5337184668, learning_rate: 0.0001942065, weight_decay: 0.0001909785

epoch time: 1357.4625244141s, input_fired_time: 27.7537269592, output_fired_time: 11.0236978531, propagate_forward_time: 525.1498413086, propagate_backward_time: 769.0704345703, weight_update_time: 0.6039481163, other_time: 24.4647827148
epoch time: 112.1498489380s, input_fired_time: 1.7369648218, output_fired_time: 0.0000000000, propagate_forward_time: 106.0724029541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3404846191
validation[          , genome  6422] predictions:    6344/  10000 (63.44%), best:    6344/10000 (63.44%), error:     10749.94629, best error:     10749.94629 on epoch:     3, epoch:    3/25, mu: 0.5590530634, learning_rate: 0.0001793738, weight_decay: 0.0001805915

epoch time: 1382.0565185547s, input_fired_time: 27.7413730621, output_fired_time: 11.0508756638, propagate_forward_time: 546.9145507813, propagate_backward_time: 771.8314208984, weight_update_time: 0.6044769287, other_time: 24.5183105469
epoch time: 111.9086608887s, input_fired_time: 1.7318336964, output_fired_time: 0.0000000000, propagate_forward_time: 105.8460388184, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3307876587
validation[          , genome  6422] predictions:    6203/  10000 (62.03%), best:    6344/10000 (63.44%), error:     11155.19434, best error:     10749.94629 on epoch:     3, epoch:    4/25, mu: 0.5829809904, learning_rate: 0.0001656739, weight_decay: 0.0001707694

epoch time: 1358.0552978516s, input_fired_time: 27.6951675415, output_fired_time: 11.0192794800, propagate_forward_time: 525.7941284180, propagate_backward_time: 769.0809936523, weight_update_time: 0.6042686105, other_time: 24.4656982422
epoch time: 112.3646163940s, input_fired_time: 1.7339453697, output_fired_time: 0.0000000000, propagate_forward_time: 106.2790374756, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3516311646
validation[          , genome  6422] predictions:    6214/  10000 (62.14%), best:    6344/10000 (63.44%), error:     11054.45605, best error:     10749.94629 on epoch:     3, epoch:    5/25, mu: 0.6055803299, learning_rate: 0.0001530204, weight_decay: 0.0001614815

epoch time: 1356.8269042969s, input_fired_time: 27.6574440002, output_fired_time: 11.0230112076, propagate_forward_time: 525.1199951172, propagate_backward_time: 768.5794677734, weight_update_time: 0.6062340140, other_time: 24.4468994141
epoch time: 111.8407363892s, input_fired_time: 1.7692003250, output_fired_time: 0.0000000000, propagate_forward_time: 105.7412414551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3302917480
validation[          , genome  6422] predictions:    5962/  10000 (59.62%), best:    6344/10000 (63.44%), error:     11679.55176, best error:     10749.94629 on epoch:     3, epoch:    6/25, mu: 0.6269248724, learning_rate: 0.0001413334, weight_decay: 0.0001526988

epoch time: 1357.5999755859s, input_fired_time: 27.6200389862, output_fired_time: 10.9725284576, propagate_forward_time: 525.4902954102, propagate_backward_time: 769.3787231445, weight_update_time: 0.5987066031, other_time: 24.1384277344
epoch time: 124.9780120850s, input_fired_time: 1.7126271725, output_fired_time: 0.0000000000, propagate_forward_time: 118.8916168213, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3737716675
validation[          , genome  6422] predictions:    6239/  10000 (62.39%), best:    6344/10000 (63.44%), error:     11140.57227, best error:     10749.94629 on epoch:     3, epoch:    7/25, mu: 0.6470842957, learning_rate: 0.0001305389, weight_decay: 0.0001443937

epoch time: 1356.6904296875s, input_fired_time: 27.5018787384, output_fired_time: 10.9212398529, propagate_forward_time: 525.0072021484, propagate_backward_time: 769.2614135742, weight_update_time: 0.5990365148, other_time: 23.9987182617
epoch time: 111.6650466919s, input_fired_time: 1.7055242062, output_fired_time: 0.0000000000, propagate_forward_time: 105.7287979126, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2307281494
validation[          , genome  6422] predictions:    5973/  10000 (59.73%), best:    6344/10000 (63.44%), error:     11715.64551, best error:     10749.94629 on epoch:     3, epoch:    8/25, mu: 0.6661244035, learning_rate: 0.0001205689, weight_decay: 0.0001365404

epoch time: 1375.7386474609s, input_fired_time: 27.6171722412, output_fired_time: 10.9630413055, propagate_forward_time: 539.4363403320, propagate_backward_time: 773.5572509766, weight_update_time: 0.6033345461, other_time: 24.1648559570
epoch time: 112.1658096313s, input_fired_time: 1.7181998491, output_fired_time: 0.0000000000, propagate_forward_time: 106.2110519409, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2365570068
validation[          , genome  6422] predictions:    5781/  10000 (57.81%), best:    6344/10000 (63.44%), error:     12074.23730, best error:     10749.94629 on epoch:     3, epoch:    9/25, mu: 0.6841073036, learning_rate: 0.0001113603, weight_decay: 0.0001291141

epoch time: 1356.3769531250s, input_fired_time: 27.5941848755, output_fired_time: 10.9123439789, propagate_forward_time: 524.7171630859, propagate_backward_time: 769.2250976563, weight_update_time: 0.5953924656, other_time: 23.9281005859
epoch time: 112.1089935303s, input_fired_time: 1.7018768787, output_fired_time: 0.0000000000, propagate_forward_time: 106.1078643799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2992553711
validation[          , genome  6422] predictions:    5755/  10000 (57.55%), best:    6344/10000 (63.44%), error:     12488.41406, best error:     10749.94629 on epoch:     3, epoch:   10/25, mu: 0.7010917068, learning_rate: 0.0001028551, weight_decay: 0.0001220918

epoch time: 1358.2788085938s, input_fired_time: 27.5399761200, output_fired_time: 10.9259881973, propagate_forward_time: 525.1635131836, propagate_backward_time: 770.5656738281, weight_update_time: 0.6000248790, other_time: 24.0836791992
epoch time: 111.9439392090s, input_fired_time: 1.7017530203, output_fired_time: 0.0000000000, propagate_forward_time: 106.0076751709, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2345123291
validation[          , genome  6422] predictions:    6011/  10000 (60.11%), best:    6344/10000 (63.44%), error:     11861.01074, best error:     10749.94629 on epoch:     3, epoch:   11/25, mu: 0.7171331048, learning_rate: 0.0000949994, weight_decay: 0.0001154514

epoch time: 1370.7703857422s, input_fired_time: 27.6184253693, output_fired_time: 10.9846162796, propagate_forward_time: 536.4912109375, propagate_backward_time: 771.6597290039, weight_update_time: 0.5964310169, other_time: 24.0164184570
epoch time: 111.7975082397s, input_fired_time: 1.7104848623, output_fired_time: 0.0000000000, propagate_forward_time: 106.1717910767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9152297974
validation[          , genome  6422] predictions:    6093/  10000 (60.93%), best:    6344/10000 (63.44%), error:     11397.99023, best error:     10749.94629 on epoch:     3, epoch:   12/25, mu: 0.7322837710, learning_rate: 0.0000877438, weight_decay: 0.0001091722

epoch time: 1355.1517333984s, input_fired_time: 27.5716609955, output_fired_time: 10.9219779968, propagate_forward_time: 524.9758911133, propagate_backward_time: 769.5160522461, weight_update_time: 0.6065448523, other_time: 22.1661376953
epoch time: 111.4907913208s, input_fired_time: 1.7137169838, output_fired_time: 0.0000000000, propagate_forward_time: 105.9202728271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8568038940
validation[          , genome  6422] predictions:    6273/  10000 (62.73%), best:    6344/10000 (63.44%), error:     11080.34570, best error:     10749.94629 on epoch:     3, epoch:   13/25, mu: 0.7465932369, learning_rate: 0.0000810423, weight_decay: 0.0001032345

epoch time: 1355.0833740234s, input_fired_time: 27.5993309021, output_fired_time: 10.9043302536, propagate_forward_time: 525.2090454102, propagate_backward_time: 769.2191162109, weight_update_time: 0.6048635244, other_time: 22.1515502930
epoch time: 111.3940277100s, input_fired_time: 1.7069188356, output_fired_time: 0.0000000000, propagate_forward_time: 105.8431091309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8440017700
validation[          , genome  6422] predictions:    6076/  10000 (60.76%), best:    6344/10000 (63.44%), error:     11326.34570, best error:     10749.94629 on epoch:     3, epoch:   14/25, mu: 0.7601081729, learning_rate: 0.0000748526, weight_decay: 0.0000976197

epoch time: 1356.9117431641s, input_fired_time: 27.5497207642, output_fired_time: 10.9141511917, propagate_forward_time: 525.5282592773, propagate_backward_time: 770.7291259766, weight_update_time: 0.5989701152, other_time: 22.1904907227
epoch time: 111.2637786865s, input_fired_time: 1.7179042101, output_fired_time: 0.0000000000, propagate_forward_time: 105.6743469238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8715286255
validation[          , genome  6422] predictions:    5665/  10000 (56.65%), best:    6344/10000 (63.44%), error:     12373.83594, best error:     10749.94629 on epoch:     3, epoch:   15/25, mu: 0.7728726864, learning_rate: 0.0000691356, weight_decay: 0.0000923103

epoch time: 1354.8441162109s, input_fired_time: 27.5550289154, output_fired_time: 10.9363012314, propagate_forward_time: 524.8717041016, propagate_backward_time: 769.3987426758, weight_update_time: 0.6029805541, other_time: 22.0823364258
epoch time: 111.3210754395s, input_fired_time: 1.7056603432, output_fired_time: 0.0000000000, propagate_forward_time: 105.7309036255, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8845138550
validation[          , genome  6422] predictions:    6256/  10000 (62.56%), best:    6344/10000 (63.44%), error:     11001.86133, best error:     10749.94629 on epoch:     3, epoch:   16/25, mu: 0.7849285007, learning_rate: 0.0000638553, weight_decay: 0.0000872897

epoch time: 1354.7258300781s, input_fired_time: 27.6538848877, output_fired_time: 10.9541273117, propagate_forward_time: 525.0645141602, propagate_backward_time: 768.8775634766, weight_update_time: 0.6021409631, other_time: 22.1757202148
epoch time: 111.2595443726s, input_fired_time: 1.6988762617, output_fired_time: 0.0000000000, propagate_forward_time: 105.6961441040, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8645248413
validation[          , genome  6422] predictions:    6033/  10000 (60.33%), best:    6344/10000 (63.44%), error:     11481.77441, best error:     10749.94629 on epoch:     3, epoch:   17/25, mu: 0.7963148952, learning_rate: 0.0000589783, weight_decay: 0.0000825421

epoch time: 1355.7036132813s, input_fired_time: 27.5449905396, output_fired_time: 10.9056615829, propagate_forward_time: 524.9973754883, propagate_backward_time: 770.0329589844, weight_update_time: 0.6088657379, other_time: 22.2225952148
epoch time: 111.2966156006s, input_fired_time: 1.7288686037, output_fired_time: 0.0000000000, propagate_forward_time: 105.7013549805, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8663940430
validation[          , genome  6422] predictions:    5924/  10000 (59.24%), best:    6344/10000 (63.44%), error:     11628.46680, best error:     10749.94629 on epoch:     3, epoch:   18/25, mu: 0.8070690632, learning_rate: 0.0000544738, weight_decay: 0.0000780528

epoch time: 1354.9411621094s, input_fired_time: 27.6054115295, output_fired_time: 10.8985300064, propagate_forward_time: 525.4518432617, propagate_backward_time: 768.8467407227, weight_update_time: 0.6012394428, other_time: 22.1385498047
epoch time: 111.2551956177s, input_fired_time: 1.7062593699, output_fired_time: 0.0000000000, propagate_forward_time: 105.6988830566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8500518799
validation[          , genome  6422] predictions:    6063/  10000 (60.63%), best:    6344/10000 (63.44%), error:     11308.23535, best error:     10749.94629 on epoch:     3, epoch:   19/25, mu: 0.8172261119, learning_rate: 0.0000503133, weight_decay: 0.0000738076

epoch time: 1369.2152099609s, input_fired_time: 27.7350711823, output_fired_time: 10.9794359207, propagate_forward_time: 526.3019409180, propagate_backward_time: 781.9603271484, weight_update_time: 0.6011750698, other_time: 22.2383422852
epoch time: 111.3043975830s, input_fired_time: 1.7005258799, output_fired_time: 0.0000000000, propagate_forward_time: 105.7158203125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8880538940
validation[          , genome  6422] predictions:    6108/  10000 (61.08%), best:    6344/10000 (63.44%), error:     11383.66602, best error:     10749.94629 on epoch:     3, epoch:   20/25, mu: 0.8268192410, learning_rate: 0.0000464706, weight_decay: 0.0000697933

epoch time: 1377.5971679688s, input_fired_time: 27.9711723328, output_fired_time: 11.0154352188, propagate_forward_time: 529.8501586914, propagate_backward_time: 786.0070800781, weight_update_time: 0.6061110497, other_time: 22.7533569336
epoch time: 111.6340026855s, input_fired_time: 1.7117993832, output_fired_time: 0.0000000000, propagate_forward_time: 105.9725494385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9496536255
validation[          , genome  6422] predictions:    6083/  10000 (60.83%), best:    6344/10000 (63.44%), error:     11272.34668, best error:     10749.94629 on epoch:     3, epoch:   21/25, mu: 0.8358796835, learning_rate: 0.0000429214, weight_decay: 0.0000659974

epoch time: 1357.1350097656s, input_fired_time: 27.6639995575, output_fired_time: 10.9935169220, propagate_forward_time: 525.6390991211, propagate_backward_time: 770.2507324219, weight_update_time: 0.6031590104, other_time: 22.5877075195
epoch time: 111.4916763306s, input_fired_time: 1.7275580168, output_fired_time: 0.0000000000, propagate_forward_time: 105.8695373535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8945846558
validation[          , genome  6422] predictions:    6286/  10000 (62.86%), best:    6344/10000 (63.44%), error:     10799.42773, best error:     10749.94629 on epoch:     3, epoch:   22/25, mu: 0.8444370627, learning_rate: 0.0000396432, weight_decay: 0.0000624079

epoch time: 1355.9125976563s, input_fired_time: 27.5452613831, output_fired_time: 10.9430027008, propagate_forward_time: 525.2540893555, propagate_backward_time: 769.9904785156, weight_update_time: 0.6005113721, other_time: 22.1797485352
epoch time: 112.1460800171s, input_fired_time: 1.8137332201, output_fired_time: 0.0000000000, propagate_forward_time: 106.4502258301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8821182251
validation[          , genome  6422] predictions:    6257/  10000 (62.57%), best:    6344/10000 (63.44%), error:     10836.36816, best error:     10749.94629 on epoch:     3, epoch:   23/25, mu: 0.8525193334, learning_rate: 0.0000366154, weight_decay: 0.0000590136

epoch time: 1355.4637451172s, input_fired_time: 27.5656375885, output_fired_time: 10.8747329712, propagate_forward_time: 524.3107299805, propagate_backward_time: 770.5353393555, weight_update_time: 0.6053561568, other_time: 22.1772460938
epoch time: 111.4771194458s, input_fired_time: 1.7057646513, output_fired_time: 0.0000000000, propagate_forward_time: 105.8647308350, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9066238403
validation[          , genome  6422] predictions:    6323/  10000 (63.23%), best:    6344/10000 (63.44%), error:     10759.66992, best error:     10749.94629 on epoch:     3, epoch:   24/25, mu: 0.8601528406, learning_rate: 0.0000338189, weight_decay: 0.0000558039

epoch time: 1355.1270751953s, input_fired_time: 27.5800094604, output_fired_time: 10.9039821625, propagate_forward_time: 524.9373779297, propagate_backward_time: 769.3626098633, weight_update_time: 0.6134779453, other_time: 22.3432006836
epoch time: 111.5139312744s, input_fired_time: 1.7449805737, output_fired_time: 0.0000000000, propagate_forward_time: 105.8670806885, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9018707275
validation[          , genome  6422] predictions:    6384/  10000 (63.84%), best:    6384/10000 (63.84%), error:     10630.51953, best error:     10630.51953 on epoch:    25, epoch:   25/25, mu: 0.8673624992, learning_rate: 0.0000312359, weight_decay: 0.0000527688

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 553.4011840820s, input_fired_time: 8.5019817352, output_fired_time: 0.0000000000, propagate_forward_time: 525.3519897461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.5472412109
best training[          , genome  6422] predictions:   34297/  50000 (68.59%), best:    6384/10000 (63.84%), error:     48368.65234, best error:     10630.51953 on epoch:    25, epoch:   26/25, mu: 0.8741718531, learning_rate: 0.0000288503, weight_decay: 0.0000498988
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 111.3756942749s, input_fired_time: 1.7137845755, output_fired_time: 0.0000000000, propagate_forward_time: 105.7066345215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9552764893
   testing[          , genome  6422] predictions:    6384/  10000 (63.84%), best:    6384/10000 (63.84%), error:     10630.51953, best error:     10630.51953 on epoch:    25, epoch:   26/25, mu: 0.8741718531, learning_rate: 0.0000288503, weight_decay: 0.0000498988
backpropagation finished successfully!
13:22:10 (4908): called boinc_finish(0)

</stderr_txt>
]]>