Nameexact_genome_1542727803_172_6422_0
Workunit4101318
Created20 Nov 2018, 20:14:40 UTC
Sent20 Nov 2018, 20:15:03 UTC
Report deadline18 Dec 2018, 17:55:53 UTC
Received21 Nov 2018, 6:16:47 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60660
Run time8 hours 48 min 20 sec
CPU time8 hours 33 min 25 sec
Validate stateValid
Credit6,975.84
Device peak FLOPS3.44 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size577.54 MB
Peak swap size597.53 MB
Peak disk usage13.22 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422_0_r438941394_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6422.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.478494
read mu: 0.478494
read mu_delta: 0.944476
read initial_learning_rate: 0.000227653
read learning_rate: 0.000227653
read learning_rate_delta: 0.923624
read initial_weight_decay: 0.000213579
read weight_decay: 0.000213579
read weight_decay_delta: 0.945611
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0897042
read input_dropout_probability: 0.00130021
read hidden_dropout_probability: 0.0860186
read velocity_reset: 1771
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '1 0.765655219554901 -2.24495768547058'
generator_str: '1722951300'
read generator: 1722951300
read generated_by_map:
1 merge_node 1
reading 108 nodes.
reading 455 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 973002952
post shuffle 1: 662913874
epoch time: 1096.51892089844s, input_fired_time: 23.6445293426514, output_fired_time: 8.18271064758301, propagate_forward_time: 401.002197265625, propagate_backward_time: 650.223571777344, weight_update_time: 0.342710226774216, other_time: 13.4658813476563
epoch time: 84.0005569458008s, input_fired_time: 1.48053252696991, output_fired_time: 0, propagate_forward_time: 80.2310180664063, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.28900909423828
validation[          , genome  6422] predictions:    6330/  10000 (63.30%), best:    6330/10000 (63.30%), error:     10811.92871, best error:     10811.92871 on epoch:     0, epoch:    0/25, mu: 0.4784935117, learning_rate: 0.0002276529, weight_decay: 0.0002135793

epoch time: 1093.7611083984s, input_fired_time: 23.6018409729, output_fired_time: 8.1359701157, propagate_forward_time: 399.8723144531, propagate_backward_time: 648.7747192383, weight_update_time: 0.3339545131, other_time: 13.3762817383
epoch time: 84.6723251343s, input_fired_time: 1.4958766699, output_fired_time: 0.0000000000, propagate_forward_time: 80.7961044312, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3803405762
validation[          , genome  6422] predictions:    6309/  10000 (63.09%), best:    6330/10000 (63.30%), error:     10865.29395, best error:     10811.92871 on epoch:     0, epoch:    1/25, mu: 0.5068944693, learning_rate: 0.0002102657, weight_decay: 0.0002019630

epoch time: 1095.6657714844s, input_fired_time: 23.6456413269, output_fired_time: 8.1313371658, propagate_forward_time: 400.6494445801, propagate_backward_time: 649.7880249023, weight_update_time: 0.3341536224, other_time: 13.4513549805
epoch time: 84.1626434326s, input_fired_time: 1.4705166817, output_fired_time: 0.0000000000, propagate_forward_time: 80.4105453491, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2815780640
validation[          , genome  6422] predictions:    6274/  10000 (62.74%), best:    6330/10000 (63.30%), error:     10937.30176, best error:     10811.92871 on epoch:     0, epoch:    2/25, mu: 0.5337184668, learning_rate: 0.0001942065, weight_decay: 0.0001909785

epoch time: 1097.2797851563s, input_fired_time: 23.7039947510, output_fired_time: 8.1364336014, propagate_forward_time: 401.2504577637, propagate_backward_time: 650.5973510742, weight_update_time: 0.3517129719, other_time: 13.5914916992
epoch time: 84.0065612793s, input_fired_time: 1.4909130335, output_fired_time: 0.0000000000, propagate_forward_time: 80.2393493652, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2762985229
validation[          , genome  6422] predictions:    6344/  10000 (63.44%), best:    6344/10000 (63.44%), error:     10749.94629, best error:     10749.94629 on epoch:     3, epoch:    3/25, mu: 0.5590530634, learning_rate: 0.0001793738, weight_decay: 0.0001805915

epoch time: 1095.2452392578s, input_fired_time: 23.6282119751, output_fired_time: 8.1345100403, propagate_forward_time: 400.5918579102, propagate_backward_time: 649.5097045898, weight_update_time: 0.3338821232, other_time: 13.3809814453
epoch time: 84.9039154053s, input_fired_time: 1.4751286507, output_fired_time: 0.0000000000, propagate_forward_time: 81.1239242554, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3048629761
validation[          , genome  6422] predictions:    6203/  10000 (62.03%), best:    6344/10000 (63.44%), error:     11155.19434, best error:     10749.94629 on epoch:     3, epoch:    4/25, mu: 0.5829809904, learning_rate: 0.0001656739, weight_decay: 0.0001707694

epoch time: 1320.1429443359s, input_fired_time: 23.6335906982, output_fired_time: 8.1442852020, propagate_forward_time: 625.1581420898, propagate_backward_time: 649.6447143555, weight_update_time: 0.3358868659, other_time: 13.5622558594
epoch time: 84.7941589355s, input_fired_time: 1.4751273394, output_fired_time: 0.0000000000, propagate_forward_time: 81.0016860962, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3173446655
validation[          , genome  6422] predictions:    6214/  10000 (62.14%), best:    6344/10000 (63.44%), error:     11054.45605, best error:     10749.94629 on epoch:     3, epoch:    5/25, mu: 0.6055803299, learning_rate: 0.0001530204, weight_decay: 0.0001614815

epoch time: 1270.1254882813s, input_fired_time: 23.6180896759, output_fired_time: 8.1089754105, propagate_forward_time: 576.1252441406, propagate_backward_time: 648.8140258789, weight_update_time: 0.3355194032, other_time: 13.4591674805
epoch time: 84.3638839722s, input_fired_time: 1.4745892286, output_fired_time: 0.0000000000, propagate_forward_time: 80.5597839355, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3295135498
validation[          , genome  6422] predictions:    5962/  10000 (59.62%), best:    6344/10000 (63.44%), error:     11679.55176, best error:     10749.94629 on epoch:     3, epoch:    6/25, mu: 0.6269248724, learning_rate: 0.0001413334, weight_decay: 0.0001526988

epoch time: 1091.5007324219s, input_fired_time: 23.5002651215, output_fired_time: 8.1500530243, propagate_forward_time: 399.1680603027, propagate_backward_time: 647.3555908203, weight_update_time: 0.3343333304, other_time: 13.3267822266
epoch time: 84.6270065308s, input_fired_time: 1.4849879742, output_fired_time: 0.0000000000, propagate_forward_time: 80.8624801636, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2795410156
validation[          , genome  6422] predictions:    6239/  10000 (62.39%), best:    6344/10000 (63.44%), error:     11140.57227, best error:     10749.94629 on epoch:     3, epoch:    7/25, mu: 0.6470842957, learning_rate: 0.0001305389, weight_decay: 0.0001443937

epoch time: 1095.0308837891s, input_fired_time: 23.6054782867, output_fired_time: 8.1156311035, propagate_forward_time: 400.5022277832, propagate_backward_time: 649.2569580078, weight_update_time: 0.3356045187, other_time: 13.5506591797
epoch time: 84.2093505859s, input_fired_time: 1.4653599262, output_fired_time: 0.0000000000, propagate_forward_time: 80.4574050903, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2865829468
validation[          , genome  6422] predictions:    5973/  10000 (59.73%), best:    6344/10000 (63.44%), error:     11715.64551, best error:     10749.94629 on epoch:     3, epoch:    8/25, mu: 0.6661244035, learning_rate: 0.0001205689, weight_decay: 0.0001365404

epoch time: 1097.0577392578s, input_fired_time: 23.6276512146, output_fired_time: 8.1252326965, propagate_forward_time: 401.3374023438, propagate_backward_time: 650.4232788086, weight_update_time: 0.3358687162, other_time: 13.5441284180
epoch time: 83.9703521729s, input_fired_time: 1.4712512493, output_fired_time: 0.0000000000, propagate_forward_time: 80.2115325928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2875671387
validation[          , genome  6422] predictions:    5781/  10000 (57.81%), best:    6344/10000 (63.44%), error:     12074.23730, best error:     10749.94629 on epoch:     3, epoch:    9/25, mu: 0.6841073036, learning_rate: 0.0001113603, weight_decay: 0.0001291141

epoch time: 1751.7087402344s, input_fired_time: 23.5260543823, output_fired_time: 8.1462020874, propagate_forward_time: 399.4461669922, propagate_backward_time: 1306.9667968750, weight_update_time: 0.3365502656, other_time: 13.6235351563
epoch time: 85.0157775879s, input_fired_time: 1.4841799736, output_fired_time: 0.0000000000, propagate_forward_time: 81.2049484253, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3266525269
validation[          , genome  6422] predictions:    5755/  10000 (57.55%), best:    6344/10000 (63.44%), error:     12488.41406, best error:     10749.94629 on epoch:     3, epoch:   10/25, mu: 0.7010917068, learning_rate: 0.0001028551, weight_decay: 0.0001220918

epoch time: 1097.6450195313s, input_fired_time: 23.7454547882, output_fired_time: 8.1866722107, propagate_forward_time: 401.3111267090, propagate_backward_time: 650.8015136719, weight_update_time: 0.3453038335, other_time: 13.6002197266
epoch time: 84.8831939697s, input_fired_time: 1.4712264538, output_fired_time: 0.0000000000, propagate_forward_time: 81.0532684326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3586959839
validation[          , genome  6422] predictions:    6011/  10000 (60.11%), best:    6344/10000 (63.44%), error:     11861.01074, best error:     10749.94629 on epoch:     3, epoch:   11/25, mu: 0.7171331048, learning_rate: 0.0000949994, weight_decay: 0.0001154514

epoch time: 1096.6040039063s, input_fired_time: 23.6074714661, output_fired_time: 8.1697139740, propagate_forward_time: 401.0120239258, propagate_backward_time: 650.4395141602, weight_update_time: 0.3362214863, other_time: 13.3753662109
epoch time: 84.3559951782s, input_fired_time: 1.4612559080, output_fired_time: 0.0000000000, propagate_forward_time: 80.6082382202, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2864990234
validation[          , genome  6422] predictions:    6093/  10000 (60.93%), best:    6344/10000 (63.44%), error:     11397.99023, best error:     10749.94629 on epoch:     3, epoch:   12/25, mu: 0.7322837710, learning_rate: 0.0000877438, weight_decay: 0.0001091722

epoch time: 1474.2623291016s, input_fired_time: 23.5461692810, output_fired_time: 8.1660795212, propagate_forward_time: 776.6362915039, propagate_backward_time: 652.5172729492, weight_update_time: 0.3445400000, other_time: 13.3964843750
epoch time: 83.9970550537s, input_fired_time: 1.4674823284, output_fired_time: 0.0000000000, propagate_forward_time: 80.2739791870, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2555923462
validation[          , genome  6422] predictions:    6273/  10000 (62.73%), best:    6344/10000 (63.44%), error:     11080.34570, best error:     10749.94629 on epoch:     3, epoch:   13/25, mu: 0.7465932369, learning_rate: 0.0000810423, weight_decay: 0.0001032345

epoch time: 1696.7711181641s, input_fired_time: 23.5380077362, output_fired_time: 8.1977767944, propagate_forward_time: 1002.9352416992, propagate_backward_time: 648.7247314453, weight_update_time: 0.3332886398, other_time: 13.3754272461
epoch time: 84.3926315308s, input_fired_time: 1.4616669416, output_fired_time: 0.0000000000, propagate_forward_time: 80.6128540039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3181076050
validation[          , genome  6422] predictions:    6076/  10000 (60.76%), best:    6344/10000 (63.44%), error:     11326.34570, best error:     10749.94629 on epoch:     3, epoch:   14/25, mu: 0.7601081729, learning_rate: 0.0000748526, weight_decay: 0.0000976197

epoch time: 1094.5771484375s, input_fired_time: 23.5050601959, output_fired_time: 8.1676721573, propagate_forward_time: 400.6203613281, propagate_backward_time: 648.8551635742, weight_update_time: 0.3401769102, other_time: 13.4288940430
epoch time: 84.4467239380s, input_fired_time: 1.4431159496, output_fired_time: 0.0000000000, propagate_forward_time: 80.6359634399, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3676452637
validation[          , genome  6422] predictions:    5665/  10000 (56.65%), best:    6344/10000 (63.44%), error:     12373.83594, best error:     10749.94629 on epoch:     3, epoch:   15/25, mu: 0.7728726864, learning_rate: 0.0000691356, weight_decay: 0.0000923103

epoch time: 1092.0483398438s, input_fired_time: 23.3620777130, output_fired_time: 8.0971469879, propagate_forward_time: 399.0469970703, propagate_backward_time: 648.1654052734, weight_update_time: 0.3335424364, other_time: 13.3767089844
epoch time: 83.8442840576s, input_fired_time: 1.4462906122, output_fired_time: 0.0000000000, propagate_forward_time: 80.1156616211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2823333740
validation[          , genome  6422] predictions:    6256/  10000 (62.56%), best:    6344/10000 (63.44%), error:     11001.86133, best error:     10749.94629 on epoch:     3, epoch:   16/25, mu: 0.7849285007, learning_rate: 0.0000638553, weight_decay: 0.0000872897

epoch time: 1093.6204833984s, input_fired_time: 23.4516143799, output_fired_time: 8.0993556976, propagate_forward_time: 400.0836181641, propagate_backward_time: 648.5299682617, weight_update_time: 0.3333628476, other_time: 13.4558715820
epoch time: 84.1144027710s, input_fired_time: 1.4538460970, output_fired_time: 0.0000000000, propagate_forward_time: 80.3369522095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3236007690
validation[          , genome  6422] predictions:    6033/  10000 (60.33%), best:    6344/10000 (63.44%), error:     11481.77441, best error:     10749.94629 on epoch:     3, epoch:   17/25, mu: 0.7963148952, learning_rate: 0.0000589783, weight_decay: 0.0000825421

epoch time: 2051.7832031250s, input_fired_time: 23.5124931335, output_fired_time: 8.1261472702, propagate_forward_time: 1139.9260253906, propagate_backward_time: 866.8710937500, weight_update_time: 0.3347012103, other_time: 13.3475341797
epoch time: 84.0205154419s, input_fired_time: 1.4510262012, output_fired_time: 0.0000000000, propagate_forward_time: 80.2721176147, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2973709106
validation[          , genome  6422] predictions:    5924/  10000 (59.24%), best:    6344/10000 (63.44%), error:     11628.46680, best error:     10749.94629 on epoch:     3, epoch:   18/25, mu: 0.8070690632, learning_rate: 0.0000544738, weight_decay: 0.0000780528

epoch time: 1092.3703613281s, input_fired_time: 23.5009040833, output_fired_time: 8.1677865982, propagate_forward_time: 399.4384765625, propagate_backward_time: 647.8356323242, weight_update_time: 0.3392283916, other_time: 13.4275512695
epoch time: 83.8892745972s, input_fired_time: 1.4368394613, output_fired_time: 0.0000000000, propagate_forward_time: 80.1730575562, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2793807983
validation[          , genome  6422] predictions:    6063/  10000 (60.63%), best:    6344/10000 (63.44%), error:     11308.23535, best error:     10749.94629 on epoch:     3, epoch:   19/25, mu: 0.8172261119, learning_rate: 0.0000503133, weight_decay: 0.0000738076

epoch time: 1087.6245117188s, input_fired_time: 23.3196296692, output_fired_time: 8.0808753967, propagate_forward_time: 397.7431030273, propagate_backward_time: 645.2583618164, weight_update_time: 0.3329946399, other_time: 13.2225341797
epoch time: 83.9721069336s, input_fired_time: 1.4411357641, output_fired_time: 0.0000000000, propagate_forward_time: 80.2433090210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2876586914
validation[          , genome  6422] predictions:    6108/  10000 (61.08%), best:    6344/10000 (63.44%), error:     11383.66602, best error:     10749.94629 on epoch:     3, epoch:   20/25, mu: 0.8268192410, learning_rate: 0.0000464706, weight_decay: 0.0000697933

epoch time: 1202.9968261719s, input_fired_time: 25.9505805969, output_fired_time: 9.0258903503, propagate_forward_time: 441.0551757813, propagate_backward_time: 711.7790527344, weight_update_time: 0.3677220941, other_time: 15.1861572266
epoch time: 111.4419097900s, input_fired_time: 2.1107039452, output_fired_time: 0.0000000000, propagate_forward_time: 106.1599578857, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1712493896
validation[          , genome  6422] predictions:    6083/  10000 (60.83%), best:    6344/10000 (63.44%), error:     11272.34668, best error:     10749.94629 on epoch:     3, epoch:   21/25, mu: 0.8358796835, learning_rate: 0.0000429214, weight_decay: 0.0000659974

epoch time: 1313.9665527344s, input_fired_time: 28.7062129974, output_fired_time: 9.6054067612, propagate_forward_time: 482.0579528809, propagate_backward_time: 776.7830810547, weight_update_time: 0.3839426935, other_time: 16.8139648438
epoch time: 107.7239990234s, input_fired_time: 1.8937853575, output_fired_time: 0.0000000000, propagate_forward_time: 102.8457870483, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9844284058
validation[          , genome  6422] predictions:    6286/  10000 (62.86%), best:    6344/10000 (63.44%), error:     10799.42773, best error:     10749.94629 on epoch:     3, epoch:   22/25, mu: 0.8444370627, learning_rate: 0.0000396432, weight_decay: 0.0000624079

epoch time: 1222.0253906250s, input_fired_time: 26.7834892273, output_fired_time: 9.0762929916, propagate_forward_time: 446.5295715332, propagate_backward_time: 723.9358520508, weight_update_time: 0.4093966484, other_time: 15.7002563477
epoch time: 106.9772033691s, input_fired_time: 1.9878461361, output_fired_time: 0.0000000000, propagate_forward_time: 101.9935607910, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9957962036
validation[          , genome  6422] predictions:    6257/  10000 (62.57%), best:    6344/10000 (63.44%), error:     10836.36816, best error:     10749.94629 on epoch:     3, epoch:   23/25, mu: 0.8525193334, learning_rate: 0.0000366154, weight_decay: 0.0000590136

epoch time: 1354.1264648438s, input_fired_time: 23.6573181152, output_fired_time: 8.1927165985, propagate_forward_time: 401.9533081055, propagate_backward_time: 906.6063842773, weight_update_time: 0.3321610689, other_time: 13.7166748047
epoch time: 83.8074722290s, input_fired_time: 1.4454009533, output_fired_time: 0.0000000000, propagate_forward_time: 80.0628280640, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2992401123
validation[          , genome  6422] predictions:    6323/  10000 (63.23%), best:    6344/10000 (63.44%), error:     10759.66992, best error:     10749.94629 on epoch:     3, epoch:   24/25, mu: 0.8601528406, learning_rate: 0.0000338189, weight_decay: 0.0000558039

epoch time: 1091.6578369141s, input_fired_time: 23.5569553375, output_fired_time: 8.1767539978, propagate_forward_time: 399.5008239746, propagate_backward_time: 646.7145385742, weight_update_time: 0.3351576328, other_time: 13.7086791992
epoch time: 83.8226699829s, input_fired_time: 1.4576185942, output_fired_time: 0.0000000000, propagate_forward_time: 80.0679321289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2971191406
validation[          , genome  6422] predictions:    6384/  10000 (63.84%), best:    6384/10000 (63.84%), error:     10630.51953, best error:     10630.51953 on epoch:    25, epoch:   25/25, mu: 0.8673624992, learning_rate: 0.0000312359, weight_decay: 0.0000527688

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 582.5853881836s, input_fired_time: 7.3248205185, output_fired_time: 0.0000000000, propagate_forward_time: 563.2459106445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.0146484375
best training[          , genome  6422] predictions:   34297/  50000 (68.59%), best:    6384/10000 (63.84%), error:     48368.65234, best error:     10630.51953 on epoch:    25, epoch:   26/25, mu: 0.8741718531, learning_rate: 0.0000288503, weight_decay: 0.0000498988
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 85.5000000000s, input_fired_time: 1.4966326952, output_fired_time: 0.0000000000, propagate_forward_time: 81.5487213135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4546432495
   testing[          , genome  6422] predictions:    6384/  10000 (63.84%), best:    6384/10000 (63.84%), error:     10630.51953, best error:     10630.51953 on epoch:    25, epoch:   26/25, mu: 0.8741718531, learning_rate: 0.0000288503, weight_decay: 0.0000498988
backpropagation finished successfully!
01:01:51 (9744): called boinc_finish(0)

</stderr_txt>
]]>