Nameexact_genome_1542727803_173_6422_1
Workunit4101314
Created20 Nov 2018, 20:14:19 UTC
Sent20 Nov 2018, 20:15:03 UTC
Report deadline29 Dec 2018, 21:12:43 UTC
Received21 Nov 2018, 13:03:50 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60660
Run time12 hours 9 min 26 sec
CPU time11 hours 47 min 45 sec
Validate stateValid
Credit9,760.01
Device peak FLOPS3.44 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size605.29 MB
Peak swap size625.20 MB
Peak disk usage28.33 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422_1_r2015903164_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549852
read mu: 0.549852
read mu_delta: 0.969923
read initial_learning_rate: 0.00161364
read learning_rate: 0.00161364
read learning_rate_delta: 0.915733
read initial_weight_decay: 0.000322808
read weight_decay: 0.000322808
read weight_decay_delta: 0.91574
read batch_size: 122
read epsilon: 1e-07
read alpha: 0.125314
read input_dropout_probability: 0.00168037
read hidden_dropout_probability: 0.0658385
read velocity_reset: 1812
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.647515892982483 1.24656867980957'
generator_str: '700611225'
read generator: 700611225
read generated_by_map:
1 split_edge 1
reading 113 nodes.
reading 466 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 520022074
post shuffle 1: 1938633477
epoch time: 1523.72839355469s, input_fired_time: 24.3371448516846, output_fired_time: 8.85114479064941, propagate_forward_time: 562.649291992188, propagate_backward_time: 913.345336914063, weight_update_time: 0.314778834581375, other_time: 14.54541015625
epoch time: 116.055763244629s, input_fired_time: 1.46436822414398, output_fired_time: 0, propagate_forward_time: 112.066200256348, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.52519226074219
validation[          , genome  6422] predictions:    5668/  10000 (56.68%), best:    5668/10000 (56.68%), error:     12504.71094, best error:     12504.71094 on epoch:     0, epoch:    0/25, mu: 0.5498522520, learning_rate: 0.0016136372, weight_decay: 0.0003228079

epoch time: 1522.6805419922s, input_fired_time: 24.4519882202, output_fired_time: 8.8185510635, propagate_forward_time: 562.2485351563, propagate_backward_time: 912.7670898438, weight_update_time: 0.3166734576, other_time: 14.3942871094
epoch time: 116.2282714844s, input_fired_time: 1.4685026407, output_fired_time: 0.0000000000, propagate_forward_time: 112.2875671387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4721984863
validation[          , genome  6422] predictions:    5990/  10000 (59.90%), best:    5990/10000 (59.90%), error:     11566.64844, best error:     11566.64844 on epoch:     1, epoch:    1/25, mu: 0.5630903840, learning_rate: 0.0014776600, weight_decay: 0.0002956082

epoch time: 1519.8784179688s, input_fired_time: 24.4298095703, output_fired_time: 8.7386379242, propagate_forward_time: 561.1326293945, propagate_backward_time: 911.2199707031, weight_update_time: 0.3148006499, other_time: 14.3573608398
epoch time: 115.9032287598s, input_fired_time: 1.4928319454, output_fired_time: 0.0000000000, propagate_forward_time: 111.9217910767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4886093140
validation[          , genome  6422] predictions:    6070/  10000 (60.70%), best:    6070/10000 (60.70%), error:     11113.66797, best error:     11113.66797 on epoch:     2, epoch:    2/25, mu: 0.5759303570, learning_rate: 0.0013531413, weight_decay: 0.0002707004

epoch time: 1524.5550537109s, input_fired_time: 24.6135406494, output_fired_time: 8.7641410828, propagate_forward_time: 563.0291137695, propagate_backward_time: 913.7018432617, weight_update_time: 0.3151921034, other_time: 14.4464111328
epoch time: 340.2646484375s, input_fired_time: 1.4923093319, output_fired_time: 0.0000000000, propagate_forward_time: 336.2727661133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4995727539
validation[          , genome  6422] predictions:    6040/  10000 (60.40%), best:    6070/10000 (60.70%), error:     11406.20703, best error:     11113.66797 on epoch:     2, epoch:    3/25, mu: 0.5883841515, learning_rate: 0.0012391155, weight_decay: 0.0002478913

epoch time: 1696.9833984375s, input_fired_time: 24.5624847412, output_fired_time: 8.7627611160, propagate_forward_time: 561.7157592773, propagate_backward_time: 1087.5433349609, weight_update_time: 0.3140096664, other_time: 14.3990478516
epoch time: 116.6166305542s, input_fired_time: 1.5069948435, output_fired_time: 0.0000000000, propagate_forward_time: 112.6125259399, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4971084595
validation[          , genome  6422] predictions:    6147/  10000 (61.47%), best:    6147/10000 (61.47%), error:     11036.12695, best error:     11036.12695 on epoch:     4, epoch:    4/25, mu: 0.6004633904, learning_rate: 0.0011346983, weight_decay: 0.0002270041

epoch time: 1523.1722412109s, input_fired_time: 24.4986667633, output_fired_time: 8.7616252899, propagate_forward_time: 562.3068847656, propagate_backward_time: 913.1904296875, weight_update_time: 0.3134760559, other_time: 14.4146728516
epoch time: 116.4798812866s, input_fired_time: 1.5116782188, output_fired_time: 0.0000000000, propagate_forward_time: 112.5115814209, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4566192627
validation[          , genome  6422] predictions:    6199/  10000 (61.99%), best:    6199/10000 (61.99%), error:     10892.78516, best error:     10892.78516 on epoch:     5, epoch:    5/25, mu: 0.6121793389, learning_rate: 0.0010390801, weight_decay: 0.0002078768

epoch time: 1523.8822021484s, input_fired_time: 24.6939373016, output_fired_time: 8.8061180115, propagate_forward_time: 562.1776123047, propagate_backward_time: 913.7888793945, weight_update_time: 0.3237779737, other_time: 14.4155883789
epoch time: 115.1514358521s, input_fired_time: 1.4970579147, output_fired_time: 0.0000000000, propagate_forward_time: 111.1477661133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5066146851
validation[          , genome  6422] predictions:    6120/  10000 (61.20%), best:    6199/10000 (61.99%), error:     10943.28906, best error:     10892.78516 on epoch:     5, epoch:    6/25, mu: 0.6235429049, learning_rate: 0.0009515194, weight_decay: 0.0001903612

epoch time: 2174.6779785156s, input_fired_time: 24.6145477295, output_fired_time: 8.7793788910, propagate_forward_time: 777.9166259766, propagate_backward_time: 1348.9326171875, weight_update_time: 0.3137418628, other_time: 14.4349365234
epoch time: 117.1972122192s, input_fired_time: 1.5364161730, output_fired_time: 0.0000000000, propagate_forward_time: 113.2065353394, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4542617798
validation[          , genome  6422] predictions:    6348/  10000 (63.48%), best:    6348/10000 (63.48%), error:     10605.16602, best error:     10605.16602 on epoch:     7, epoch:    7/25, mu: 0.6345646977, learning_rate: 0.0008713373, weight_decay: 0.0001743214

epoch time: 1519.6671142578s, input_fired_time: 24.7635650635, output_fired_time: 8.7946624756, propagate_forward_time: 560.3701171875, propagate_backward_time: 911.4077758789, weight_update_time: 0.3131066859, other_time: 14.3309936523
epoch time: 117.9723815918s, input_fired_time: 1.5258550644, output_fired_time: 0.0000000000, propagate_forward_time: 113.8974914551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5490341187
validation[          , genome  6422] predictions:    6213/  10000 (62.13%), best:    6348/10000 (63.48%), error:     10849.24707, best error:     10605.16602 on epoch:     7, epoch:    8/25, mu: 0.6452549696, learning_rate: 0.0007979118, weight_decay: 0.0001596332

epoch time: 1523.3076171875s, input_fired_time: 24.6593704224, output_fired_time: 8.7758655548, propagate_forward_time: 562.1578369141, propagate_backward_time: 913.3771972656, weight_update_time: 0.3135036826, other_time: 14.3372802734
epoch time: 115.5699615479s, input_fired_time: 1.5063127279, output_fired_time: 0.0000000000, propagate_forward_time: 111.5757446289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4879074097
validation[          , genome  6422] predictions:    6168/  10000 (61.68%), best:    6348/10000 (63.48%), error:     10996.20605, best error:     10605.16602 on epoch:     7, epoch:    9/25, mu: 0.6556237340, learning_rate: 0.0007306738, weight_decay: 0.0001461825

epoch time: 2504.1572265625s, input_fired_time: 24.5853271484, output_fired_time: 8.7881269455, propagate_forward_time: 561.1148681641, propagate_backward_time: 1895.2856445313, weight_update_time: 0.3128015399, other_time: 14.3831787109
epoch time: 116.4422607422s, input_fired_time: 1.5362710953, output_fired_time: 0.0000000000, propagate_forward_time: 112.3834686279, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5225219727
validation[          , genome  6422] predictions:    6157/  10000 (61.57%), best:    6348/10000 (63.48%), error:     11007.75586, best error:     10605.16602 on epoch:     7, epoch:   10/25, mu: 0.6656806469, learning_rate: 0.0006691017, weight_decay: 0.0001338653

epoch time: 1516.3848876953s, input_fired_time: 24.5732116699, output_fired_time: 8.7724542618, propagate_forward_time: 559.2836303711, propagate_backward_time: 909.3787841797, weight_update_time: 0.3115972579, other_time: 14.3767700195
epoch time: 115.7241668701s, input_fired_time: 1.5282672644, output_fired_time: 0.0000000000, propagate_forward_time: 111.6128082275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5830917358
validation[          , genome  6422] predictions:    6292/  10000 (62.92%), best:    6348/10000 (63.48%), error:     10642.71973, best error:     10605.16602 on epoch:     7, epoch:   11/25, mu: 0.6754350662, learning_rate: 0.0006127182, weight_decay: 0.0001225858

epoch time: 1514.5009765625s, input_fired_time: 24.6356658936, output_fired_time: 8.7640247345, propagate_forward_time: 558.9037475586, propagate_backward_time: 907.7955932617, weight_update_time: 0.3108578622, other_time: 14.4019775391
epoch time: 116.2227554321s, input_fired_time: 1.5455033779, output_fired_time: 0.0000000000, propagate_forward_time: 112.1190032959, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5582504272
validation[          , genome  6422] predictions:    6378/  10000 (63.78%), best:    6378/10000 (63.78%), error:     10372.88184, best error:     10372.88184 on epoch:    12, epoch:   12/25, mu: 0.6848961115, learning_rate: 0.0005610859, weight_decay: 0.0001122568

epoch time: 2487.2524414063s, input_fired_time: 24.8977260590, output_fired_time: 8.7926025391, propagate_forward_time: 783.2866210938, propagate_backward_time: 1655.8229980469, weight_update_time: 0.3151784837, other_time: 14.4523925781
epoch time: 115.5716094971s, input_fired_time: 1.5354465246, output_fired_time: 0.0000000000, propagate_forward_time: 111.5873718262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4487915039
validation[          , genome  6422] predictions:    6324/  10000 (63.24%), best:    6378/10000 (63.78%), error:     10433.01270, best error:     10372.88184 on epoch:    12, epoch:   13/25, mu: 0.6940726042, learning_rate: 0.0005138046, weight_decay: 0.0001027981

epoch time: 1515.6916503906s, input_fired_time: 24.6432552338, output_fired_time: 8.7250595093, propagate_forward_time: 559.6085815430, propagate_backward_time: 908.4245605469, weight_update_time: 0.3121633232, other_time: 14.2901000977
epoch time: 115.3731918335s, input_fired_time: 1.5260846615, output_fired_time: 0.0000000000, propagate_forward_time: 111.4488830566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3982238770
validation[          , genome  6422] predictions:    6193/  10000 (61.93%), best:    6378/10000 (63.78%), error:     10976.60156, best error:     10372.88184 on epoch:    12, epoch:   14/25, mu: 0.7029730678, learning_rate: 0.0004705076, weight_decay: 0.0000941363

epoch time: 1674.4525146484s, input_fired_time: 27.1744556427, output_fired_time: 9.7865476608, propagate_forward_time: 618.3649902344, propagate_backward_time: 1002.9521484375, weight_update_time: 0.3507670164, other_time: 16.1744384766
epoch time: 154.8419494629s, input_fired_time: 2.1103577614, output_fired_time: 0.0000000000, propagate_forward_time: 149.4503936768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2812042236
validation[          , genome  6422] predictions:    6310/  10000 (63.10%), best:    6378/10000 (63.78%), error:     10565.17773, best error:     10372.88184 on epoch:    12, epoch:   15/25, mu: 0.7116058469, learning_rate: 0.0004308591, weight_decay: 0.0000862044

epoch time: 1725.9377441406s, input_fired_time: 28.7671966553, output_fired_time: 10.0112466812, propagate_forward_time: 636.1582031250, propagate_backward_time: 1034.0861816406, weight_update_time: 0.3449996114, other_time: 16.9149169922
epoch time: 145.4280242920s, input_fired_time: 2.0297298431, output_fired_time: 0.0000000000, propagate_forward_time: 140.3954467773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0028533936
validation[          , genome  6422] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10166.17188, best error:     10166.17188 on epoch:    16, epoch:   16/25, mu: 0.7199789882, learning_rate: 0.0003945517, weight_decay: 0.0000789409

epoch time: 1862.0606689453s, input_fired_time: 26.3962249756, output_fired_time: 9.4871826172, propagate_forward_time: 849.1813354492, propagate_backward_time: 961.3838500977, weight_update_time: 0.3392409384, other_time: 15.6120605469
epoch time: 115.8702545166s, input_fired_time: 1.5642483234, output_fired_time: 0.0000000000, propagate_forward_time: 111.7211990356, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5848083496
validation[          , genome  6422] predictions:    6497/  10000 (64.97%), best:    6497/10000 (64.97%), error:     10082.85547, best error:     10082.85547 on epoch:    17, epoch:   17/25, mu: 0.7281002998, learning_rate: 0.0003613038, weight_decay: 0.0000722894

epoch time: 1677.7905273438s, input_fired_time: 24.6884078979, output_fired_time: 8.7971839905, propagate_forward_time: 558.5905151367, propagate_backward_time: 1071.1445312500, weight_update_time: 0.3109576106, other_time: 14.5698242188
epoch time: 115.1276397705s, input_fired_time: 1.5402683020, output_fired_time: 0.0000000000, propagate_forward_time: 111.0690536499, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5183181763
validation[          , genome  6422] predictions:    6452/  10000 (64.52%), best:    6497/10000 (64.97%), error:     10103.45605, best error:     10082.85547 on epoch:    17, epoch:   18/25, mu: 0.7359773517, learning_rate: 0.0003308576, weight_decay: 0.0000661983

epoch time: 1513.5325927734s, input_fired_time: 24.7970695496, output_fired_time: 8.7789182663, propagate_forward_time: 559.6215820313, propagate_backward_time: 906.0584106445, weight_update_time: 0.3079405129, other_time: 14.2765502930
epoch time: 116.4263839722s, input_fired_time: 1.5756081343, output_fired_time: 0.0000000000, propagate_forward_time: 112.3757171631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4750595093
validation[          , genome  6422] predictions:    6556/  10000 (65.56%), best:    6556/10000 (65.56%), error:     10040.54590, best error:     10040.54590 on epoch:    19, epoch:   19/25, mu: 0.7436174750, learning_rate: 0.0003029771, weight_decay: 0.0000606204

epoch time: 8306.0302734375s, input_fired_time: 24.9195003510, output_fired_time: 8.7326946259, propagate_forward_time: 1721.6712646484, propagate_backward_time: 6536.5576171875, weight_update_time: 0.3079133928, other_time: 14.1489257813
epoch time: 116.1340866089s, input_fired_time: 1.5760060549, output_fired_time: 0.0000000000, propagate_forward_time: 112.0279541016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5301284790
validation[          , genome  6422] predictions:    6510/  10000 (65.10%), best:    6510/10000 (65.10%), error:     10007.61035, best error:     10007.61035 on epoch:    20, epoch:   20/25, mu: 0.7510278225, learning_rate: 0.0002774459, weight_decay: 0.0000555126

epoch time: 2632.7900390625s, input_fired_time: 24.7438774109, output_fired_time: 8.7269802094, propagate_forward_time: 1464.3745117188, propagate_backward_time: 1120.6760253906, weight_update_time: 0.3076957166, other_time: 14.2685546875
epoch time: 118.7381668091s, input_fired_time: 1.5795220137, output_fired_time: 0.0000000000, propagate_forward_time: 114.6452484131, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5133972168
validation[          , genome  6422] predictions:    6524/  10000 (65.24%), best:    6524/10000 (65.24%), error:      9905.63574, best error:      9905.63574 on epoch:    21, epoch:   21/25, mu: 0.7582153082, learning_rate: 0.0002540663, weight_decay: 0.0000508351

epoch time: 1507.6810302734s, input_fired_time: 24.7316093445, output_fired_time: 8.7213401794, propagate_forward_time: 556.8450317383, propagate_backward_time: 903.0748291016, weight_update_time: 0.3092505932, other_time: 14.3082885742
epoch time: 115.5847625732s, input_fired_time: 1.5648247004, output_fired_time: 0.0000000000, propagate_forward_time: 111.5256118774, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4943237305
validation[          , genome  6422] predictions:    6456/  10000 (64.56%), best:    6524/10000 (65.24%), error:     10119.50195, best error:      9905.63574 on epoch:    21, epoch:   22/25, mu: 0.7651866078, learning_rate: 0.0002326567, weight_decay: 0.0000465518

epoch time: 2806.5385742188s, input_fired_time: 24.7123146057, output_fired_time: 8.7441949844, propagate_forward_time: 558.0906982422, propagate_backward_time: 2200.6203613281, weight_update_time: 0.3078645766, other_time: 14.3708496094
epoch time: 115.3542022705s, input_fired_time: 1.5821875334, output_fired_time: 0.0000000000, propagate_forward_time: 111.2374114990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5346069336
validation[          , genome  6422] predictions:    6478/  10000 (64.78%), best:    6524/10000 (65.24%), error:     10083.16992, best error:      9905.63574 on epoch:    21, epoch:   23/25, mu: 0.7719482183, learning_rate: 0.0002130513, weight_decay: 0.0000426293

epoch time: 1524.1917724609s, input_fired_time: 25.1665000916, output_fired_time: 8.9413566589, propagate_forward_time: 562.1359863281, propagate_backward_time: 912.9810180664, weight_update_time: 0.3176971972, other_time: 14.9668579102
epoch time: 117.2152252197s, input_fired_time: 1.6124653816, output_fired_time: 0.0000000000, propagate_forward_time: 113.0299148560, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5728454590
validation[          , genome  6422] predictions:    6550/  10000 (65.50%), best:    6524/10000 (65.24%), error:      9956.15527, best error:      9905.63574 on epoch:    21, epoch:   24/25, mu: 0.7785064578, learning_rate: 0.0001950980, weight_decay: 0.0000390374

epoch time: 1533.9320068359s, input_fired_time: 25.5367107391, output_fired_time: 9.0390663147, propagate_forward_time: 566.1911010742, propagate_backward_time: 918.0165405273, weight_update_time: 0.3208376467, other_time: 15.1485595703
epoch time: 118.2108535767s, input_fired_time: 1.6403617859, output_fired_time: 0.0000000000, propagate_forward_time: 113.9509963989, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6194992065
validation[          , genome  6422] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:      9891.76953, best error:      9891.76953 on epoch:    25, epoch:   25/25, mu: 0.7848674655, learning_rate: 0.0001786576, weight_decay: 0.0000357481

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1044.6757812500s, input_fired_time: 8.1258621216, output_fired_time: 0.0000000000, propagate_forward_time: 1023.6987304688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.8511962891
best training[          , genome  6422] predictions:   36372/  50000 (72.74%), best:    6546/10000 (65.46%), error:     40387.73438, best error:      9891.76953 on epoch:    25, epoch:   26/25, mu: 0.7910371423, learning_rate: 0.0001636026, weight_decay: 0.0000327360
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 117.4131622314s, input_fired_time: 1.6221878529, output_fired_time: 0.0000000000, propagate_forward_time: 113.1668395996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6241378784
   testing[          , genome  6422] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:      9891.76953, best error:      9891.76953 on epoch:    25, epoch:   26/25, mu: 0.7910371423, learning_rate: 0.0001636026, weight_decay: 0.0000327360
backpropagation finished successfully!
07:04:07 (1536): called boinc_finish(0)

</stderr_txt>
]]>