Nameexact_genome_1542692703_178_6371_3
Workunit4100304
Created4 Feb 2019, 12:48:16 UTC
Sent4 Feb 2019, 12:48:30 UTC
Report deadline21 Apr 2019, 18:26:29 UTC
Received5 Feb 2019, 6:36:03 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time17 hours 44 min 37 sec
CPU time17 hours 40 min 26 sec
Validate stateValid
Credit19,058.68
Device peak FLOPS4.37 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size569.86 MB
Peak swap size589.70 MB
Peak disk usage16.42 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6371.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6371_3_r1447296945_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6371.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487938
read mu: 0.487938
read mu_delta: 0.95195
read initial_learning_rate: 0.000690388
read learning_rate: 0.000690388
read learning_rate_delta: 0.920021
read initial_weight_decay: 0.000503204
read weight_decay: 0.000503204
read weight_decay_delta: 0.926033
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0862348
read input_dropout_probability: 0.0011378
read hidden_dropout_probability: 0.109211
read velocity_reset: 2170
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6371
read normal distribution: '1 -0.359059572219849 0.62816196680069'
generator_str: '714351106'
read generator: 714351106
read generated_by_map:
1 crossover 1
reading 120 nodes.
reading 629 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1665451812
post shuffle 1: 795412496
epoch time: 1171.10900878906s, input_fired_time: 18.3309421539307, output_fired_time: 4.95036554336548, propagate_forward_time: 418.645843505859, propagate_backward_time: 708.869934082031, weight_update_time: 0.314247220754623, other_time: 20.3119506835938
epoch time: 87.2619552612305s, input_fired_time: 1.00982618331909, output_fired_time: 0, propagate_forward_time: 82.3963394165039, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.85578918457031
validation[          , genome  6371] predictions:    5851/  10000 (58.51%), best:    5851/10000 (58.51%), error:     12205.16016, best error:     12205.16016 on epoch:     0, epoch:    0/50, mu: 0.4879384041, learning_rate: 0.0006903875, weight_decay: 0.0005032038

epoch time: 1147.4366455078s, input_fired_time: 18.1512680054, output_fired_time: 4.8344087601, propagate_forward_time: 408.7622985840, propagate_backward_time: 694.8596801758, weight_update_time: 0.2958233953, other_time: 20.8290405273
epoch time: 87.3984069824s, input_fired_time: 0.9999215007, output_fired_time: 0.0000000000, propagate_forward_time: 82.5556869507, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8427963257
validation[          , genome  6371] predictions:    5241/  10000 (52.41%), best:    5851/10000 (58.51%), error:     14612.85449, best error:     12205.16016 on epoch:     0, epoch:    1/50, mu: 0.5120623708, learning_rate: 0.0006351707, weight_decay: 0.0004659831

epoch time: 1146.9587402344s, input_fired_time: 18.1381034851, output_fired_time: 4.8281340599, propagate_forward_time: 408.9361877441, propagate_backward_time: 694.2449951172, weight_update_time: 0.2947723866, other_time: 20.8114013672
epoch time: 87.4873886108s, input_fired_time: 0.9875639081, output_fired_time: 0.0000000000, propagate_forward_time: 82.5564575195, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9433670044
validation[          , genome  6371] predictions:    5679/  10000 (56.79%), best:    5851/10000 (58.51%), error:     12278.61328, best error:     12205.16016 on epoch:     0, epoch:    2/50, mu: 0.5350272059, learning_rate: 0.0005843700, weight_decay: 0.0004315157

epoch time: 1147.7773437500s, input_fired_time: 18.1153488159, output_fired_time: 4.8414707184, propagate_forward_time: 408.9851379395, propagate_backward_time: 694.7136840820, weight_update_time: 0.2969872355, other_time: 21.1217651367
epoch time: 87.4324951172s, input_fired_time: 0.9958930016, output_fired_time: 0.0000000000, propagate_forward_time: 82.5045928955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9320068359
validation[          , genome  6371] predictions:    5977/  10000 (59.77%), best:    5977/10000 (59.77%), error:     11618.38281, best error:     11618.38281 on epoch:     3, epoch:    3/50, mu: 0.5568885803, learning_rate: 0.0005376324, weight_decay: 0.0003995976

epoch time: 1147.2086181641s, input_fired_time: 18.0936355591, output_fired_time: 4.8373985291, propagate_forward_time: 408.6090698242, propagate_backward_time: 694.6453857422, weight_update_time: 0.2943717241, other_time: 21.0231323242
epoch time: 87.1927490234s, input_fired_time: 0.9861366153, output_fired_time: 0.0000000000, propagate_forward_time: 82.3136978149, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8929138184
validation[          , genome  6371] predictions:    5597/  10000 (55.97%), best:    5977/10000 (59.77%), error:     12647.25195, best error:     11618.38281 on epoch:     3, epoch:    4/50, mu: 0.5776994824, learning_rate: 0.0004946329, weight_decay: 0.0003700405

epoch time: 1147.1717529297s, input_fired_time: 18.0660591125, output_fired_time: 4.8380298615, propagate_forward_time: 408.5275878906, propagate_backward_time: 694.6584472656, weight_update_time: 0.2932381630, other_time: 21.0816650391
epoch time: 87.2936401367s, input_fired_time: 0.9838654995, output_fired_time: 0.0000000000, propagate_forward_time: 82.4146118164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8951644897
validation[          , genome  6371] predictions:    5629/  10000 (56.29%), best:    5977/10000 (59.77%), error:     13068.20996, best error:     11618.38281 on epoch:     3, epoch:    5/50, mu: 0.5975104570, learning_rate: 0.0004550724, weight_decay: 0.0003426696

epoch time: 1148.7604980469s, input_fired_time: 18.0952224731, output_fired_time: 4.8472127914, propagate_forward_time: 409.3087158203, propagate_backward_time: 695.4993896484, weight_update_time: 0.2978025079, other_time: 21.0100097656
epoch time: 87.2807922363s, input_fired_time: 0.9817367196, output_fired_time: 0.0000000000, propagate_forward_time: 82.4929504395, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8061065674
validation[          , genome  6371] predictions:    6037/  10000 (60.37%), best:    6037/10000 (60.37%), error:     11459.94336, best error:     11459.94336 on epoch:     6, epoch:    6/50, mu: 0.6163694859, learning_rate: 0.0004186759, weight_decay: 0.0003173233

epoch time: 1150.1528320313s, input_fired_time: 18.0783157349, output_fired_time: 4.8631801605, propagate_forward_time: 409.6589050293, propagate_backward_time: 696.8057861328, weight_update_time: 0.3001708686, other_time: 20.7465820313
epoch time: 87.5386962891s, input_fired_time: 0.9856735468, output_fired_time: 0.0000000000, propagate_forward_time: 82.6879119873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8651123047
validation[          , genome  6371] predictions:    5720/  10000 (57.20%), best:    6037/10000 (60.37%), error:     12904.93555, best error:     11459.94336 on epoch:     6, epoch:    7/50, mu: 0.6343223453, learning_rate: 0.0003851904, weight_decay: 0.0002938517

epoch time: 1157.1087646484s, input_fired_time: 18.1331710815, output_fired_time: 4.8889918327, propagate_forward_time: 412.5122680664, propagate_backward_time: 700.8859863281, weight_update_time: 0.3043406308, other_time: 20.6882934570
epoch time: 87.8213958740s, input_fired_time: 0.9787873626, output_fired_time: 0.0000000000, propagate_forward_time: 83.0192642212, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8233413696
validation[          , genome  6371] predictions:    5837/  10000 (58.37%), best:    6037/10000 (60.37%), error:     12101.86328, best error:     11459.94336 on epoch:     6, epoch:    8/50, mu: 0.6514126062, learning_rate: 0.0003543831, weight_decay: 0.0002721163

epoch time: 1154.2469482422s, input_fired_time: 18.1291065216, output_fired_time: 4.8883838654, propagate_forward_time: 411.1437377930, propagate_backward_time: 699.2936401367, weight_update_time: 0.3043254614, other_time: 20.7919921875
epoch time: 87.6192703247s, input_fired_time: 0.9892877340, output_fired_time: 0.0000000000, propagate_forward_time: 82.6513442993, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9786376953
validation[          , genome  6371] predictions:    5997/  10000 (59.97%), best:    6037/10000 (60.37%), error:     11624.63477, best error:     11459.94336 on epoch:     6, epoch:    9/50, mu: 0.6676816940, learning_rate: 0.0003260397, weight_decay: 0.0002519886

epoch time: 1154.2623291016s, input_fired_time: 18.1369113922, output_fired_time: 4.8690471649, propagate_forward_time: 411.1906433105, propagate_backward_time: 699.2697143555, weight_update_time: 0.3015769422, other_time: 20.7959594727
epoch time: 88.0776748657s, input_fired_time: 0.9844425321, output_fired_time: 0.0000000000, propagate_forward_time: 83.1797027588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9135284424
validation[          , genome  6371] predictions:    5873/  10000 (58.73%), best:    6037/10000 (60.37%), error:     11981.72461, best error:     11459.94336 on epoch:     6, epoch:   10/50, mu: 0.6831690073, learning_rate: 0.0002999633, weight_decay: 0.0002333497

epoch time: 1156.5800781250s, input_fired_time: 18.1558952332, output_fired_time: 4.8945698738, propagate_forward_time: 412.1633300781, propagate_backward_time: 700.8163452148, weight_update_time: 0.3050734103, other_time: 20.5499877930
epoch time: 87.7616500854s, input_fired_time: 0.9849869609, output_fired_time: 0.0000000000, propagate_forward_time: 82.9021224976, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8745422363
validation[          , genome  6371] predictions:    6041/  10000 (60.41%), best:    6037/10000 (60.37%), error:     11605.66992, best error:     11459.94336 on epoch:     6, epoch:   11/50, mu: 0.6979121566, learning_rate: 0.0002759724, weight_decay: 0.0002160894

epoch time: 1155.9851074219s, input_fired_time: 18.0685386658, output_fired_time: 4.8909325600, propagate_forward_time: 412.0840148926, propagate_backward_time: 700.3084716797, weight_update_time: 0.3056939542, other_time: 20.6331787109
epoch time: 88.1485137939s, input_fired_time: 0.9903801084, output_fired_time: 0.0000000000, propagate_forward_time: 83.2826538086, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8754806519
validation[          , genome  6371] predictions:    6079/  10000 (60.79%), best:    6079/10000 (60.79%), error:     11390.74121, best error:     11390.74121 on epoch:    12, epoch:   12/50, mu: 0.7119469047, learning_rate: 0.0002539002, weight_decay: 0.0002001059

epoch time: 1154.5294189453s, input_fired_time: 18.0944423676, output_fired_time: 4.8868126869, propagate_forward_time: 411.2436218262, propagate_backward_time: 699.3098754883, weight_update_time: 0.3009253442, other_time: 20.9945678711
epoch time: 87.4824676514s, input_fired_time: 0.9857016802, output_fired_time: 0.0000000000, propagate_forward_time: 82.4692993164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0274658203
validation[          , genome  6371] predictions:    6139/  10000 (61.39%), best:    6139/10000 (61.39%), error:     11370.71191, best error:     11370.71191 on epoch:    13, epoch:   13/50, mu: 0.7253072858, learning_rate: 0.0002335934, weight_decay: 0.0001853046

epoch time: 1149.7935791016s, input_fired_time: 18.0609569550, output_fired_time: 4.8644509315, propagate_forward_time: 409.7585754395, propagate_backward_time: 695.7081298828, weight_update_time: 0.2966526151, other_time: 21.4014892578
epoch time: 87.3257827759s, input_fired_time: 0.9851695895, output_fired_time: 0.0000000000, propagate_forward_time: 82.4462738037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8943405151
validation[          , genome  6371] predictions:    6205/  10000 (62.05%), best:    6205/10000 (62.05%), error:     11146.46875, best error:     11146.46875 on epoch:    14, epoch:   14/50, mu: 0.7380257249, learning_rate: 0.0002149108, weight_decay: 0.0001715981

epoch time: 1148.9555664063s, input_fired_time: 18.0515804291, output_fired_time: 4.8579368591, propagate_forward_time: 409.0466613770, propagate_backward_time: 695.6886596680, weight_update_time: 0.2977997363, other_time: 21.3107299805
epoch time: 87.3888778687s, input_fired_time: 0.9858330488, output_fired_time: 0.0000000000, propagate_forward_time: 82.4623184204, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9407272339
validation[          , genome  6371] predictions:    6152/  10000 (61.52%), best:    6152/10000 (61.52%), error:     11131.75977, best error:     11131.75977 on epoch:    15, epoch:   15/50, mu: 0.7501330376, learning_rate: 0.0001977223, weight_decay: 0.0001589055

epoch time: 1148.5631103516s, input_fired_time: 18.0642356873, output_fired_time: 4.8612866402, propagate_forward_time: 408.9456787109, propagate_backward_time: 695.4141845703, weight_update_time: 0.2954003215, other_time: 21.2777099609
epoch time: 88.0587615967s, input_fired_time: 0.9883140326, output_fired_time: 0.0000000000, propagate_forward_time: 82.9854888916, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0849609375
validation[          , genome  6371] predictions:    6174/  10000 (61.74%), best:    6152/10000 (61.52%), error:     11131.76660, best error:     11131.75977 on epoch:    15, epoch:   16/50, mu: 0.7616586089, learning_rate: 0.0001819086, weight_decay: 0.0001471517

epoch time: 1148.5583496094s, input_fired_time: 18.0782146454, output_fired_time: 4.8564519882, propagate_forward_time: 408.9272155762, propagate_backward_time: 695.4164428711, weight_update_time: 0.2957459688, other_time: 21.2799682617
epoch time: 87.4488677979s, input_fired_time: 0.9899655581, output_fired_time: 0.0000000000, propagate_forward_time: 82.4585952759, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0003051758
validation[          , genome  6371] predictions:    6073/  10000 (60.73%), best:    6152/10000 (61.52%), error:     11640.44336, best error:     11131.75977 on epoch:    15, epoch:   17/50, mu: 0.7726303935, learning_rate: 0.0001673596, weight_decay: 0.0001362673

epoch time: 1148.4772949219s, input_fired_time: 18.0393848419, output_fired_time: 4.8534607887, propagate_forward_time: 409.0261840820, propagate_backward_time: 695.2284545898, weight_update_time: 0.2946647704, other_time: 21.3297119141
epoch time: 87.3952865601s, input_fired_time: 0.9834694862, output_fired_time: 0.0000000000, propagate_forward_time: 82.4404144287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9714050293
validation[          , genome  6371] predictions:    5922/  10000 (59.22%), best:    6152/10000 (61.52%), error:     11971.86328, best error:     11131.75977 on epoch:    15, epoch:   18/50, mu: 0.7830749750, learning_rate: 0.0001539743, weight_decay: 0.0001261880

epoch time: 1148.4689941406s, input_fired_time: 18.0591659546, output_fired_time: 4.8665437698, propagate_forward_time: 408.9475402832, propagate_backward_time: 695.3703613281, weight_update_time: 0.2955916822, other_time: 21.2253417969
epoch time: 87.7844619751s, input_fired_time: 0.9954255819, output_fired_time: 0.0000000000, propagate_forward_time: 82.8424758911, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9465637207
validation[          , genome  6371] predictions:    6290/  10000 (62.90%), best:    6290/10000 (62.90%), error:     10951.33887, best error:     10951.33887 on epoch:    19, epoch:   19/50, mu: 0.7930176854, learning_rate: 0.0001416595, weight_decay: 0.0001168542

epoch time: 1148.7050781250s, input_fired_time: 18.0624637604, output_fired_time: 4.8481092453, propagate_forward_time: 409.1722717285, propagate_backward_time: 695.3501586914, weight_update_time: 0.2947011292, other_time: 21.2720336914
epoch time: 87.3842468262s, input_fired_time: 0.9945378900, output_fired_time: 0.0000000000, propagate_forward_time: 82.4098739624, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9798355103
validation[          , genome  6371] predictions:    6255/  10000 (62.55%), best:    6255/10000 (62.55%), error:     10813.48242, best error:     10813.48242 on epoch:    20, epoch:   20/50, mu: 0.8024826646, learning_rate: 0.0001303296, weight_decay: 0.0001082108

epoch time: 1148.7103271484s, input_fired_time: 18.0852870941, output_fired_time: 4.8577179909, propagate_forward_time: 408.9739990234, propagate_backward_time: 695.4437255859, weight_update_time: 0.2966004312, other_time: 21.3496093750
epoch time: 87.3366241455s, input_fired_time: 0.9912420511, output_fired_time: 0.0000000000, propagate_forward_time: 82.3878631592, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9575195313
validation[          , genome  6371] predictions:    6309/  10000 (63.09%), best:    6255/10000 (62.55%), error:     10864.45410, best error:     10813.48242 on epoch:    20, epoch:   21/50, mu: 0.8114928603, learning_rate: 0.0001199059, weight_decay: 0.0001002067

epoch time: 1148.0670166016s, input_fired_time: 18.0805091858, output_fired_time: 4.8554568291, propagate_forward_time: 408.7672729492, propagate_backward_time: 695.0910034180, weight_update_time: 0.2955374420, other_time: 21.2727050781
epoch time: 87.4052658081s, input_fired_time: 0.9930962324, output_fired_time: 0.0000000000, propagate_forward_time: 82.4020767212, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0100936890
validation[          , genome  6371] predictions:    6234/  10000 (62.34%), best:    6255/10000 (62.55%), error:     11021.36914, best error:     10813.48242 on epoch:    20, epoch:   22/50, mu: 0.8200700879, learning_rate: 0.0001103159, weight_decay: 0.0000927947

epoch time: 1146.9670410156s, input_fired_time: 18.0811290741, output_fired_time: 4.8348765373, propagate_forward_time: 408.4191284180, propagate_backward_time: 694.3334960938, weight_update_time: 0.2924633026, other_time: 21.2984008789
epoch time: 87.2149963379s, input_fired_time: 0.9959757924, output_fired_time: 0.0000000000, propagate_forward_time: 82.3214340210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8975830078
validation[          , genome  6371] predictions:    6328/  10000 (63.28%), best:    6328/10000 (63.28%), error:     10735.56934, best error:     10735.56934 on epoch:    23, epoch:   23/50, mu: 0.8282352090, learning_rate: 0.0001014929, weight_decay: 0.0000859309

epoch time: 1147.1868896484s, input_fired_time: 18.0899028778, output_fired_time: 4.8415255547, propagate_forward_time: 408.5261840820, propagate_backward_time: 694.4725952148, weight_update_time: 0.2908887267, other_time: 21.2567138672
epoch time: 87.2491912842s, input_fired_time: 0.9946820736, output_fired_time: 0.0000000000, propagate_forward_time: 82.3402938843, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9142150879
validation[          , genome  6371] predictions:    6332/  10000 (63.32%), best:    6332/10000 (63.32%), error:     10690.27148, best error:     10690.27148 on epoch:    24, epoch:   24/50, mu: 0.8360079527, learning_rate: 0.0000933756, weight_decay: 0.0000795749

epoch time: 1146.8033447266s, input_fired_time: 18.0877704620, output_fired_time: 4.8389863968, propagate_forward_time: 408.4808654785, propagate_backward_time: 694.3153076172, weight_update_time: 0.2916831374, other_time: 21.0804443359
epoch time: 87.2485275269s, input_fired_time: 0.9901549220, output_fired_time: 0.0000000000, propagate_forward_time: 82.3295211792, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9288482666
validation[          , genome  6371] predictions:    6301/  10000 (63.01%), best:    6332/10000 (63.32%), error:     10757.70215, best error:     10690.27148 on epoch:    24, epoch:   25/50, mu: 0.8434072137, learning_rate: 0.0000859074, weight_decay: 0.0000736889

epoch time: 1147.6359863281s, input_fired_time: 18.1106605530, output_fired_time: 4.8465628624, propagate_forward_time: 408.7738342285, propagate_backward_time: 694.5135498047, weight_update_time: 0.2906827927, other_time: 21.3913574219
epoch time: 87.3232650757s, input_fired_time: 1.0002330542, output_fired_time: 0.0000000000, propagate_forward_time: 82.3874359131, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9355926514
validation[          , genome  6371] predictions:    6368/  10000 (63.68%), best:    6368/10000 (63.68%), error:     10538.23438, best error:     10538.23438 on epoch:    26, epoch:   26/50, mu: 0.8504509926, learning_rate: 0.0000790366, weight_decay: 0.0000682384

epoch time: 1147.2917480469s, input_fired_time: 18.1275062561, output_fired_time: 4.8423342705, propagate_forward_time: 408.7068481445, propagate_backward_time: 694.5612182617, weight_update_time: 0.2918126583, other_time: 21.0538330078
epoch time: 87.3429946899s, input_fired_time: 0.9983304143, output_fired_time: 0.0000000000, propagate_forward_time: 82.3481597900, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9965057373
validation[          , genome  6371] predictions:    6335/  10000 (63.35%), best:    6368/10000 (63.68%), error:     10689.48730, best error:     10538.23438 on epoch:    26, epoch:   27/50, mu: 0.8571562767, learning_rate: 0.0000727153, weight_decay: 0.0000631909

epoch time: 1147.4482421875s, input_fired_time: 18.1272678375, output_fired_time: 4.8459239006, propagate_forward_time: 408.7299804688, propagate_backward_time: 694.4594116211, weight_update_time: 0.2898078561, other_time: 21.2855834961
epoch time: 87.5017547607s, input_fired_time: 0.9986122251, output_fired_time: 0.0000000000, propagate_forward_time: 82.4252243042, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0779190063
validation[          , genome  6371] predictions:    6267/  10000 (62.67%), best:    6368/10000 (63.68%), error:     10830.14258, best error:     10538.23438 on epoch:    26, epoch:   28/50, mu: 0.8635393977, learning_rate: 0.0000668996, weight_decay: 0.0000585169

epoch time: 1146.8947753906s, input_fired_time: 18.1182270050, output_fired_time: 4.8454484940, propagate_forward_time: 408.4319763184, propagate_backward_time: 694.3144531250, weight_update_time: 0.2932167053, other_time: 21.1845703125
epoch time: 87.3265228271s, input_fired_time: 0.9965069890, output_fired_time: 0.0000000000, propagate_forward_time: 82.3653945923, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9646224976
validation[          , genome  6371] predictions:    6184/  10000 (61.84%), best:    6368/10000 (63.68%), error:     10858.52637, best error:     10538.23438 on epoch:    26, epoch:   29/50, mu: 0.8696157932, learning_rate: 0.0000615490, weight_decay: 0.0000541885

epoch time: 1147.5407714844s, input_fired_time: 18.1470165253, output_fired_time: 4.8456053734, propagate_forward_time: 408.6826782227, propagate_backward_time: 694.6006469727, weight_update_time: 0.2901221514, other_time: 21.2648925781
epoch time: 87.3518981934s, input_fired_time: 1.0009354353, output_fired_time: 0.0000000000, propagate_forward_time: 82.4590225220, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8919372559
validation[          , genome  6371] predictions:    6353/  10000 (63.53%), best:    6368/10000 (63.68%), error:     10578.80762, best error:     10538.23438 on epoch:    26, epoch:   30/50, mu: 0.8754002452, learning_rate: 0.0000566263, weight_decay: 0.0000501804

epoch time: 1147.1917724609s, input_fired_time: 18.1418819427, output_fired_time: 4.8459010124, propagate_forward_time: 408.5787658691, propagate_backward_time: 694.3585815430, weight_update_time: 0.2915074229, other_time: 21.2666625977
epoch time: 87.5269165039s, input_fired_time: 1.0106549263, output_fired_time: 0.0000000000, propagate_forward_time: 82.5062484741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0100097656
validation[          , genome  6371] predictions:    6421/  10000 (64.21%), best:    6421/10000 (64.21%), error:     10487.63281, best error:     10487.63281 on epoch:    31, epoch:   31/50, mu: 0.8809067607, learning_rate: 0.0000520974, weight_decay: 0.0000464687

epoch time: 1147.5227050781s, input_fired_time: 18.1351013184, output_fired_time: 4.8478212357, propagate_forward_time: 408.6801757813, propagate_backward_time: 694.5925292969, weight_update_time: 0.2921815813, other_time: 21.2670898438
epoch time: 87.3934326172s, input_fired_time: 1.0029977560, output_fired_time: 0.0000000000, propagate_forward_time: 82.4716720581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9187622070
validation[          , genome  6371] predictions:    6277/  10000 (62.77%), best:    6421/10000 (64.21%), error:     10716.55859, best error:     10487.63281 on epoch:    31, epoch:   32/50, mu: 0.8861486912, learning_rate: 0.0000479307, weight_decay: 0.0000430315

epoch time: 1147.4151611328s, input_fired_time: 18.0989589691, output_fired_time: 4.8438935280, propagate_forward_time: 408.5138854980, propagate_backward_time: 694.7321777344, weight_update_time: 0.2926710844, other_time: 21.2261962891
epoch time: 87.4891738892s, input_fired_time: 1.0068082809, output_fired_time: 0.0000000000, propagate_forward_time: 82.4569320679, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0254364014
validation[          , genome  6371] predictions:    6504/  10000 (65.04%), best:    6504/10000 (65.04%), error:     10252.10742, best error:     10252.10742 on epoch:    33, epoch:   33/50, mu: 0.8911387324, learning_rate: 0.0000440972, weight_decay: 0.0000398486

epoch time: 1147.4097900391s, input_fired_time: 18.1134357452, output_fired_time: 4.8438477516, propagate_forward_time: 408.5508117676, propagate_backward_time: 694.5990600586, weight_update_time: 0.2906489372, other_time: 21.3026733398
epoch time: 87.3651199341s, input_fired_time: 1.0092374086, output_fired_time: 0.0000000000, propagate_forward_time: 82.3461074829, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0097732544
validation[          , genome  6371] predictions:    6376/  10000 (63.76%), best:    6504/10000 (65.04%), error:     10473.71387, best error:     10252.10742 on epoch:    33, epoch:   34/50, mu: 0.8958889842, learning_rate: 0.0000405703, weight_decay: 0.0000369011

epoch time: 1147.2558593750s, input_fired_time: 18.1082305908, output_fired_time: 4.8458566666, propagate_forward_time: 408.6697998047, propagate_backward_time: 694.4639892578, weight_update_time: 0.2928123176, other_time: 21.1679687500
epoch time: 87.3221130371s, input_fired_time: 1.0098427534, output_fired_time: 0.0000000000, propagate_forward_time: 82.3477935791, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9644775391
validation[          , genome  6371] predictions:    6438/  10000 (64.38%), best:    6504/10000 (65.04%), error:     10336.01758, best error:     10252.10742 on epoch:    33, epoch:   35/50, mu: 0.9004110098, learning_rate: 0.0000373255, weight_decay: 0.0000341716

epoch time: 1147.4915771484s, input_fired_time: 18.1350021362, output_fired_time: 4.8592429161, propagate_forward_time: 408.8365783691, propagate_backward_time: 694.5227050781, weight_update_time: 0.2899936736, other_time: 21.1380615234
epoch time: 87.4179687500s, input_fired_time: 1.0087592602, output_fired_time: 0.0000000000, propagate_forward_time: 82.4126358032, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9965744019
validation[          , genome  6371] predictions:    6509/  10000 (65.09%), best:    6509/10000 (65.09%), error:     10245.30957, best error:     10245.30957 on epoch:    36, epoch:   36/50, mu: 0.9047157168, learning_rate: 0.0000343402, weight_decay: 0.0000316440

epoch time: 1147.4132080078s, input_fired_time: 18.1406364441, output_fired_time: 4.8398013115, propagate_forward_time: 408.6481628418, propagate_backward_time: 694.4117431641, weight_update_time: 0.2921994925, other_time: 21.3728027344
epoch time: 87.3617630005s, input_fired_time: 1.0084458590, output_fired_time: 0.0000000000, propagate_forward_time: 82.3160171509, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0373001099
validation[          , genome  6371] predictions:    6473/  10000 (64.73%), best:    6509/10000 (65.09%), error:     10264.06543, best error:     10245.30957 on epoch:    36, epoch:   37/50, mu: 0.9088135958, learning_rate: 0.0000315937, weight_decay: 0.0000293034

epoch time: 1147.5364990234s, input_fired_time: 18.1263351440, output_fired_time: 4.8433489799, propagate_forward_time: 408.5727233887, propagate_backward_time: 694.7598266602, weight_update_time: 0.2900631726, other_time: 21.2341918945
epoch time: 87.4019088745s, input_fired_time: 1.0141310692, output_fired_time: 0.0000000000, propagate_forward_time: 82.4944458008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8933334351
validation[          , genome  6371] predictions:    6563/  10000 (65.63%), best:    6563/10000 (65.63%), error:     10053.33398, best error:     10053.33398 on epoch:    38, epoch:   38/50, mu: 0.9127146006, learning_rate: 0.0000290669, weight_decay: 0.0000271359

epoch time: 1146.9150390625s, input_fired_time: 18.1467781067, output_fired_time: 4.8483967781, propagate_forward_time: 408.5712890625, propagate_backward_time: 694.0884399414, weight_update_time: 0.2901195884, other_time: 21.2601928711
epoch time: 87.3743591309s, input_fired_time: 1.0164026022, output_fired_time: 0.0000000000, propagate_forward_time: 82.3982925415, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9596633911
validation[          , genome  6371] predictions:    6478/  10000 (64.78%), best:    6563/10000 (65.63%), error:     10206.45996, best error:     10053.33398 on epoch:    38, epoch:   39/50, mu: 0.9164281487, learning_rate: 0.0000267421, weight_decay: 0.0000251287

epoch time: 1147.5771484375s, input_fired_time: 18.1520671844, output_fired_time: 4.8403801918, propagate_forward_time: 408.4711608887, propagate_backward_time: 694.9101562500, weight_update_time: 0.2931480110, other_time: 21.2033691406
epoch time: 87.4002838135s, input_fired_time: 1.0148421526, output_fired_time: 0.0000000000, propagate_forward_time: 82.3748550415, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0105895996
validation[          , genome  6371] predictions:    6641/  10000 (66.41%), best:    6641/10000 (66.41%), error:      9956.67383, best error:      9956.67383 on epoch:    40, epoch:   40/50, mu: 0.9199632406, learning_rate: 0.0000246033, weight_decay: 0.0000232700

epoch time: 1148.0306396484s, input_fired_time: 18.1579875946, output_fired_time: 4.8533577919, propagate_forward_time: 408.6982116699, propagate_backward_time: 695.0195312500, weight_update_time: 0.2934254408, other_time: 21.3015136719
epoch time: 87.4910049438s, input_fired_time: 1.0205522776, output_fired_time: 0.0000000000, propagate_forward_time: 82.4581832886, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0122680664
validation[          , genome  6371] predictions:    6562/  10000 (65.62%), best:    6641/10000 (66.41%), error:     10104.66504, best error:      9956.67383 on epoch:    40, epoch:   41/50, mu: 0.9233285189, learning_rate: 0.0000226355, weight_decay: 0.0000215488

epoch time: 1157.7810058594s, input_fired_time: 18.2658710480, output_fired_time: 4.8932256699, propagate_forward_time: 412.4824523926, propagate_backward_time: 701.1903686523, weight_update_time: 0.3018700480, other_time: 20.9491577148
epoch time: 87.5384445190s, input_fired_time: 1.0223317146, output_fired_time: 0.0000000000, propagate_forward_time: 82.6085052490, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9076080322
validation[          , genome  6371] predictions:    6497/  10000 (64.97%), best:    6641/10000 (66.41%), error:     10154.91016, best error:      9956.67383 on epoch:    40, epoch:   42/50, mu: 0.9265320897, learning_rate: 0.0000208252, weight_decay: 0.0000199549

epoch time: 1153.2415771484s, input_fired_time: 18.2422904968, output_fired_time: 4.8886790276, propagate_forward_time: 411.1188964844, propagate_backward_time: 697.9524536133, weight_update_time: 0.2991687953, other_time: 21.0392456055
epoch time: 87.6751098633s, input_fired_time: 1.0213476419, output_fired_time: 0.0000000000, propagate_forward_time: 82.7393341064, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9144287109
validation[          , genome  6371] predictions:    6340/  10000 (63.40%), best:    6641/10000 (66.41%), error:     10515.34668, best error:      9956.67383 on epoch:    40, epoch:   43/50, mu: 0.9295817018, learning_rate: 0.0000191596, weight_decay: 0.0000184789

epoch time: 1148.9958496094s, input_fired_time: 18.1948242188, output_fired_time: 4.8585209846, propagate_forward_time: 409.1729125977, propagate_backward_time: 695.4137573242, weight_update_time: 0.2941549420, other_time: 21.3558349609
epoch time: 87.4432373047s, input_fired_time: 1.0196309090, output_fired_time: 0.0000000000, propagate_forward_time: 82.3594665527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0641403198
validation[          , genome  6371] predictions:    6595/  10000 (65.95%), best:    6641/10000 (66.41%), error:      9999.88281, best error:      9956.67383 on epoch:    40, epoch:   44/50, mu: 0.9324848056, learning_rate: 0.0000176272, weight_decay: 0.0000171121

epoch time: 1147.7321777344s, input_fired_time: 18.2007732391, output_fired_time: 4.8475937843, propagate_forward_time: 408.6801757813, propagate_backward_time: 694.5064086914, weight_update_time: 0.2922809124, other_time: 21.4972534180
epoch time: 87.8004379272s, input_fired_time: 1.0277490616, output_fired_time: 0.0000000000, propagate_forward_time: 82.8738479614, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8988418579
validation[          , genome  6371] predictions:    6487/  10000 (64.87%), best:    6641/10000 (66.41%), error:     10212.01758, best error:      9956.67383 on epoch:    40, epoch:   45/50, mu: 0.9352483749, learning_rate: 0.0000162174, weight_decay: 0.0000158463

epoch time: 1159.9270019531s, input_fired_time: 18.3275909424, output_fired_time: 4.9357042313, propagate_forward_time: 413.8521423340, propagate_backward_time: 702.3336791992, weight_update_time: 0.3091780543, other_time: 20.4778442383
epoch time: 87.6378479004s, input_fired_time: 1.0256141424, output_fired_time: 0.0000000000, propagate_forward_time: 82.6838150024, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9284210205
validation[          , genome  6371] predictions:    6609/  10000 (66.09%), best:    6641/10000 (66.41%), error:      9974.49219, best error:      9956.67383 on epoch:    40, epoch:   46/50, mu: 0.9378792048, learning_rate: 0.0000149203, weight_decay: 0.0000146742

epoch time: 1149.8104248047s, input_fired_time: 18.2301712036, output_fired_time: 4.8575124741, propagate_forward_time: 409.8005676270, propagate_backward_time: 695.7481079102, weight_update_time: 0.2927389741, other_time: 21.1740112305
epoch time: 87.3546295166s, input_fired_time: 1.0272654295, output_fired_time: 0.0000000000, propagate_forward_time: 82.4702453613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8571166992
validation[          , genome  6371] predictions:    6632/  10000 (66.32%), best:    6632/10000 (66.32%), error:      9891.79883, best error:      9891.79883 on epoch:    47, epoch:   47/50, mu: 0.9403836131, learning_rate: 0.0000137270, weight_decay: 0.0000135888

epoch time: 1149.5240478516s, input_fired_time: 18.2407341003, output_fired_time: 4.8746290207, propagate_forward_time: 409.6575317383, propagate_backward_time: 695.9300537109, weight_update_time: 0.2962021232, other_time: 20.8211059570
epoch time: 87.4011840820s, input_fired_time: 1.0293641090, output_fired_time: 0.0000000000, propagate_forward_time: 82.4924697876, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8793487549
validation[          , genome  6371] predictions:    6635/  10000 (66.35%), best:    6632/10000 (66.32%), error:      9944.47363, best error:      9891.79883 on epoch:    47, epoch:   48/50, mu: 0.9427676797, learning_rate: 0.0000126291, weight_decay: 0.0000125837

epoch time: 1150.5384521484s, input_fired_time: 18.2545528412, output_fired_time: 4.8653192520, propagate_forward_time: 409.6050415039, propagate_backward_time: 697.1302490234, weight_update_time: 0.3000155985, other_time: 20.6832885742
epoch time: 87.6065368652s, input_fired_time: 1.0302944183, output_fired_time: 0.0000000000, propagate_forward_time: 82.6895751953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8866653442
validation[          , genome  6371] predictions:    6603/  10000 (66.03%), best:    6632/10000 (66.32%), error:      9939.00879, best error:      9891.79883 on epoch:    47, epoch:   49/50, mu: 0.9450371861, learning_rate: 0.0000116191, weight_decay: 0.0000116529

epoch time: 1150.2725830078s, input_fired_time: 18.2920379639, output_fired_time: 4.8715543747, propagate_forward_time: 409.6109313965, propagate_backward_time: 696.6345825195, weight_update_time: 0.2971815169, other_time: 20.8634643555
epoch time: 87.4601364136s, input_fired_time: 1.0250747204, output_fired_time: 0.0000000000, propagate_forward_time: 82.4362716675, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9987869263
validation[          , genome  6371] predictions:    6554/  10000 (65.54%), best:    6632/10000 (66.32%), error:     10078.63184, best error:      9891.79883 on epoch:    47, epoch:   50/50, mu: 0.9471976161, learning_rate: 0.0000106898, weight_decay: 0.0000107910

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 433.8759765625s, input_fired_time: 5.1069622040, output_fired_time: 0.0000000000, propagate_forward_time: 409.0378417969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.7311706543
best training[          , genome  6371] predictions:   35672/  50000 (71.34%), best:    6632/10000 (66.32%), error:     43073.14063, best error:      9891.79883 on epoch:    47, epoch:   51/50, mu: 0.9492542744, learning_rate: 0.0000098348, weight_decay: 0.0000099928
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 87.4531173706s, input_fired_time: 1.0260245800, output_fired_time: 0.0000000000, propagate_forward_time: 82.4395980835, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9874954224
   testing[          , genome  6371] predictions:    6632/  10000 (66.32%), best:    6632/10000 (66.32%), error:      9891.79883, best error:      9891.79883 on epoch:    47, epoch:   51/50, mu: 0.9492542744, learning_rate: 0.0000098348, weight_decay: 0.0000099928
backpropagation finished successfully!
10:34:02 (10724): called boinc_finish(0)

</stderr_txt>
]]>