Nameexact_genome_1542031203_188_5633_1
Workunit4082554
Created12 Nov 2018, 20:56:01 UTC
Sent12 Nov 2018, 21:06:45 UTC
Report deadline3 Feb 2019, 23:15:28 UTC
Received23 Dec 2018, 3:47:50 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64195
Run time1 days 7 hours 17 min 10 sec
CPU time1 days 6 hours 58 min 30 sec
Validate stateValid
Credit20,772.35
Device peak FLOPS3.83 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size536.61 MB
Peak swap size556.24 MB
Peak disk usage10.61 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
0, other_time: 1.6068115234
validation[          , genome  5633] predictions:    6145/  10000 (61.45%), best:    6145/10000 (61.45%), error:     11052.07031, best error:     11052.07031 on epoch:    28, epoch:   28/100, mu: 0.8936323524, learning_rate: 0.0001070187, weight_decay: 0.0000934111

epoch time: 1040.6990966797s, input_fired_time: 15.3676757813, output_fired_time: 6.1416792870, propagate_forward_time: 386.8375244141, propagate_backward_time: 621.9553833008, weight_update_time: 0.3499194682, other_time: 10.3968505859
epoch time: 87.9687271118s, input_fired_time: 1.0859636068, output_fired_time: 0.0000000000, propagate_forward_time: 85.1393737793, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7433929443
validation[          , genome  5633] predictions:    6072/  10000 (60.72%), best:    6145/10000 (61.45%), error:     11233.24512, best error:     11052.07031 on epoch:    28, epoch:   29/100, mu: 0.8990266919, learning_rate: 0.0000998750, weight_decay: 0.0000882829

epoch time: 1139.2717285156s, input_fired_time: 15.7415113449, output_fired_time: 6.5997796059, propagate_forward_time: 426.5383911133, propagate_backward_time: 679.6214599609, weight_update_time: 0.3732643127, other_time: 10.7706909180
epoch time: 82.2655181885s, input_fired_time: 1.0475381613, output_fired_time: 0.0000000000, propagate_forward_time: 79.5105438232, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7074356079
validation[          , genome  5633] predictions:    6033/  10000 (60.33%), best:    6145/10000 (61.45%), error:     11480.87988, best error:     11052.07031 on epoch:    28, epoch:   30/100, mu: 0.9041190743, learning_rate: 0.0000932082, weight_decay: 0.0000834361

epoch time: 1129.3946533203s, input_fired_time: 15.6997890472, output_fired_time: 6.5307664871, propagate_forward_time: 423.5386047363, propagate_backward_time: 672.9135742188, weight_update_time: 0.3699019551, other_time: 10.7119140625
epoch time: 74.1955032349s, input_fired_time: 0.9837434888, output_fired_time: 0.0000000000, propagate_forward_time: 71.5732040405, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6385574341
validation[          , genome  5633] predictions:    5925/  10000 (59.25%), best:    6145/10000 (61.45%), error:     11586.96973, best error:     11052.07031 on epoch:    28, epoch:   31/100, mu: 0.9089264274, learning_rate: 0.0000869864, weight_decay: 0.0000788555

epoch time: 1136.1287841797s, input_fired_time: 15.7688035965, output_fired_time: 6.5507202148, propagate_forward_time: 426.8905029297, propagate_backward_time: 676.0885009766, weight_update_time: 0.3739727437, other_time: 10.8303222656
epoch time: 106.9378051758s, input_fired_time: 1.2665692568, output_fired_time: 0.0000000000, propagate_forward_time: 103.6943359375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9768981934
validation[          , genome  5633] predictions:    6114/  10000 (61.14%), best:    6114/10000 (61.14%), error:     11045.45801, best error:     11045.45801 on epoch:    32, epoch:   32/100, mu: 0.9134646654, learning_rate: 0.0000811798, weight_decay: 0.0000745263

epoch time: 1241.9161376953s, input_fired_time: 16.2324504852, output_fired_time: 7.1391563416, propagate_forward_time: 471.4558715820, propagate_backward_time: 735.8317260742, weight_update_time: 0.4005070031, other_time: 11.2569580078
epoch time: 93.8866958618s, input_fired_time: 1.1613202095, output_fired_time: 0.0000000000, propagate_forward_time: 90.8645248413, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8608474731
validation[          , genome  5633] predictions:    6142/  10000 (61.42%), best:    6114/10000 (61.14%), error:     11167.55078, best error:     11045.45801 on epoch:    32, epoch:   33/100, mu: 0.9177488685, learning_rate: 0.0000757609, weight_decay: 0.0000704349

epoch time: 1264.7873535156s, input_fired_time: 16.3707923889, output_fired_time: 7.1888022423, propagate_forward_time: 483.7542114258, propagate_backward_time: 746.0655517578, weight_update_time: 0.4071662426, other_time: 11.4078979492
epoch time: 68.0629653931s, input_fired_time: 0.9358081222, output_fired_time: 0.0000000000, propagate_forward_time: 65.5670089722, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5601501465
validation[          , genome  5633] predictions:    6232/  10000 (62.32%), best:    6232/10000 (62.32%), error:     10935.73926, best error:     10935.73926 on epoch:    34, epoch:   34/100, mu: 0.9217932224, learning_rate: 0.0000707038, weight_decay: 0.0000665680

epoch time: 1255.2586669922s, input_fired_time: 16.3007469177, output_fired_time: 7.1476144791, propagate_forward_time: 475.9762878418, propagate_backward_time: 744.4960937500, weight_update_time: 0.4033203125, other_time: 11.3378906250
epoch time: 107.7896423340s, input_fired_time: 1.2663247585, output_fired_time: 0.0000000000, propagate_forward_time: 104.5642166138, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9590988159
validation[          , genome  5633] predictions:    6132/  10000 (61.32%), best:    6232/10000 (62.32%), error:     11084.38867, best error:     10935.73926 on epoch:    34, epoch:   35/100, mu: 0.9256111979, learning_rate: 0.0000659841, weight_decay: 0.0000629134

epoch time: 1182.7855224609s, input_fired_time: 16.0076599121, output_fired_time: 6.8186511993, propagate_forward_time: 445.5894165039, propagate_backward_time: 703.3339843750, weight_update_time: 0.3859051764, other_time: 11.0358276367
epoch time: 102.6855316162s, input_fired_time: 1.2318139076, output_fired_time: 0.0000000000, propagate_forward_time: 99.5511779785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9025421143
validation[          , genome  5633] predictions:    6119/  10000 (61.19%), best:    6232/10000 (62.32%), error:     11136.17383, best error:     10935.73926 on epoch:    34, epoch:   36/100, mu: 0.9292154908, learning_rate: 0.0000615796, weight_decay: 0.0000594595

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5633.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5633_1_r956389885_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50648
read mu: 0.932618
read mu_delta: 0.944023
read initial_learning_rate: 0.000740526
read learning_rate: 5.7469e-05
read learning_rate_delta: 0.933248
read initial_weight_decay: 0.000453962
read weight_decay: 5.61952e-05
read weight_decay_delta: 0.9451
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0960096
read input_dropout_probability: 0.00113252
read hidden_dropout_probability: 0.0877287
read velocity_reset: 1788
read epoch: 37
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 34
read number_validation_images: 10000
read best_validation_predictions: 6232
read best_validation_error: 10935.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5633
read normal distribution: '0 1.67819249629974 0.374094843864441'
generator_str: '617617808'
read generator: 617617808
read generated_by_map:
1 change_size_x 1
reading 68 nodes.
reading 292 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1829.54809570313s, input_fired_time: 20.8503837585449, output_fired_time: 10.5958280563354, propagate_forward_time: 703.780700683594, propagate_backward_time: 1079.36267089844, weight_update_time: 0.46889540553093, other_time: 14.9586181640625
epoch time: 139.554656982422s, input_fired_time: 1.61333644390106, output_fired_time: 0, propagate_forward_time: 135.342803955078, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.5985107421875
validation[          , genome  5633] predictions:    6236/  10000 (62.36%), best:    6236/10000 (62.36%), error:     10931.69043, best error:     10931.69043 on epoch:    37, epoch:   37/100, mu: 0.9326180220, learning_rate: 0.0000574690, weight_decay: 0.0000561952

epoch time: 1765.4409179688s, input_fired_time: 21.3134174347, output_fired_time: 10.0043258667, propagate_forward_time: 677.9554443359, propagate_backward_time: 1041.3774414063, weight_update_time: 0.5789981484, other_time: 14.7902832031
epoch time: 140.7192535400s, input_fired_time: 1.5294389725, output_fired_time: 0.0000000000, propagate_forward_time: 136.7147674561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4750518799
validation[          , genome  5633] predictions:    6210/  10000 (62.10%), best:    6236/10000 (62.36%), error:     11029.45898, best error:     10931.69043 on epoch:    37, epoch:   38/100, mu: 0.9358300567, learning_rate: 0.0000536329, weight_decay: 0.0000531101

epoch time: 1739.0395507813s, input_fired_time: 20.7488937378, output_fired_time: 9.8524675369, propagate_forward_time: 668.1560058594, propagate_backward_time: 1025.9846191406, weight_update_time: 0.5473836660, other_time: 14.2976074219
epoch time: 137.6441040039s, input_fired_time: 1.5471018553, output_fired_time: 0.0000000000, propagate_forward_time: 133.4973602295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5996398926
validation[          , genome  5633] predictions:    6163/  10000 (61.63%), best:    6236/10000 (62.36%), error:     11029.85547, best error:     10931.69043 on epoch:    37, epoch:   39/100, mu: 0.9388623238, learning_rate: 0.0000500528, weight_decay: 0.0000501943

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5633.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5633_1_r956389885_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50648
read mu: 0.941725
read mu_delta: 0.944023
read initial_learning_rate: 0.000740526
read learning_rate: 4.67117e-05
read learning_rate_delta: 0.933248
read initial_weight_decay: 0.000453962
read weight_decay: 4.74387e-05
read weight_decay_delta: 0.9451
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0960096
read input_dropout_probability: 0.00113252
read hidden_dropout_probability: 0.0877287
read velocity_reset: 1788
read epoch: 40
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 37
read number_validation_images: 10000
read best_validation_predictions: 6236
read best_validation_error: 10931.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5633
read normal distribution: '0 1.67819249629974 0.374094843864441'
generator_str: '1114605271'
read generator: 1114605271
read generated_by_map:
1 change_size_x 1
reading 68 nodes.
reading 292 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1245.64929199219s, input_fired_time: 16.4158058166504, output_fired_time: 7.3258752822876, propagate_forward_time: 473.555236816406, propagate_backward_time: 736.597351074219, weight_update_time: 0.404980689287186, other_time: 11.7550048828125
epoch time: 100.168312072754s, input_fired_time: 1.22678852081299, output_fired_time: 0, propagate_forward_time: 96.6934356689453, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.24808502197266
validation[          , genome  5633] predictions:    6217/  10000 (62.17%), best:    6217/10000 (62.17%), error:     10820.02344, best error:     10820.02344 on epoch:    40, epoch:   40/100, mu: 0.9417248368, learning_rate: 0.0000467117, weight_decay: 0.0000474387

epoch time: 1252.9147949219s, input_fired_time: 16.4468097687, output_fired_time: 7.3667006493, propagate_forward_time: 476.9367675781, propagate_backward_time: 740.3998413086, weight_update_time: 0.4114985168, other_time: 11.7647094727
epoch time: 86.4142303467s, input_fired_time: 1.1035400629, output_fired_time: 0.0000000000, propagate_forward_time: 83.3452529907, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9654388428
validation[          , genome  5633] predictions:    6270/  10000 (62.70%), best:    6270/10000 (62.70%), error:     10732.22656, best error:     10732.22656 on epoch:    41, epoch:   41/100, mu: 0.9444271326, learning_rate: 0.0000435936, weight_decay: 0.0000448343

epoch time: 1120.5124511719s, input_fired_time: 16.1011600494, output_fired_time: 6.8024425507, propagate_forward_time: 420.5409240723, propagate_backward_time: 664.6085815430, weight_update_time: 0.3945507705, other_time: 12.4592895508
epoch time: 59.7152252197s, input_fired_time: 0.8936008811, output_fired_time: 0.0000000000, propagate_forward_time: 56.7473449707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0742797852
validation[          , genome  5633] predictions:    6231/  10000 (62.31%), best:    6270/10000 (62.70%), error:     10886.40234, best error:     10732.22656 on epoch:    41, epoch:   42/100, mu: 0.9469781518, learning_rate: 0.0000406836, weight_decay: 0.0000423729

epoch time: 793.2420654297s, input_fired_time: 14.6440496445, output_fired_time: 5.0699458122, propagate_forward_time: 282.5205078125, propagate_backward_time: 478.7364807129, weight_update_time: 0.3258286119, other_time: 12.2710876465
epoch time: 62.2055587769s, input_fired_time: 1.2851426601, output_fired_time: 0.0000000000, propagate_forward_time: 57.4679069519, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4525108337
validation[          , genome  5633] predictions:    6209/  10000 (62.09%), best:    6270/10000 (62.70%), error:     10828.63086, best error:     10732.22656 on epoch:    41, epoch:   43/100, mu: 0.9493863583, learning_rate: 0.0000379679, weight_decay: 0.0000400466

epoch time: 814.8620605469s, input_fired_time: 16.5238456726, output_fired_time: 6.9684023857, propagate_forward_time: 286.6782531738, propagate_backward_time: 485.2169189453, weight_update_time: 0.5760306120, other_time: 19.4746398926
epoch time: 62.1498031616s, input_fired_time: 1.2720353603, output_fired_time: 0.0000000000, propagate_forward_time: 57.4468154907, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4309539795
validation[          , genome  5633] predictions:    6166/  10000 (61.66%), best:    6270/10000 (62.70%), error:     10884.92188, best error:     10732.22656 on epoch:    41, epoch:   44/100, mu: 0.9516597986, learning_rate: 0.0000354335, weight_decay: 0.0000378481

epoch time: 837.5781860352s, input_fired_time: 19.0700778961, output_fired_time: 9.3758859634, propagate_forward_time: 292.2687072754, propagate_backward_time: 493.5179748535, weight_update_time: 0.7774464488, other_time: 23.3455505371
epoch time: 68.0474853516s, input_fired_time: 2.5423352718, output_fired_time: 0.0000000000, propagate_forward_time: 60.1686210632, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3365287781
validation[          , genome  5633] predictions:    6218/  10000 (62.18%), best:    6270/10000 (62.70%), error:     10920.98535, best error:     10732.22656 on epoch:    41, epoch:   45/100, mu: 0.9538059831, learning_rate: 0.0000330682, weight_decay: 0.0000357702

epoch time: 867.5812377930s, input_fired_time: 22.6821327209, output_fired_time: 12.7715768814, propagate_forward_time: 299.0938720703, propagate_backward_time: 504.3960571289, weight_update_time: 1.0256291628, other_time: 28.6375732422
epoch time: 68.0443878174s, input_fired_time: 2.5456597805, output_fired_time: 0.0000000000, propagate_forward_time: 60.1500511169, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3486747742
validation[          , genome  5633] predictions:    6256/  10000 (62.56%), best:    6270/10000 (62.70%), error:     10757.63770, best error:     10732.22656 on epoch:    41, epoch:   46/100, mu: 0.9558320045, learning_rate: 0.0000308609, weight_decay: 0.0000338064

epoch time: 876.4715576172s, input_fired_time: 23.5974731445, output_fired_time: 13.6492443085, propagate_forward_time: 301.0715026855, propagate_backward_time: 508.0151062012, weight_update_time: 1.0966078043, other_time: 30.1382141113
epoch time: 67.8902130127s, input_fired_time: 2.5201306343, output_fired_time: 0.0000000000, propagate_forward_time: 60.0412445068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3288345337
validation[          , genome  5633] predictions:    6248/  10000 (62.48%), best:    6270/10000 (62.70%), error:     10769.44922, best error:     10732.22656 on epoch:    41, epoch:   47/100, mu: 0.9577445984, learning_rate: 0.0000288008, weight_decay: 0.0000319505

epoch time: 994.8319702148s, input_fired_time: 22.8896942139, output_fired_time: 12.6522741318, propagate_forward_time: 350.4292602539, propagate_backward_time: 577.7299194336, weight_update_time: 1.0274411440, other_time: 31.1307983398
epoch time: 80.2263565063s, input_fired_time: 1.7444937229, output_fired_time: 0.0000000000, propagate_forward_time: 74.3983535767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0835113525
validation[          , genome  5633] predictions:    6269/  10000 (62.69%), best:    6270/10000 (62.70%), error:     10757.64551, best error:     10732.22656 on epoch:    41, epoch:   48/100, mu: 0.9595501423, learning_rate: 0.0000268783, weight_decay: 0.0000301964

epoch time: 912.1912841797s, input_fired_time: 19.8472938538, output_fired_time: 9.9084672928, propagate_forward_time: 321.6445617676, propagate_backward_time: 535.2022705078, weight_update_time: 0.8162968755, other_time: 25.5887451172
epoch time: 77.1366729736s, input_fired_time: 1.8707211018, output_fired_time: 0.0000000000, propagate_forward_time: 70.2346649170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0312881470
validation[          , genome  5633] predictions:    6313/  10000 (63.13%), best:    6313/10000 (63.13%), error:     10653.27344, best error:     10653.27344 on epoch:    49, epoch:   49/100, mu: 0.9612545967, learning_rate: 0.0000250841, weight_decay: 0.0000285386

epoch time: 1009.4988403320s, input_fired_time: 20.9824295044, output_fired_time: 10.9133234024, propagate_forward_time: 359.1973266602, propagate_backward_time: 589.4427490234, weight_update_time: 0.8925734758, other_time: 28.9630126953
epoch time: 65.5703430176s, input_fired_time: 1.8901478052, output_fired_time: 0.0000000000, propagate_forward_time: 59.2758369446, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4043579102
validation[          , genome  5633] predictions:    6229/  10000 (62.29%), best:    6313/10000 (63.13%), error:     10801.12402, best error:     10653.27344 on epoch:    49, epoch:   50/100, mu: 0.9628636837, learning_rate: 0.0000234097, weight_decay: 0.0000269718

epoch time: 957.0355224609s, input_fired_time: 20.3819541931, output_fired_time: 10.4706802368, propagate_forward_time: 338.9407348633, propagate_backward_time: 560.6375732422, weight_update_time: 0.8555462360, other_time: 26.6045532227
epoch time: 65.8990936279s, input_fired_time: 1.5238146782, output_fired_time: 0.0000000000, propagate_forward_time: 60.4810791016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8942031860
validation[          , genome  5633] predictions:    6288/  10000 (62.88%), best:    6313/10000 (63.13%), error:     10735.61621, best error:     10653.27344 on epoch:    49, epoch:   51/100, mu: 0.9643827081, learning_rate: 0.0000218471, weight_decay: 0.0000254911

epoch time: 1012.9138793945s, input_fired_time: 20.4686832428, output_fired_time: 10.4886035919, propagate_forward_time: 360.8749389648, propagate_backward_time: 593.2218017578, weight_update_time: 0.8611826301, other_time: 27.8598632813
epoch time: 80.1657333374s, input_fired_time: 1.8918998241, output_fired_time: 0.0000000000, propagate_forward_time: 73.5573577881, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7164764404
validation[          , genome  5633] predictions:    6226/  10000 (62.26%), best:    6313/10000 (63.13%), error:     10862.43555, best error:     10653.27344 on epoch:    49, epoch:   52/100, mu: 0.9658166766, learning_rate: 0.0000203888, weight_decay: 0.0000240916

epoch time: 993.5155029297s, input_fired_time: 20.0053901672, output_fired_time: 9.9505615234, propagate_forward_time: 354.8129882813, propagate_backward_time: 582.1416625977, weight_update_time: 0.8176701069, other_time: 26.6049194336
epoch time: 93.4681167603s, input_fired_time: 2.2336542606, output_fired_time: 0.0000000000, propagate_forward_time: 85.6050491333, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6294097900
validation[          , genome  5633] predictions:    6370/  10000 (63.70%), best:    6370/10000 (63.70%), error:     10548.43359, best error:     10548.43359 on epoch:    53, epoch:   53/100, mu: 0.9671703577, learning_rate: 0.0000190278, weight_decay: 0.0000227690

epoch time: 991.4438476563s, input_fired_time: 19.7544574738, output_fired_time: 9.6974258423, propagate_forward_time: 352.5839233398, propagate_backward_time: 583.1243286133, weight_update_time: 0.7940729260, other_time: 26.2836914063
epoch time: 68.3806457520s, input_fired_time: 1.7811961174, output_fired_time: 0.0000000000, propagate_forward_time: 62.4411468506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1583023071
validation[          , genome  5633] predictions:    6216/  10000 (62.16%), best:    6370/10000 (63.70%), error:     10877.02539, best error:     10548.43359 on epoch:    53, epoch:   54/100, mu: 0.9684482813, learning_rate: 0.0000177576, weight_decay: 0.0000215190

epoch time: 1004.3463134766s, input_fired_time: 20.0622863770, output_fired_time: 10.0819959641, propagate_forward_time: 358.5172729492, propagate_backward_time: 588.8126831055, weight_update_time: 0.8383033872, other_time: 26.8721313477
epoch time: 70.2382278442s, input_fired_time: 1.8220511675, output_fired_time: 0.0000000000, propagate_forward_time: 63.9163627625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4998130798
validation[          , genome  5633] predictions:    6307/  10000 (63.07%), best:    6370/10000 (63.70%), error:     10658.99902, best error:     10548.43359 on epoch:    53, epoch:   55/100, mu: 0.9696546793, learning_rate: 0.0000165723, weight_decay: 0.0000203376

epoch time: 1010.5653686523s, input_fired_time: 20.8292999268, output_fired_time: 10.6807966232, propagate_forward_time: 358.8821105957, propagate_backward_time: 591.6643676758, weight_update_time: 0.8749241829, other_time: 28.5088500977
epoch time: 91.2021636963s, input_fired_time: 2.1007592678, output_fired_time: 0.0000000000, propagate_forward_time: 83.6514129639, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4499893188
validation[          , genome  5633] predictions:    6322/  10000 (63.22%), best:    6370/10000 (63.70%), error:     10568.55664, best error:     10548.43359 on epoch:    53, epoch:   56/100, mu: 0.9707935452, learning_rate: 0.0000154660, weight_decay: 0.0000192211

epoch time: 992.1240234375s, input_fired_time: 20.1807041168, output_fired_time: 10.1309738159, propagate_forward_time: 354.9761962891, propagate_backward_time: 580.2661743164, weight_update_time: 0.8323783278, other_time: 26.5699462891
epoch time: 67.0519943237s, input_fired_time: 1.7445497513, output_fired_time: 0.0000000000, propagate_forward_time: 61.0806617737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2267799377
validation[          , genome  5633] predictions:    6313/  10000 (63.13%), best:    6370/10000 (63.70%), error:     10562.30469, best error:     10548.43359 on epoch:    53, epoch:   57/100, mu: 0.9718686342, learning_rate: 0.0000144337, weight_decay: 0.0000181658

epoch time: 1006.2573852539s, input_fired_time: 20.5949497223, output_fired_time: 10.5661964417, propagate_forward_time: 359.9106750488, propagate_backward_time: 587.5772094727, weight_update_time: 0.8766911626, other_time: 27.6083374023
epoch time: 88.3618774414s, input_fired_time: 1.8993813992, output_fired_time: 0.0000000000, propagate_forward_time: 80.7975387573, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6649551392
validation[          , genome  5633] predictions:    6303/  10000 (63.03%), best:    6370/10000 (63.70%), error:     10732.86719, best error:     10548.43359 on epoch:    53, epoch:   58/100, mu: 0.9728835821, learning_rate: 0.0000134702, weight_decay: 0.0000171685

epoch time: 948.7564086914s, input_fired_time: 20.5605201721, output_fired_time: 10.6068992615, propagate_forward_time: 336.3247070313, propagate_backward_time: 554.9337768555, weight_update_time: 0.8622979522, other_time: 26.3305053711
epoch time: 72.8566818237s, input_fired_time: 1.9816112518, output_fired_time: 0.0000000000, propagate_forward_time: 66.1343460083, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7407226563
validation[          , genome  5633] predictions:    6349/  10000 (63.49%), best:    6370/10000 (63.70%), error:     10567.79980, best error:     10548.43359 on epoch:    53, epoch:   59/100, mu: 0.9738417268, learning_rate: 0.0000125710, weight_decay: 0.0000162260

epoch time: 997.1354980469s, input_fired_time: 20.7847194672, output_fired_time: 10.8342695236, propagate_forward_time: 354.6884460449, propagate_backward_time: 584.1579589844, weight_update_time: 0.8720408082, other_time: 26.6700439453
epoch time: 74.0595016479s, input_fired_time: 1.9816120863, output_fired_time: 0.0000000000, propagate_forward_time: 67.0898284912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9880599976
validation[          , genome  5633] predictions:    6308/  10000 (63.08%), best:    6370/10000 (63.70%), error:     10692.17676, best error:     10548.43359 on epoch:    53, epoch:   60/100, mu: 0.9747462273, learning_rate: 0.0000117319, weight_decay: 0.0000153352

epoch time: 1037.0507812500s, input_fired_time: 20.8196258545, output_fired_time: 10.6633520126, propagate_forward_time: 373.5250549316, propagate_backward_time: 603.8214111328, weight_update_time: 0.8557634354, other_time: 28.2213134766
epoch time: 84.0920181274s, input_fired_time: 1.8683984280, output_fired_time: 0.0000000000, propagate_forward_time: 77.2032089233, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0204086304
validation[          , genome  5633] predictions:    6332/  10000 (63.32%), best:    6332/10000 (63.32%), error:     10509.06543, best error:     10509.06543 on epoch:    61, epoch:   61/100, mu: 0.9756000638, learning_rate: 0.0000109488, weight_decay: 0.0000144933

epoch time: 1065.0865478516s, input_fired_time: 19.8689193726, output_fired_time: 9.9014492035, propagate_forward_time: 381.7634887695, propagate_backward_time: 627.2218627930, weight_update_time: 0.8161761165, other_time: 26.3308105469
epoch time: 83.9358291626s, input_fired_time: 1.9084366560, output_fired_time: 0.0000000000, propagate_forward_time: 77.2476196289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7797698975
validation[          , genome  5633] predictions:    6378/  10000 (63.78%), best:    6332/10000 (63.32%), error:     10512.06348, best error:     10509.06543 on epoch:    61, epoch:   62/100, mu: 0.9764060974, learning_rate: 0.0000102179, weight_decay: 0.0000136976

epoch time: 1027.4926757813s, input_fired_time: 19.8615589142, output_fired_time: 9.7977895737, propagate_forward_time: 369.7707519531, propagate_backward_time: 601.0802001953, weight_update_time: 0.8110538721, other_time: 26.9823608398
epoch time: 82.5158538818s, input_fired_time: 1.9658889771, output_fired_time: 0.0000000000, propagate_forward_time: 75.4414901733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1084747314
validation[          , genome  5633] predictions:    6335/  10000 (63.35%), best:    6332/10000 (63.32%), error:     10647.29883, best error:     10509.06543 on epoch:    61, epoch:   63/100, mu: 0.9771670103, learning_rate: 0.0000095358, weight_decay: 0.0000129456

epoch time: 1082.4621582031s, input_fired_time: 20.1897773743, output_fired_time: 10.2074308395, propagate_forward_time: 385.0859985352, propagate_backward_time: 639.5829467773, weight_update_time: 0.8251546025, other_time: 27.3959960938
epoch time: 88.8355789185s, input_fired_time: 1.8311762810, output_fired_time: 0.0000000000, propagate_forward_time: 81.9570541382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0473480225
validation[          , genome  5633] predictions:    6394/  10000 (63.94%), best:    6394/10000 (63.94%), error:     10411.59277, best error:     10411.59277 on epoch:    64, epoch:   64/100, mu: 0.9778853655, learning_rate: 0.0000088993, weight_decay: 0.0000122349

epoch time: 1143.9671630859s, input_fired_time: 20.3793315887, output_fired_time: 10.2627000809, propagate_forward_time: 414.8015136719, propagate_backward_time: 670.5019531250, weight_update_time: 0.8195446134, other_time: 28.0217285156
epoch time: 87.7333221436s, input_fired_time: 1.8659820557, output_fired_time: 0.0000000000, propagate_forward_time: 81.1005935669, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7667465210
validation[          , genome  5633] predictions:    6323/  10000 (63.23%), best:    6394/10000 (63.94%), error:     10557.66797, best error:     10411.59277 on epoch:    64, epoch:   65/100, mu: 0.9785634875, learning_rate: 0.0000083053, weight_decay: 0.0000115632

epoch time: 1144.7799072266s, input_fired_time: 19.7452068329, output_fired_time: 9.7732782364, propagate_forward_time: 415.9130859375, propagate_backward_time: 672.1298217773, weight_update_time: 0.7652261257, other_time: 27.2184448242
epoch time: 89.6851501465s, input_fired_time: 1.4403535128, output_fired_time: 0.0000000000, propagate_forward_time: 83.9561767578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2886199951
validation[          , genome  5633] predictions:    6309/  10000 (63.09%), best:    6394/10000 (63.94%), error:     10627.90820, best error:     10411.59277 on epoch:    64, epoch:   66/100, mu: 0.9792036414, learning_rate: 0.0000077509, weight_decay: 0.0000109284

epoch time: 1101.5760498047s, input_fired_time: 17.4932880402, output_fired_time: 7.8620591164, propagate_forward_time: 405.9827270508, propagate_backward_time: 648.9504394531, weight_update_time: 0.5844344497, other_time: 21.2875366211
epoch time: 83.3675918579s, input_fired_time: 1.3252569437, output_fired_time: 0.0000000000, propagate_forward_time: 78.5335922241, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5087432861
validation[          , genome  5633] predictions:    6329/  10000 (63.29%), best:    6394/10000 (63.94%), error:     10463.96191, best error:     10411.59277 on epoch:    64, epoch:   67/100, mu: 0.9798079729, learning_rate: 0.0000072335, weight_decay: 0.0000103284

epoch time: 1106.0422363281s, input_fired_time: 17.3586101532, output_fired_time: 7.7695331573, propagate_forward_time: 407.7072753906, propagate_backward_time: 652.4249877930, weight_update_time: 0.5856778026, other_time: 20.7817993164
epoch time: 94.3974609375s, input_fired_time: 1.4574950933, output_fired_time: 0.0000000000, propagate_forward_time: 88.7684249878, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1715393066
validation[          , genome  5633] predictions:    6414/  10000 (64.14%), best:    6414/10000 (64.14%), error:     10389.00000, best error:     10389.00000 on epoch:    68, epoch:   68/100, mu: 0.9803785086, learning_rate: 0.0000067506, weight_decay: 0.0000097614

epoch time: 1124.9689941406s, input_fired_time: 17.4962673187, output_fired_time: 7.9062333107, propagate_forward_time: 412.5859069824, propagate_backward_time: 665.6287841797, weight_update_time: 0.5872088075, other_time: 21.3518066406
epoch time: 88.2019042969s, input_fired_time: 1.3923642635, output_fired_time: 0.0000000000, propagate_forward_time: 82.9007034302, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9088363647
validation[          , genome  5633] predictions:    6386/  10000 (63.86%), best:    6386/10000 (63.86%), error:     10377.58301, best error:     10377.58301 on epoch:    69, epoch:   69/100, mu: 0.9809170961, learning_rate: 0.0000063000, weight_decay: 0.0000092255

epoch time: 1148.8872070313s, input_fired_time: 17.7951488495, output_fired_time: 8.1191787720, propagate_forward_time: 426.5659179688, propagate_backward_time: 674.4085693359, weight_update_time: 0.5986155868, other_time: 21.9984130859
epoch time: 91.7242355347s, input_fired_time: 1.4139879942, output_fired_time: 0.0000000000, propagate_forward_time: 86.4583435059, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8519058228
validation[          , genome  5633] predictions:    6418/  10000 (64.18%), best:    6418/10000 (64.18%), error:     10355.40234, best error:     10355.40234 on epoch:    70, epoch:   70/100, mu: 0.9814255238, learning_rate: 0.0000058795, weight_decay: 0.0000087190

epoch time: 1119.8421630859s, input_fired_time: 17.6017246246, output_fired_time: 7.9173378944, propagate_forward_time: 410.8056945801, propagate_backward_time: 662.0096435547, weight_update_time: 0.5960305929, other_time: 21.5078125000
epoch time: 78.7123870850s, input_fired_time: 1.3012359142, output_fired_time: 0.0000000000, propagate_forward_time: 73.9185867310, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4925613403
validation[          , genome  5633] predictions:    6341/  10000 (63.41%), best:    6418/10000 (64.18%), error:     10468.04883, best error:     10355.40234 on epoch:    70, epoch:   71/100, mu: 0.9819055200, learning_rate: 0.0000054870, weight_decay: 0.0000082403

epoch time: 1072.7008056641s, input_fired_time: 17.3109455109, output_fired_time: 7.6055822372, propagate_forward_time: 395.5711364746, propagate_backward_time: 631.6981811523, weight_update_time: 0.5715425014, other_time: 20.5149536133
epoch time: 78.8578414917s, input_fired_time: 1.3227478266, output_fired_time: 0.0000000000, propagate_forward_time: 73.7945480347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7405471802
validation[          , genome  5633] predictions:    6421/  10000 (64.21%), best:    6421/10000 (64.21%), error:     10307.50977, best error:     10307.50977 on epoch:    72, epoch:   72/100, mu: 0.9823586345, learning_rate: 0.0000051207, weight_decay: 0.0000077879

epoch time: 1087.7635498047s, input_fired_time: 17.5877246857, output_fired_time: 7.8913111687, propagate_forward_time: 397.9579772949, propagate_backward_time: 642.8080444336, weight_update_time: 0.6042231917, other_time: 21.5183715820
epoch time: 91.7495346069s, input_fired_time: 1.4907560349, output_fired_time: 0.0000000000, propagate_forward_time: 86.0213317871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2374496460
validation[          , genome  5633] predictions:    6441/  10000 (64.41%), best:    6421/10000 (64.21%), error:     10318.69531, best error:     10307.50977 on epoch:    72, epoch:   73/100, mu: 0.9827863574, learning_rate: 0.0000047789, weight_decay: 0.0000073604

epoch time: 1113.5992431641s, input_fired_time: 17.6068630219, output_fired_time: 7.8982853889, propagate_forward_time: 409.9027099609, propagate_backward_time: 656.2493286133, weight_update_time: 0.6028876305, other_time: 21.9420776367
epoch time: 97.7738037109s, input_fired_time: 1.4892647266, output_fired_time: 0.0000000000, propagate_forward_time: 92.1504211426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1341171265
validation[          , genome  5633] predictions:    6393/  10000 (63.93%), best:    6421/10000 (64.21%), error:     10451.74414, best error:     10307.50977 on epoch:    72, epoch:   74/100, mu: 0.9831901789, learning_rate: 0.0000044599, weight_decay: 0.0000069563

epoch time: 1142.8419189453s, input_fired_time: 17.8704376221, output_fired_time: 8.1051855087, propagate_forward_time: 422.7008056641, propagate_backward_time: 672.0229492188, weight_update_time: 0.6120784283, other_time: 22.1424560547
epoch time: 93.8061599731s, input_fired_time: 1.4484373331, output_fired_time: 0.0000000000, propagate_forward_time: 88.2108383179, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1468811035
validation[          , genome  5633] predictions:    6423/  10000 (64.23%), best:    6421/10000 (64.21%), error:     10327.41602, best error:     10307.50977 on epoch:    72, epoch:   75/100, mu: 0.9835713506, learning_rate: 0.0000041622, weight_decay: 0.0000065744

epoch time: 1186.9348144531s, input_fired_time: 17.9695281982, output_fired_time: 8.3918485641, propagate_forward_time: 435.7894592285, propagate_backward_time: 702.7639770508, weight_update_time: 0.6150252223, other_time: 22.0200805664
epoch time: 94.4349594116s, input_fired_time: 1.4276385307, output_fired_time: 0.0000000000, propagate_forward_time: 89.1996154785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8077087402
validation[          , genome  5633] predictions:    6409/  10000 (64.09%), best:    6421/10000 (64.21%), error:     10315.07129, best error:     10307.50977 on epoch:    72, epoch:   76/100, mu: 0.9839311838, learning_rate: 0.0000038844, weight_decay: 0.0000062135

epoch time: 1164.7406005859s, input_fired_time: 18.0917892456, output_fired_time: 8.3149003983, propagate_forward_time: 432.0692749023, propagate_backward_time: 683.8427734375, weight_update_time: 0.6208036542, other_time: 22.4218139648
epoch time: 94.1203308105s, input_fired_time: 1.5211061239, output_fired_time: 0.0000000000, propagate_forward_time: 88.2313003540, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3679275513
validation[          , genome  5633] predictions:    6440/  10000 (64.40%), best:    6440/10000 (64.40%), error:     10283.31836, best error:     10283.31836 on epoch:    77, epoch:   77/100, mu: 0.9842708707, learning_rate: 0.0000036251, weight_decay: 0.0000058723

epoch time: 1091.7844238281s, input_fired_time: 17.4177951813, output_fired_time: 7.7236623764, propagate_forward_time: 400.1249694824, propagate_backward_time: 646.1164550781, weight_update_time: 0.5701981783, other_time: 20.4014892578
epoch time: 77.1143264771s, input_fired_time: 1.1187838316, output_fired_time: 0.0000000000, propagate_forward_time: 73.3412780762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6542663574
validation[          , genome  5633] predictions:    6455/  10000 (64.55%), best:    6455/10000 (64.55%), error:     10251.48242, best error:     10251.48242 on epoch:    78, epoch:   78/100, mu: 0.9845915437, learning_rate: 0.0000033831, weight_decay: 0.0000055500

epoch time: 1003.8098144531s, input_fired_time: 16.2662296295, output_fired_time: 6.6241040230, propagate_forward_time: 368.4110107422, propagate_backward_time: 595.3072509766, weight_update_time: 0.4689937234, other_time: 17.2012329102
epoch time: 79.8544616699s, input_fired_time: 1.1854417324, output_fired_time: 0.0000000000, propagate_forward_time: 75.2902069092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3788146973
validation[          , genome  5633] predictions:    6360/  10000 (63.60%), best:    6455/10000 (64.55%), error:     10419.20605, best error:     10251.48242 on epoch:    78, epoch:   79/100, mu: 0.9848942757, learning_rate: 0.0000031573, weight_decay: 0.0000052453

epoch time: 952.3828125000s, input_fired_time: 15.9208707809, output_fired_time: 6.2603769302, propagate_forward_time: 347.9515686035, propagate_backward_time: 565.9125366211, weight_update_time: 0.4488083422, other_time: 16.3374633789
epoch time: 83.4866333008s, input_fired_time: 1.2244224548, output_fired_time: 0.0000000000, propagate_forward_time: 79.2121353149, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0500717163
validation[          , genome  5633] predictions:    6425/  10000 (64.25%), best:    6455/10000 (64.55%), error:     10286.21094, best error:     10251.48242 on epoch:    78, epoch:   80/100, mu: 0.9851800799, learning_rate: 0.0000029465, weight_decay: 0.0000049573

epoch time: 1021.1091918945s, input_fired_time: 15.9391622543, output_fired_time: 6.3484730721, propagate_forward_time: 377.0050354004, propagate_backward_time: 607.6937866211, weight_update_time: 0.4080368578, other_time: 14.1227416992
epoch time: 74.7701797485s, input_fired_time: 1.0619534254, output_fired_time: 0.0000000000, propagate_forward_time: 71.3867111206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3215179443
validation[          , genome  5633] predictions:    6464/  10000 (64.64%), best:    6464/10000 (64.64%), error:     10250.85645, best error:     10250.85645 on epoch:    81, epoch:   81/100, mu: 0.9854499102, learning_rate: 0.0000027498, weight_decay: 0.0000046851

epoch time: 1081.7707519531s, input_fired_time: 16.4333267212, output_fired_time: 6.9044866562, propagate_forward_time: 402.5492858887, propagate_backward_time: 640.9143066406, weight_update_time: 0.4314412475, other_time: 14.9692382813
epoch time: 85.6439056396s, input_fired_time: 1.1596639156, output_fired_time: 0.0000000000, propagate_forward_time: 82.2396240234, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2446212769
validation[          , genome  5633] predictions:    6438/  10000 (64.38%), best:    6464/10000 (64.64%), error:     10261.98633, best error:     10250.85645 on epoch:    81, epoch:   82/100, mu: 0.9857046008, learning_rate: 0.0000025663, weight_decay: 0.0000044279

epoch time: 960.2050170898s, input_fired_time: 15.6080751419, output_fired_time: 6.0266923904, propagate_forward_time: 352.7170715332, propagate_backward_time: 573.0921020508, weight_update_time: 0.3726810515, other_time: 12.7610473633
epoch time: 79.3152313232s, input_fired_time: 1.0767735243, output_fired_time: 0.0000000000, propagate_forward_time: 76.2725143433, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9659423828
validation[          , genome  5633] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10249.20020, best error:     10249.20020 on epoch:    83, epoch:   83/100, mu: 0.9859450459, learning_rate: 0.0000023950, weight_decay: 0.0000041848

epoch time: 1033.3543701172s, input_fired_time: 15.9755926132, output_fired_time: 6.4501209259, propagate_forward_time: 384.4103088379, propagate_backward_time: 613.9455566406, weight_update_time: 0.3881166875, other_time: 12.5727539063
epoch time: 77.4990158081s, input_fired_time: 1.0608229637, output_fired_time: 0.0000000000, propagate_forward_time: 74.5767135620, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8614807129
validation[          , genome  5633] predictions:    6474/  10000 (64.74%), best:    6474/10000 (64.74%), error:     10200.04004, best error:     10200.04004 on epoch:    84, epoch:   84/100, mu: 0.9861720204, learning_rate: 0.0000022351, weight_decay: 0.0000039551

epoch time: 1005.2794799805s, input_fired_time: 15.7013359070, output_fired_time: 6.1889362335, propagate_forward_time: 373.2915039063, propagate_backward_time: 598.6877441406, weight_update_time: 0.3683587909, other_time: 11.4099121094
epoch time: 70.9455718994s, input_fired_time: 1.0163363218, output_fired_time: 0.0000000000, propagate_forward_time: 68.1500473022, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7791900635
validation[          , genome  5633] predictions:    6454/  10000 (64.54%), best:    6474/10000 (64.74%), error:     10239.32227, best error:     10200.04004 on epoch:    84, epoch:   85/100, mu: 0.9863862991, learning_rate: 0.0000020859, weight_decay: 0.0000037380

epoch time: 1082.6469726563s, input_fired_time: 16.1304092407, output_fired_time: 6.6707262993, propagate_forward_time: 404.7978210449, propagate_backward_time: 642.7813110352, weight_update_time: 0.3973315358, other_time: 12.2666625977
epoch time: 64.7047348022s, input_fired_time: 0.9588492513, output_fired_time: 0.0000000000, propagate_forward_time: 62.0347557068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7111282349
validation[          , genome  5633] predictions:    6477/  10000 (64.77%), best:    6474/10000 (64.74%), error:     10206.96582, best error:     10200.04004 on epoch:    84, epoch:   86/100, mu: 0.9865885973, learning_rate: 0.0000019467, weight_decay: 0.0000035327

epoch time: 1035.1920166016s, input_fired_time: 15.8329410553, output_fired_time: 6.3868131638, propagate_forward_time: 387.1123962402, propagate_backward_time: 614.5546875000, weight_update_time: 0.3831880391, other_time: 11.3051757813
epoch time: 79.2718658447s, input_fired_time: 1.0930805206, output_fired_time: 0.0000000000, propagate_forward_time: 76.3440246582, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8347625732
validation[          , genome  5633] predictions:    6410/  10000 (64.10%), best:    6474/10000 (64.74%), error:     10306.89355, best error:     10200.04004 on epoch:    84, epoch:   87/100, mu: 0.9867795706, learning_rate: 0.0000018167, weight_decay: 0.0000033388

epoch time: 1114.5383300781s, input_fired_time: 16.2994899750, output_fired_time: 6.8901009560, propagate_forward_time: 419.2508850098, propagate_backward_time: 660.0032958984, weight_update_time: 0.4114877582, other_time: 12.0946044922
epoch time: 91.5859756470s, input_fired_time: 1.2078547478, output_fired_time: 0.0000000000, propagate_forward_time: 88.4235534668, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9545669556
validation[          , genome  5633] predictions:    6458/  10000 (64.58%), best:    6474/10000 (64.74%), error:     10212.37305, best error:     10200.04004 on epoch:    84, epoch:   88/100, mu: 0.9869598150, learning_rate: 0.0000016955, weight_decay: 0.0000031555

epoch time: 1098.2225341797s, input_fired_time: 16.3440399170, output_fired_time: 6.8885846138, propagate_forward_time: 410.9155883789, propagate_backward_time: 651.3965454102, weight_update_time: 0.4124169052, other_time: 12.6778564453
epoch time: 73.4038619995s, input_fired_time: 1.0191013813, output_fired_time: 0.0000000000, propagate_forward_time: 70.7054748535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6792831421
validation[          , genome  5633] predictions:    6464/  10000 (64.64%), best:    6474/10000 (64.74%), error:     10210.53418, best error:     10200.04004 on epoch:    84, epoch:   89/100, mu: 0.9871299863, learning_rate: 0.0000015823, weight_decay: 0.0000029823

epoch time: 978.4614257813s, input_fired_time: 15.4648056030, output_fired_time: 5.9932022095, propagate_forward_time: 364.3449096680, propagate_backward_time: 582.2463378906, weight_update_time: 0.3604328632, other_time: 10.4121704102
epoch time: 118.8123245239s, input_fired_time: 1.4846479893, output_fired_time: 0.0000000000, propagate_forward_time: 114.2416229248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0860519409
validation[          , genome  5633] predictions:    6475/  10000 (64.75%), best:    6474/10000 (64.74%), error:     10235.85059, best error:     10200.04004 on epoch:    84, epoch:   90/100, mu: 0.9872906208, learning_rate: 0.0000014767, weight_decay: 0.0000028185

epoch time: 926.6259155273s, input_fired_time: 15.2949295044, output_fired_time: 5.7583994865, propagate_forward_time: 340.7182922363, propagate_backward_time: 554.3996582031, weight_update_time: 0.3450464606, other_time: 10.4545898438
epoch time: 62.1031646729s, input_fired_time: 0.9203431606, output_fired_time: 0.0000000000, propagate_forward_time: 59.6755638123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5072593689
validation[          , genome  5633] predictions:    6449/  10000 (64.49%), best:    6474/10000 (64.74%), error:     10287.66797, best error:     10200.04004 on epoch:    84, epoch:   91/100, mu: 0.9874422550, learning_rate: 0.0000013781, weight_decay: 0.0000026638

epoch time: 896.3250732422s, input_fired_time: 15.0938787460, output_fired_time: 5.5885977745, propagate_forward_time: 329.1531066895, propagate_backward_time: 536.6448974609, weight_update_time: 0.3369773626, other_time: 9.8446044922
epoch time: 113.8524398804s, input_fired_time: 1.3590401411, output_fired_time: 0.0000000000, propagate_forward_time: 110.4797210693, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0136795044
validation[          , genome  5633] predictions:    6439/  10000 (64.39%), best:    6474/10000 (64.74%), error:     10216.63867, best error:     10200.04004 on epoch:    84, epoch:   92/100, mu: 0.9875854254, learning_rate: 0.0000012861, weight_decay: 0.0000025176

epoch time: 874.0518188477s, input_fired_time: 14.9771890640, output_fired_time: 5.4452390671, propagate_forward_time: 319.4271545410, propagate_backward_time: 524.4248657227, weight_update_time: 0.3287526071, other_time: 9.7774047852
epoch time: 105.1842498779s, input_fired_time: 1.3274964094, output_fired_time: 0.0000000000, propagate_forward_time: 101.8233718872, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0333786011
validation[          , genome  5633] predictions:    6465/  10000 (64.65%), best:    6465/10000 (64.65%), error:     10193.96484, best error:     10193.96484 on epoch:    93, epoch:   93/100, mu: 0.9877206087, learning_rate: 0.0000012003, weight_decay: 0.0000023793

epoch time: 852.2151489258s, input_fired_time: 14.9022598267, output_fired_time: 5.3606686592, propagate_forward_time: 309.9445190430, propagate_backward_time: 512.3571777344, weight_update_time: 0.3246648312, other_time: 9.6505126953
epoch time: 78.6546249390s, input_fired_time: 1.0568008423, output_fired_time: 0.0000000000, propagate_forward_time: 75.9336929321, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6641311646
validation[          , genome  5633] predictions:    6488/  10000 (64.88%), best:    6488/10000 (64.88%), error:     10192.81641, best error:     10192.81641 on epoch:    94, epoch:   94/100, mu: 0.9878482223, learning_rate: 0.0000011201, weight_decay: 0.0000022487

epoch time: 994.1652221680s, input_fired_time: 15.5387191772, output_fired_time: 6.1233444214, propagate_forward_time: 370.4534606934, propagate_backward_time: 591.7102661133, weight_update_time: 0.3690818846, other_time: 10.3394165039
epoch time: 80.3546981812s, input_fired_time: 1.0974376202, output_fired_time: 0.0000000000, propagate_forward_time: 77.5108947754, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7463684082
validation[          , genome  5633] predictions:    6481/  10000 (64.81%), best:    6488/10000 (64.88%), error:     10217.82227, best error:     10192.81641 on epoch:    94, epoch:   95/100, mu: 0.9879686832, learning_rate: 0.0000010454, weight_decay: 0.0000021253

epoch time: 917.7308959961s, input_fired_time: 15.1604299545, output_fired_time: 5.6519122124, propagate_forward_time: 337.7817382813, propagate_backward_time: 549.2337646484, weight_update_time: 0.3445691764, other_time: 9.9030761719
epoch time: 62.9121665955s, input_fired_time: 0.9323251843, output_fired_time: 0.0000000000, propagate_forward_time: 60.4388351440, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5410079956
validation[          , genome  5633] predictions:    6482/  10000 (64.82%), best:    6482/10000 (64.82%), error:     10171.19824, best error:     10171.19824 on epoch:    96, epoch:   96/100, mu: 0.9880824089, learning_rate: 0.0000009756, weight_decay: 0.0000020086

epoch time: 973.3178710938s, input_fired_time: 15.5341081619, output_fired_time: 6.1348195076, propagate_forward_time: 363.4622497559, propagate_backward_time: 577.8365478516, weight_update_time: 0.3743767142, other_time: 10.3500976563
epoch time: 78.4989318848s, input_fired_time: 1.0864106417, output_fired_time: 0.0000000000, propagate_forward_time: 75.6837081909, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7288131714
validation[          , genome  5633] predictions:    6447/  10000 (64.47%), best:    6482/10000 (64.82%), error:     10219.18945, best error:     10171.19824 on epoch:    96, epoch:   97/100, mu: 0.9881897569, learning_rate: 0.0000009105, weight_decay: 0.0000018983

epoch time: 858.2385864258s, input_fired_time: 14.9215078354, output_fired_time: 5.3825058937, propagate_forward_time: 313.2259216309, propagate_backward_time: 515.0355834961, weight_update_time: 0.3325303793, other_time: 9.6730346680
epoch time: 65.4944458008s, input_fired_time: 0.9557851553, output_fired_time: 0.0000000000, propagate_forward_time: 62.9851608276, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5534973145
validation[          , genome  5633] predictions:    6471/  10000 (64.71%), best:    6471/10000 (64.71%), error:     10169.54980, best error:     10169.54980 on epoch:    98, epoch:   98/100, mu: 0.9882910848, learning_rate: 0.0000008497, weight_decay: 0.0000017941

epoch time: 887.9212646484s, input_fired_time: 15.0669403076, output_fired_time: 5.5780630112, propagate_forward_time: 325.4064636230, propagate_backward_time: 532.0564575195, weight_update_time: 0.3428293765, other_time: 9.8132934570
epoch time: 67.9799728394s, input_fired_time: 0.9761840105, output_fired_time: 0.0000000000, propagate_forward_time: 65.4284973145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5752944946
validation[          , genome  5633] predictions:    6470/  10000 (64.70%), best:    6471/10000 (64.71%), error:     10215.93457, best error:     10169.54980 on epoch:    98, epoch:   99/100, mu: 0.9883867502, learning_rate: 0.0000007930, weight_decay: 0.0000016956

epoch time: 862.7490844727s, input_fired_time: 14.8791399002, output_fired_time: 5.3135972023, propagate_forward_time: 313.4248657227, propagate_backward_time: 519.5454711914, weight_update_time: 0.3296226561, other_time: 9.5859985352
epoch time: 103.9278106689s, input_fired_time: 1.2877472639, output_fired_time: 0.0000000000, propagate_forward_time: 100.6851882935, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9548721313
validation[          , genome  5633] predictions:    6466/  10000 (64.66%), best:    6471/10000 (64.71%), error:     10222.00000, best error:     10169.54980 on epoch:    98, epoch:  100/100, mu: 0.9884770513, learning_rate: 0.0000007400, weight_decay: 0.0000016025

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 336.9961242676s, input_fired_time: 4.8653321266, output_fired_time: 0.0000000000, propagate_forward_time: 324.1641845703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9666137695
best training[          , genome  5633] predictions:   35375/  50000 (70.75%), best:    6471/10000 (64.71%), error:     44395.42969, best error:     10169.54980 on epoch:    98, epoch:  101/100, mu: 0.9885622859, learning_rate: 0.0000006906, weight_decay: 0.0000015145
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 77.7827453613s, input_fired_time: 1.0719913244, output_fired_time: 0.0000000000, propagate_forward_time: 74.9889755249, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7217788696
   testing[          , genome  5633] predictions:    6471/  10000 (64.71%), best:    6471/10000 (64.71%), error:     10169.54980, best error:     10169.54980 on epoch:    98, epoch:  101/100, mu: 0.9885622859, learning_rate: 0.0000006906, weight_decay: 0.0000015145
backpropagation finished successfully!
03:55:01 (15096): called boinc_finish(0)

</stderr_txt>
]]>