Nameexact_genome_1542692703_186_6387_0
Workunit4100331
Created20 Nov 2018, 7:12:24 UTC
Sent20 Nov 2018, 7:25:56 UTC
Report deadline26 Feb 2019, 21:15:29 UTC
Received25 Nov 2018, 8:26:39 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID78968
Run time2 days 9 hours 0 min 17 sec
CPU time2 days 8 hours 29 min 49 sec
Validate stateValid
Credit24,644.02
Device peak FLOPS2.20 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size545.31 MB
Peak swap size565.01 MB
Peak disk usage10.99 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
h 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1958.48413085938s, input_fired_time: 28.8067531585693, output_fired_time: 11.524941444397, propagate_forward_time: 712.616333007813, propagate_backward_time: 1183.70556640625, weight_update_time: 1.04688370227814, other_time: 21.83056640625
epoch time: 146.206298828125s, input_fired_time: 1.90933394432068, output_fired_time: 0, propagate_forward_time: 140.883651733398, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.41331481933594
validation[          , genome  6387] predictions:    6453/  10000 (64.53%), best:    6453/10000 (64.53%), error:     10279.16113, best error:     10279.16113 on epoch:    43, epoch:   43/100, mu: 0.9544684291, learning_rate: 0.0001617134, weight_decay: 0.0000669122

epoch time: 1982.2322998047s, input_fired_time: 28.9309558868, output_fired_time: 11.2694559097, propagate_forward_time: 717.3764038086, propagate_backward_time: 1202.5476074219, weight_update_time: 1.0705314875, other_time: 22.1079101563
epoch time: 152.5460968018s, input_fired_time: 1.9799982309, output_fired_time: 0.0000000000, propagate_forward_time: 146.7523498535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8137512207
validation[          , genome  6387] predictions:    6411/  10000 (64.11%), best:    6411/10000 (64.11%), error:     10277.70117, best error:     10277.70117 on epoch:    44, epoch:   44/100, mu: 0.9565150738, learning_rate: 0.0001518952, weight_decay: 0.0000638358

epoch time: 2002.9697265625s, input_fired_time: 29.1710186005, output_fired_time: 11.5808324814, propagate_forward_time: 727.1482543945, propagate_backward_time: 1212.2836914063, weight_update_time: 1.0784596205, other_time: 22.7858886719
epoch time: 143.0168304443s, input_fired_time: 1.8787199259, output_fired_time: 0.0000000000, propagate_forward_time: 137.5489807129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5891265869
validation[          , genome  6387] predictions:    6244/  10000 (62.44%), best:    6411/10000 (64.11%), error:     10761.45508, best error:     10277.70117 on epoch:    44, epoch:   45/100, mu: 0.9584438205, learning_rate: 0.0001426732, weight_decay: 0.0000609007

epoch time: 1968.9841308594s, input_fired_time: 29.2599258423, output_fired_time: 11.5437040329, propagate_forward_time: 719.8189697266, propagate_backward_time: 1186.5577392578, weight_update_time: 1.0353531837, other_time: 21.8038330078
epoch time: 152.5479125977s, input_fired_time: 1.9992388487, output_fired_time: 0.0000000000, propagate_forward_time: 147.0182800293, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5303955078
validation[          , genome  6387] predictions:    6398/  10000 (63.98%), best:    6398/10000 (63.98%), error:     10249.74609, best error:     10249.74609 on epoch:    46, epoch:   46/100, mu: 0.9602615237, learning_rate: 0.0001340111, weight_decay: 0.0000581007

epoch time: 1997.4334716797s, input_fired_time: 29.3611068726, output_fired_time: 11.7029848099, propagate_forward_time: 726.5058593750, propagate_backward_time: 1207.7770996094, weight_update_time: 1.0431183577, other_time: 22.0864257813
epoch time: 153.7025756836s, input_fired_time: 1.9794520140, output_fired_time: 0.0000000000, propagate_forward_time: 148.1685791016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5545501709
validation[          , genome  6387] predictions:    6397/  10000 (63.97%), best:    6398/10000 (63.98%), error:     10321.54004, best error:     10249.74609 on epoch:    46, epoch:   47/100, mu: 0.9619745016, learning_rate: 0.0001258749, weight_decay: 0.0000554293

epoch time: 2006.5340576172s, input_fired_time: 28.5569419861, output_fired_time: 11.5287914276, propagate_forward_time: 737.0341186523, propagate_backward_time: 1207.0592041016, weight_update_time: 1.0622403622, other_time: 22.3551025391
epoch time: 157.0052795410s, input_fired_time: 2.0303289890, output_fired_time: 0.0000000000, propagate_forward_time: 151.0197143555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9552307129
validation[          , genome  6387] predictions:    6416/  10000 (64.16%), best:    6398/10000 (63.98%), error:     10297.80566, best error:     10249.74609 on epoch:    46, epoch:   48/100, mu: 0.9635888338, learning_rate: 0.0001182326, weight_decay: 0.0000528808

epoch time: 2038.7127685547s, input_fired_time: 29.0861530304, output_fired_time: 11.9566183090, propagate_forward_time: 748.6956176758, propagate_backward_time: 1225.9256591797, weight_update_time: 1.0755441189, other_time: 23.0487060547
epoch time: 148.7054443359s, input_fired_time: 1.9138318300, output_fired_time: 0.0000000000, propagate_forward_time: 143.2485961914, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5430145264
validation[          , genome  6387] predictions:    6455/  10000 (64.55%), best:    6398/10000 (63.98%), error:     10331.86133, best error:     10249.74609 on epoch:    46, epoch:   49/100, mu: 0.9651101232, learning_rate: 0.0001110544, weight_decay: 0.0000504495

epoch time: 2046.6770019531s, input_fired_time: 29.1007881165, output_fired_time: 11.8690967560, propagate_forward_time: 750.9896850586, propagate_backward_time: 1232.1478271484, weight_update_time: 1.0624557734, other_time: 22.5694580078
epoch time: 146.8511657715s, input_fired_time: 1.8945482969, output_fired_time: 0.0000000000, propagate_forward_time: 141.2667083740, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6899108887
validation[          , genome  6387] predictions:    6385/  10000 (63.85%), best:    6398/10000 (63.98%), error:     10307.41992, best error:     10249.74609 on epoch:    46, epoch:   50/100, mu: 0.9665437937, learning_rate: 0.0001043119, weight_decay: 0.0000481299

epoch time: 2039.9068603516s, input_fired_time: 29.3841514587, output_fired_time: 11.7764415741, propagate_forward_time: 743.1679077148, propagate_backward_time: 1232.6734619141, weight_update_time: 1.0837897062, other_time: 22.9049072266
epoch time: 151.0554656982s, input_fired_time: 1.9846415520, output_fired_time: 0.0000000000, propagate_forward_time: 145.4694671631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6013641357
validation[          , genome  6387] predictions:    6475/  10000 (64.75%), best:    6475/10000 (64.75%), error:     10186.04492, best error:     10186.04492 on epoch:    51, epoch:   51/100, mu: 0.9678949118, learning_rate: 0.0000979788, weight_decay: 0.0000459170

epoch time: 2017.1102294922s, input_fired_time: 29.9147720337, output_fired_time: 12.2093248367, propagate_forward_time: 736.7775268555, propagate_backward_time: 1214.8334960938, weight_update_time: 1.1132285595, other_time: 23.3750000000
epoch time: 162.0878143311s, input_fired_time: 2.1405379772, output_fired_time: 0.0000000000, propagate_forward_time: 155.8371582031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1101226807
validation[          , genome  6387] predictions:    6483/  10000 (64.83%), best:    6475/10000 (64.75%), error:     10202.50391, best error:     10186.04492 on epoch:    51, epoch:   52/100, mu: 0.9691681862, learning_rate: 0.0000920302, weight_decay: 0.0000438059

epoch time: 2026.4333496094s, input_fired_time: 29.7239742279, output_fired_time: 11.6201086044, propagate_forward_time: 737.8728637695, propagate_backward_time: 1224.2460937500, weight_update_time: 1.0916730165, other_time: 22.9702148438
epoch time: 156.5212860107s, input_fired_time: 2.0653030872, output_fired_time: 0.0000000000, propagate_forward_time: 150.6223754883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8336029053
validation[          , genome  6387] predictions:    6461/  10000 (64.61%), best:    6461/10000 (64.61%), error:     10118.71289, best error:     10118.71289 on epoch:    53, epoch:   53/100, mu: 0.9703681469, learning_rate: 0.0000864428, weight_decay: 0.0000417918

epoch time: 1991.9932861328s, input_fired_time: 29.0365047455, output_fired_time: 11.4937543869, propagate_forward_time: 721.1873168945, propagate_backward_time: 1207.9908447266, weight_update_time: 1.0538704395, other_time: 22.2847900391
epoch time: 149.8684082031s, input_fired_time: 1.9776623249, output_fired_time: 0.0000000000, propagate_forward_time: 144.2700958252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6206512451
validation[          , genome  6387] predictions:    6308/  10000 (63.08%), best:    6461/10000 (64.61%), error:     10631.33789, best error:     10118.71289 on epoch:    53, epoch:   54/100, mu: 0.9714989662, learning_rate: 0.0000811946, weight_decay: 0.0000398703

epoch time: 2001.8973388672s, input_fired_time: 29.0716686249, output_fired_time: 11.6394863129, propagate_forward_time: 726.9014282227, propagate_backward_time: 1211.9921875000, weight_update_time: 1.0585385561, other_time: 22.2924804688
epoch time: 156.4322967529s, input_fired_time: 2.0874557495, output_fired_time: 0.0000000000, propagate_forward_time: 150.4339752197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9108734131
validation[          , genome  6387] predictions:    6480/  10000 (64.80%), best:    6480/10000 (64.80%), error:     10114.33105, best error:     10114.33105 on epoch:    55, epoch:   55/100, mu: 0.9725646377, learning_rate: 0.0000762650, weight_decay: 0.0000380371

epoch time: 2036.9517822266s, input_fired_time: 29.4821186066, output_fired_time: 11.8075342178, propagate_forward_time: 741.6751098633, propagate_backward_time: 1231.2624511719, weight_update_time: 1.0850795507, other_time: 22.7246093750
epoch time: 152.6831054688s, input_fired_time: 2.0026688576, output_fired_time: 0.0000000000, propagate_forward_time: 146.9066772461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7737579346
validation[          , genome  6387] predictions:    6493/  10000 (64.93%), best:    6493/10000 (64.93%), error:     10101.57129, best error:     10101.57129 on epoch:    56, epoch:   56/100, mu: 0.9735689163, learning_rate: 0.0000716347, weight_decay: 0.0000362883

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387_0_r774593390_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.534465
read mu: 0.974515
read mu_delta: 0.942399
read initial_learning_rate: 0.00239012
read learning_rate: 6.72856e-05
read learning_rate_delta: 0.939287
read initial_weight_decay: 0.000506395
read weight_decay: 3.46198e-05
read weight_decay_delta: 0.954022
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0652811
read input_dropout_probability: 0.00116783
read hidden_dropout_probability: 0.102143
read velocity_reset: 2062
read epoch: 57
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 56
read number_validation_images: 10000
read best_validation_predictions: 6493
read best_validation_error: 10101.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6387
read normal distribution: '0 -1.21096432209015 -1.04274129867554'
generator_str: '1006974522'
read generator: 1006974522
read generated_by_map:
1 crossover 1
reading 84 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1822.10766601563s, input_fired_time: 26.3978881835938, output_fired_time: 10.9400634765625, propagate_forward_time: 673.234313964844, propagate_backward_time: 1090.595703125, weight_update_time: 1.00196969509125, other_time: 20.939697265625
epoch time: 141.494186401367s, input_fired_time: 1.8576956987381, output_fired_time: 0, propagate_forward_time: 136.258453369141, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.37803649902344
validation[          , genome  6387] predictions:    6463/  10000 (64.63%), best:    6493/10000 (64.93%), error:     10203.07715, best error:     10101.57129 on epoch:    56, epoch:   57/100, mu: 0.9745153785, learning_rate: 0.0000672856, weight_decay: 0.0000346198

epoch time: 1864.6802978516s, input_fired_time: 26.4555854797, output_fired_time: 11.1859102249, propagate_forward_time: 686.7092285156, propagate_backward_time: 1119.2224121094, weight_update_time: 0.9868732095, other_time: 21.1071777344
epoch time: 136.1689910889s, input_fired_time: 1.7710998058, output_fired_time: 0.0000000000, propagate_forward_time: 131.2127227783, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1851654053
validation[          , genome  6387] predictions:    6537/  10000 (65.37%), best:    6537/10000 (65.37%), error:     10049.52441, best error:     10049.52441 on epoch:    58, epoch:   58/100, mu: 0.9754073024, learning_rate: 0.0000632005, weight_decay: 0.0000330281

epoch time: 1831.9912109375s, input_fired_time: 26.0830459595, output_fired_time: 10.8275527954, propagate_forward_time: 677.1457519531, propagate_backward_time: 1096.9078369141, weight_update_time: 1.0269147158, other_time: 21.0270996094
epoch time: 127.3921127319s, input_fired_time: 1.7265259027, output_fired_time: 0.0000000000, propagate_forward_time: 122.4326095581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2329788208
validation[          , genome  6387] predictions:    6521/  10000 (65.21%), best:    6537/10000 (65.37%), error:     10078.33105, best error:     10049.52441 on epoch:    58, epoch:   59/100, mu: 0.9762478471, learning_rate: 0.0000593634, weight_decay: 0.0000315095

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387_0_r774593390_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.534465
read mu: 0.97704
read mu_delta: 0.942399
read initial_learning_rate: 0.00239012
read learning_rate: 5.57592e-05
read learning_rate_delta: 0.939287
read initial_weight_decay: 0.000506395
read weight_decay: 3.00608e-05
read weight_decay_delta: 0.954022
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0652811
read input_dropout_probability: 0.00116783
read hidden_dropout_probability: 0.102143
read velocity_reset: 2062
read epoch: 60
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 58
read number_validation_images: 10000
read best_validation_predictions: 6537
read best_validation_error: 10049.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6387
read normal distribution: '0 -1.21096432209015 -1.04274129867554'
generator_str: '1553095107'
read generator: 1553095107
read generated_by_map:
1 crossover 1
reading 84 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1833.25122070313s, input_fired_time: 25.9465103149414, output_fired_time: 10.765923500061, propagate_forward_time: 677.413208007813, propagate_backward_time: 1099.19360351563, weight_update_time: 0.956730961799622, other_time: 19.9320068359375
epoch time: 145.117980957031s, input_fired_time: 1.97361481189728, output_fired_time: 0, propagate_forward_time: 139.382919311523, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.76144409179688
validation[          , genome  6387] predictions:    6561/  10000 (65.61%), best:    6537/10000 (65.37%), error:     10092.68848, best error:     10049.52441 on epoch:    58, epoch:   60/100, mu: 0.9770399928, learning_rate: 0.0000557592, weight_decay: 0.0000300608

epoch time: 1955.5974121094s, input_fired_time: 27.6589145660, output_fired_time: 11.6076974869, propagate_forward_time: 719.1743164063, propagate_backward_time: 1174.9392089844, weight_update_time: 1.0556432009, other_time: 22.2172851563
epoch time: 138.1039733887s, input_fired_time: 1.8944971561, output_fired_time: 0.0000000000, propagate_forward_time: 132.7275238037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4819488525
validation[          , genome  6387] predictions:    6474/  10000 (64.74%), best:    6537/10000 (65.37%), error:     10266.17090, best error:     10049.52441 on epoch:    58, epoch:   61/100, mu: 0.9777864814, learning_rate: 0.0000523739, weight_decay: 0.0000286787

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387_0_r774593390_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.534465
read mu: 0.97849
read mu_delta: 0.942399
read initial_learning_rate: 0.00239012
read learning_rate: 4.91941e-05
read learning_rate_delta: 0.939287
read initial_weight_decay: 0.000506395
read weight_decay: 2.73601e-05
read weight_decay_delta: 0.954022
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0652811
read input_dropout_probability: 0.00116783
read hidden_dropout_probability: 0.102143
read velocity_reset: 2062
read epoch: 62
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 58
read number_validation_images: 10000
read best_validation_predictions: 6537
read best_validation_error: 10049.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6387
read normal distribution: '0 -1.21096432209015 -1.04274129867554'
generator_str: '1125167268'
read generator: 1125167268
read generated_by_map:
1 crossover 1
reading 84 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1809.15637207031s, input_fired_time: 26.8605880737305, output_fired_time: 11.6914834976196, propagate_forward_time: 663.505004882813, propagate_backward_time: 1085.06628417969, weight_update_time: 1.01938927173615, other_time: 22.032958984375
epoch time: 127.379364013672s, input_fired_time: 1.67188715934753, output_fired_time: 0, propagate_forward_time: 122.527503967285, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.17996978759766
validation[          , genome  6387] predictions:    6512/  10000 (65.12%), best:    6537/10000 (65.37%), error:     10076.50781, best error:     10049.52441 on epoch:    58, epoch:   62/100, mu: 0.9784899950, learning_rate: 0.0000491941, weight_decay: 0.0000273601

epoch time: 1697.4041748047s, input_fired_time: 24.7891960144, output_fired_time: 10.3474512100, propagate_forward_time: 620.7203369141, propagate_backward_time: 1022.7340087891, weight_update_time: 0.8945184350, other_time: 18.8132324219
epoch time: 134.8636627197s, input_fired_time: 1.8065918684, output_fired_time: 0.0000000000, propagate_forward_time: 129.7988586426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2582092285
validation[          , genome  6387] predictions:    6522/  10000 (65.22%), best:    6522/10000 (65.22%), error:      9976.72168, best error:      9976.72168 on epoch:    63, epoch:   63/100, mu: 0.9791529775, learning_rate: 0.0000462074, weight_decay: 0.0000261021

epoch time: 1763.2794189453s, input_fired_time: 25.1100006104, output_fired_time: 10.3656473160, propagate_forward_time: 652.0750122070, propagate_backward_time: 1056.7099609375, weight_update_time: 0.9259558916, other_time: 19.0187988281
epoch time: 124.7177429199s, input_fired_time: 1.7066917419, output_fired_time: 0.0000000000, propagate_forward_time: 119.8906097412, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1204376221
validation[          , genome  6387] predictions:    6502/  10000 (65.02%), best:    6522/10000 (65.22%), error:     10210.67578, best error:      9976.72168 on epoch:    63, epoch:   64/100, mu: 0.9797777534, learning_rate: 0.0000434020, weight_decay: 0.0000249020

epoch time: 1886.3764648438s, input_fired_time: 26.5037765503, output_fired_time: 11.0995197296, propagate_forward_time: 699.0760498047, propagate_backward_time: 1128.1348876953, weight_update_time: 1.0221925974, other_time: 21.5622558594
epoch time: 140.8374023438s, input_fired_time: 1.8528932333, output_fired_time: 0.0000000000, propagate_forward_time: 135.3783416748, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6061706543
validation[          , genome  6387] predictions:    6553/  10000 (65.53%), best:    6522/10000 (65.22%), error:     10005.26855, best error:      9976.72168 on epoch:    63, epoch:   65/100, mu: 0.9803665876, learning_rate: 0.0000407670, weight_decay: 0.0000237571

epoch time: 1924.2843017578s, input_fired_time: 27.1218509674, output_fired_time: 11.7552986145, propagate_forward_time: 711.6604614258, propagate_backward_time: 1151.9360351563, weight_update_time: 1.0196253061, other_time: 21.8107910156
epoch time: 144.8372192383s, input_fired_time: 1.9101083279, output_fired_time: 0.0000000000, propagate_forward_time: 139.4247894287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5023193359
validation[          , genome  6387] predictions:    6560/  10000 (65.60%), best:    6560/10000 (65.60%), error:      9946.81641, best error:      9946.81641 on epoch:    66, epoch:   66/100, mu: 0.9809215069, learning_rate: 0.0000382919, weight_decay: 0.0000226648

epoch time: 1850.7077636719s, input_fired_time: 26.6250438690, output_fired_time: 10.8858909607, propagate_forward_time: 684.3232421875, propagate_backward_time: 1107.8013916016, weight_update_time: 1.0525950193, other_time: 21.0722656250
epoch time: 151.7189331055s, input_fired_time: 1.9686477184, output_fired_time: 0.0000000000, propagate_forward_time: 146.0696563721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6806335449
validation[          , genome  6387] predictions:    6597/  10000 (65.97%), best:    6597/10000 (65.97%), error:      9912.83887, best error:      9912.83887 on epoch:    67, epoch:   67/100, mu: 0.9814444184, learning_rate: 0.0000359670, weight_decay: 0.0000216227

epoch time: 1796.0347900391s, input_fired_time: 26.1092014313, output_fired_time: 10.7575578690, propagate_forward_time: 661.8992919922, propagate_backward_time: 1076.8093261719, weight_update_time: 0.9608803988, other_time: 20.4593505859
epoch time: 147.2177276611s, input_fired_time: 2.0509366989, output_fired_time: 0.0000000000, propagate_forward_time: 141.4758758545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6909179688
validation[          , genome  6387] predictions:    6567/  10000 (65.67%), best:    6597/10000 (65.97%), error:      9997.36328, best error:      9912.83887 on epoch:    67, epoch:   68/100, mu: 0.9819372296, learning_rate: 0.0000337834, weight_decay: 0.0000206285

epoch time: 1889.1865234375s, input_fired_time: 26.1799011230, output_fired_time: 11.1259937286, propagate_forward_time: 699.8413696289, propagate_backward_time: 1131.0595703125, weight_update_time: 1.0236120224, other_time: 20.9797363281
epoch time: 140.8746185303s, input_fired_time: 1.8310283422, output_fired_time: 0.0000000000, propagate_forward_time: 135.7828216553, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2607727051
validation[          , genome  6387] predictions:    6517/  10000 (65.17%), best:    6597/10000 (65.97%), error:     10078.54688, best error:      9912.83887 on epoch:    67, epoch:   69/100, mu: 0.9824016690, learning_rate: 0.0000317323, weight_decay: 0.0000196801

epoch time: 1861.4758300781s, input_fired_time: 26.2498168945, output_fired_time: 11.1127643585, propagate_forward_time: 686.3883666992, propagate_backward_time: 1117.1350097656, weight_update_time: 0.9771120548, other_time: 20.5898437500
epoch time: 131.6942749023s, input_fired_time: 1.8292487860, output_fired_time: 0.0000000000, propagate_forward_time: 126.4436874390, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4213333130
validation[          , genome  6387] predictions:    6590/  10000 (65.90%), best:    6590/10000 (65.90%), error:      9854.35059, best error:      9854.35059 on epoch:    70, epoch:   70/100, mu: 0.9828393459, learning_rate: 0.0000298057, weight_decay: 0.0000187752

epoch time: 1839.9219970703s, input_fired_time: 27.1036071777, output_fired_time: 11.7790470123, propagate_forward_time: 679.7188720703, propagate_backward_time: 1099.1353759766, weight_update_time: 1.0251967907, other_time: 22.1850585938
epoch time: 132.6935272217s, input_fired_time: 1.7157993317, output_fired_time: 0.0000000000, propagate_forward_time: 127.6981735229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2795486450
validation[          , genome  6387] predictions:    6607/  10000 (66.07%), best:    6590/10000 (65.90%), error:      9906.98340, best error:      9854.35059 on epoch:    70, epoch:   71/100, mu: 0.9832518101, learning_rate: 0.0000279961, weight_decay: 0.0000179120

epoch time: 1903.2185058594s, input_fired_time: 26.8774471283, output_fired_time: 11.3121042252, propagate_forward_time: 707.2889404297, propagate_backward_time: 1137.2340087891, weight_update_time: 0.9815847874, other_time: 20.5059814453
epoch time: 144.4212341309s, input_fired_time: 1.9092645645, output_fired_time: 0.0000000000, propagate_forward_time: 139.1006774902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4112854004
validation[          , genome  6387] predictions:    6557/  10000 (65.57%), best:    6590/10000 (65.90%), error:      9960.20703, best error:      9854.35059 on epoch:    70, epoch:   72/100, mu: 0.9836404920, learning_rate: 0.0000262964, weight_decay: 0.0000170884

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_186_6387_0_r774593390_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.534465
read mu: 0.984007
read mu_delta: 0.942399
read initial_learning_rate: 0.00239012
read learning_rate: 2.46999e-05
read learning_rate_delta: 0.939287
read initial_weight_decay: 0.000506395
read weight_decay: 1.63028e-05
read weight_decay_delta: 0.954022
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0652811
read input_dropout_probability: 0.00116783
read hidden_dropout_probability: 0.102143
read velocity_reset: 2062
read epoch: 73
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 70
read number_validation_images: 10000
read best_validation_predictions: 6590
read best_validation_error: 9854.35
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6387
read normal distribution: '0 -1.21096432209015 -1.04274129867554'
generator_str: '1167403355'
read generator: 1167403355
read generated_by_map:
1 crossover 1
reading 84 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1832.20141601563s, input_fired_time: 26.0180053710938, output_fired_time: 10.9934577941895, propagate_forward_time: 672.381958007813, propagate_backward_time: 1102.6728515625, weight_update_time: 0.983454942703247, other_time: 20.1351318359375
epoch time: 140.228088378906s, input_fired_time: 1.81382346153259, output_fired_time: 0, propagate_forward_time: 135.132461547852, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.28179931640625
validation[          , genome  6387] predictions:    6536/  10000 (65.36%), best:    6590/10000 (65.90%), error:      9876.43652, best error:      9854.35059 on epoch:    70, epoch:   73/100, mu: 0.9840068221, learning_rate: 0.0000246999, weight_decay: 0.0000163028

epoch time: 1774.5552978516s, input_fired_time: 25.5656375885, output_fired_time: 10.5303134918, propagate_forward_time: 656.0529174805, propagate_backward_time: 1062.7176513672, weight_update_time: 0.9597003460, other_time: 19.6888427734
epoch time: 124.0540008545s, input_fired_time: 1.6394478083, output_fired_time: 0.0000000000, propagate_forward_time: 119.4831085205, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9314422607
validation[          , genome  6387] predictions:    6564/  10000 (65.64%), best:    6590/10000 (65.90%), error:      9898.20410, best error:      9854.35059 on epoch:    70, epoch:   74/100, mu: 0.9843520522, learning_rate: 0.0000232003, weight_decay: 0.0000155532

epoch time: 1775.8435058594s, input_fired_time: 25.5560092926, output_fired_time: 10.6764526367, propagate_forward_time: 651.6310424805, propagate_backward_time: 1068.4370117188, weight_update_time: 0.9565364122, other_time: 19.5429687500
epoch time: 140.2921600342s, input_fired_time: 1.8892723322, output_fired_time: 0.0000000000, propagate_forward_time: 135.1050720215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2978210449
validation[          , genome  6387] predictions:    6587/  10000 (65.87%), best:    6587/10000 (65.87%), error:      9836.35449, best error:      9836.35449 on epoch:    75, epoch:   75/100, mu: 0.9846773744, learning_rate: 0.0000217917, weight_decay: 0.0000148381

epoch time: 1867.0551757813s, input_fired_time: 26.4398441315, output_fired_time: 11.0278692245, propagate_forward_time: 690.3963623047, propagate_backward_time: 1118.2065429688, weight_update_time: 1.0183452368, other_time: 20.9846191406
epoch time: 131.2532653809s, input_fired_time: 1.7590594292, output_fired_time: 0.0000000000, propagate_forward_time: 126.4565811157, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0376205444
validation[          , genome  6387] predictions:    6628/  10000 (66.28%), best:    6587/10000 (65.87%), error:      9840.83008, best error:      9836.35449 on epoch:    75, epoch:   76/100, mu: 0.9849839807, learning_rate: 0.0000204687, weight_decay: 0.0000141559

epoch time: 1886.7047119141s, input_fired_time: 26.8114013672, output_fired_time: 11.3591461182, propagate_forward_time: 697.2885131836, propagate_backward_time: 1129.7022705078, weight_update_time: 1.0388337374, other_time: 21.5433349609
epoch time: 135.4359741211s, input_fired_time: 1.8231650591, output_fired_time: 0.0000000000, propagate_forward_time: 130.1301422119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4826660156
validation[          , genome  6387] predictions:    6595/  10000 (65.95%), best:    6587/10000 (65.87%), error:      9914.70020, best error:      9836.35449 on epoch:    75, epoch:   77/100, mu: 0.9852728844, learning_rate: 0.0000192259, weight_decay: 0.0000135050

epoch time: 1802.1585693359s, input_fired_time: 25.9991188049, output_fired_time: 10.7399654388, propagate_forward_time: 662.3070678711, propagate_backward_time: 1083.0020751953, weight_update_time: 0.9727925658, other_time: 20.1102294922
epoch time: 150.7287139893s, input_fired_time: 1.9500766993, output_fired_time: 0.0000000000, propagate_forward_time: 145.2450866699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5335540771
validation[          , genome  6387] predictions:    6590/  10000 (65.90%), best:    6587/10000 (65.87%), error:      9857.03516, best error:      9836.35449 on epoch:    75, epoch:   78/100, mu: 0.9855451584, learning_rate: 0.0000180587, weight_decay: 0.0000128841

epoch time: 1869.8945312500s, input_fired_time: 26.3634872437, output_fired_time: 11.1294221878, propagate_forward_time: 691.6241455078, propagate_backward_time: 1119.7830810547, weight_update_time: 1.0016779900, other_time: 20.9943847656
epoch time: 149.0544891357s, input_fired_time: 2.0309355259, output_fired_time: 0.0000000000, propagate_forward_time: 143.4825897217, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5409698486
validation[          , genome  6387] predictions:    6631/  10000 (66.31%), best:    6631/10000 (66.31%), error:      9791.52344, best error:      9791.52344 on epoch:    79, epoch:   79/100, mu: 0.9858017564, learning_rate: 0.0000169623, weight_decay: 0.0000122917

epoch time: 1899.6915283203s, input_fired_time: 27.1737365723, output_fired_time: 11.6430263519, propagate_forward_time: 701.2395019531, propagate_backward_time: 1138.3084716797, weight_update_time: 1.0133776665, other_time: 21.3267822266
epoch time: 147.7984619141s, input_fired_time: 2.0448291302, output_fired_time: 0.0000000000, propagate_forward_time: 141.9810333252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7725982666
validation[          , genome  6387] predictions:    6616/  10000 (66.16%), best:    6616/10000 (66.16%), error:      9790.19922, best error:      9790.19922 on epoch:    80, epoch:   80/100, mu: 0.9860435724, learning_rate: 0.0000159324, weight_decay: 0.0000117266

epoch time: 1783.3927001953s, input_fired_time: 25.8530025482, output_fired_time: 10.5925607681, propagate_forward_time: 655.8594970703, propagate_backward_time: 1071.2432861328, weight_update_time: 0.9791091084, other_time: 19.8443603516
epoch time: 129.9836883545s, input_fired_time: 1.7476271391, output_fired_time: 0.0000000000, propagate_forward_time: 125.1234130859, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1126556396
validation[          , genome  6387] predictions:    6577/  10000 (65.77%), best:    6616/10000 (66.16%), error:      9882.44824, best error:      9790.19922 on epoch:    80, epoch:   81/100, mu: 0.9862714410, learning_rate: 0.0000149651, weight_decay: 0.0000111874

epoch time: 1760.8641357422s, input_fired_time: 25.6954441071, output_fired_time: 10.5204601288, propagate_forward_time: 647.0177612305, propagate_backward_time: 1057.5588378906, weight_update_time: 0.9896100163, other_time: 20.0715332031
epoch time: 123.0649032593s, input_fired_time: 1.6504416466, output_fired_time: 0.0000000000, propagate_forward_time: 118.4912796021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9231796265
validation[          , genome  6387] predictions:    6583/  10000 (65.83%), best:    6616/10000 (66.16%), error:      9808.35938, best error:      9790.19922 on epoch:    80, epoch:   82/100, mu: 0.9864861965, learning_rate: 0.0000140566, weight_decay: 0.0000106730

epoch time: 1819.5621337891s, input_fired_time: 26.8357601166, output_fired_time: 10.9371566772, propagate_forward_time: 665.8208007813, propagate_backward_time: 1095.5126953125, weight_update_time: 0.9826365709, other_time: 20.4556884766
epoch time: 136.9727020264s, input_fired_time: 1.8320990801, output_fired_time: 0.0000000000, propagate_forward_time: 131.9718017578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1688079834
validation[          , genome  6387] predictions:    6610/  10000 (66.10%), best:    6610/10000 (66.10%), error:      9770.96094, best error:      9770.96094 on epoch:    83, epoch:   83/100, mu: 0.9866886139, learning_rate: 0.0000132031, weight_decay: 0.0000101823

epoch time: 1732.8632812500s, input_fired_time: 25.2266635895, output_fired_time: 10.2274141312, propagate_forward_time: 634.1994628906, propagate_backward_time: 1043.4792480469, weight_update_time: 0.9473077655, other_time: 19.7304687500
epoch time: 130.5206298828s, input_fired_time: 1.7225027084, output_fired_time: 0.0000000000, propagate_forward_time: 125.5218505859, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2762756348
validation[          , genome  6387] predictions:    6636/  10000 (66.36%), best:    6610/10000 (66.10%), error:      9807.47852, best error:      9770.96094 on epoch:    83, epoch:   84/100, mu: 0.9868793488, learning_rate: 0.0000124015, weight_decay: 0.0000097142

epoch time: 1851.4605712891s, input_fired_time: 26.9135360718, output_fired_time: 10.9490480423, propagate_forward_time: 685.3660888672, propagate_backward_time: 1107.4512939453, weight_update_time: 1.0157715082, other_time: 20.7805175781
epoch time: 134.6909942627s, input_fired_time: 1.8523510695, output_fired_time: 0.0000000000, propagate_forward_time: 129.5584716797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2801666260
validation[          , genome  6387] predictions:    6602/  10000 (66.02%), best:    6602/10000 (66.02%), error:      9761.25781, best error:      9761.25781 on epoch:    85, epoch:   85/100, mu: 0.9870591164, learning_rate: 0.0000116486, weight_decay: 0.0000092675

epoch time: 1808.4993896484s, input_fired_time: 25.9297199249, output_fired_time: 10.7457618713, propagate_forward_time: 663.3421020508, propagate_backward_time: 1088.4598388672, weight_update_time: 0.9834958315, other_time: 20.0220947266
epoch time: 151.6432952881s, input_fired_time: 2.0671806335, output_fired_time: 0.0000000000, propagate_forward_time: 145.7264099121, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8497009277
validation[          , genome  6387] predictions:    6583/  10000 (65.83%), best:    6602/10000 (66.02%), error:      9834.82813, best error:      9761.25781 on epoch:    85, epoch:   86/100, mu: 0.9872285128, learning_rate: 0.0000109414, weight_decay: 0.0000088414

epoch time: 1751.5296630859s, input_fired_time: 25.2217807770, output_fired_time: 10.3748197556, propagate_forward_time: 645.7274780273, propagate_backward_time: 1050.8748779297, weight_update_time: 0.9511306286, other_time: 19.3306884766
epoch time: 145.4102478027s, input_fired_time: 1.9217525721, output_fired_time: 0.0000000000, propagate_forward_time: 140.0197906494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4687042236
validation[          , genome  6387] predictions:    6644/  10000 (66.44%), best:    6602/10000 (66.02%), error:      9792.28223, best error:      9761.25781 on epoch:    85, epoch:   87/100, mu: 0.9873881340, learning_rate: 0.0000102771, weight_decay: 0.0000084349

epoch time: 1878.8448486328s, input_fired_time: 26.6122417450, output_fired_time: 11.0173521042, propagate_forward_time: 697.0021362305, propagate_backward_time: 1123.4265136719, weight_update_time: 1.0127550364, other_time: 20.7866210938
epoch time: 139.2266998291s, input_fired_time: 1.8392360210, output_fired_time: 0.0000000000, propagate_forward_time: 134.0666656494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3208007813
validation[          , genome  6387] predictions:    6639/  10000 (66.39%), best:    6639/10000 (66.39%), error:      9745.87695, best error:      9745.87695 on epoch:    88, epoch:   88/100, mu: 0.9875385761, learning_rate: 0.0000096531, weight_decay: 0.0000080471

epoch time: 1882.4211425781s, input_fired_time: 26.5864429474, output_fired_time: 11.2611198425, propagate_forward_time: 692.5145263672, propagate_backward_time: 1131.0013427734, weight_update_time: 1.0343044996, other_time: 21.0577392578
epoch time: 145.6239013672s, input_fired_time: 2.0615785122, output_fired_time: 0.0000000000, propagate_forward_time: 139.7178192139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8444976807
validation[          , genome  6387] predictions:    6604/  10000 (66.04%), best:    6639/10000 (66.39%), error:      9753.85840, best error:      9745.87695 on epoch:    88, epoch:   89/100, mu: 0.9876803756, learning_rate: 0.0000090671, weight_decay: 0.0000076771

epoch time: 1856.8377685547s, input_fired_time: 26.5829811096, output_fired_time: 11.5211162567, propagate_forward_time: 685.7387084961, propagate_backward_time: 1112.3677978516, weight_update_time: 1.0033788681, other_time: 20.6270751953
epoch time: 146.7662963867s, input_fired_time: 2.0105242729, output_fired_time: 0.0000000000, propagate_forward_time: 141.0250396729, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7307281494
validation[          , genome  6387] predictions:    6638/  10000 (66.38%), best:    6639/10000 (66.39%), error:      9766.89453, best error:      9745.87695 on epoch:    88, epoch:   90/100, mu: 0.9878140092, learning_rate: 0.0000085166, weight_decay: 0.0000073241

epoch time: 1838.3718261719s, input_fired_time: 25.8456840515, output_fired_time: 10.8638515472, propagate_forward_time: 677.5140991211, propagate_backward_time: 1104.4848632813, weight_update_time: 0.9609755874, other_time: 19.6633300781
epoch time: 146.4688110352s, input_fired_time: 1.9717653990, output_fired_time: 0.0000000000, propagate_forward_time: 141.0156860352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4813537598
validation[          , genome  6387] predictions:    6605/  10000 (66.05%), best:    6605/10000 (66.05%), error:      9739.32422, best error:      9739.32422 on epoch:    91, epoch:   91/100, mu: 0.9879399538, learning_rate: 0.0000079995, weight_decay: 0.0000069874

epoch time: 1894.6434326172s, input_fired_time: 27.3607025146, output_fired_time: 12.0968265533, propagate_forward_time: 705.4409179688, propagate_backward_time: 1127.5036621094, weight_update_time: 1.0261352062, other_time: 22.2413330078
epoch time: 148.9839324951s, input_fired_time: 2.0728652477, output_fired_time: 0.0000000000, propagate_forward_time: 143.0775146484, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8335571289
validation[          , genome  6387] predictions:    6619/  10000 (66.19%), best:    6619/10000 (66.19%), error:      9723.59961, best error:      9723.59961 on epoch:    92, epoch:   92/100, mu: 0.9880586267, learning_rate: 0.0000075138, weight_decay: 0.0000066661

epoch time: 1851.5734863281s, input_fired_time: 27.5627326965, output_fired_time: 12.4567003250, propagate_forward_time: 685.5869750977, propagate_backward_time: 1103.1868896484, weight_update_time: 1.0853710175, other_time: 22.7801513672
epoch time: 151.3467864990s, input_fired_time: 2.1473631859, output_fired_time: 0.0000000000, propagate_forward_time: 145.2746582031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9247589111
validation[          , genome  6387] predictions:    6630/  10000 (66.30%), best:    6630/10000 (66.30%), error:      9699.99414, best error:      9699.99414 on epoch:    93, epoch:   93/100, mu: 0.9881704450, learning_rate: 0.0000070577, weight_decay: 0.0000063596

epoch time: 1796.2038574219s, input_fired_time: 25.9951610565, output_fired_time: 10.6068515778, propagate_forward_time: 664.2492675781, propagate_backward_time: 1075.2934570313, weight_update_time: 0.9725317955, other_time: 20.0592041016
epoch time: 132.3662719727s, input_fired_time: 1.7935633659, output_fired_time: 0.0000000000, propagate_forward_time: 127.2942047119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2785034180
validation[          , genome  6387] predictions:    6613/  10000 (66.13%), best:    6630/10000 (66.30%), error:      9727.77441, best error:      9699.99414 on epoch:    93, epoch:   94/100, mu: 0.9882758260, learning_rate: 0.0000066292, weight_decay: 0.0000060672

epoch time: 1833.8752441406s, input_fired_time: 26.2026596069, output_fired_time: 10.8035764694, propagate_forward_time: 676.1022949219, propagate_backward_time: 1100.1192626953, weight_update_time: 1.0004018545, other_time: 20.6474609375
epoch time: 139.6229553223s, input_fired_time: 1.9241480827, output_fired_time: 0.0000000000, propagate_forward_time: 134.2521209717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4466857910
validation[          , genome  6387] predictions:    6629/  10000 (66.29%), best:    6630/10000 (66.30%), error:      9733.22949, best error:      9699.99414 on epoch:    93, epoch:   95/100, mu: 0.9883751273, learning_rate: 0.0000062267, weight_decay: 0.0000057883

epoch time: 1814.1237792969s, input_fired_time: 27.6666870117, output_fired_time: 12.0158014297, propagate_forward_time: 662.6024780273, propagate_backward_time: 1088.7200927734, weight_update_time: 1.0914130211, other_time: 23.1187744141
epoch time: 170.0804748535s, input_fired_time: 4.0692405701, output_fired_time: 0.0000000000, propagate_forward_time: 157.5450744629, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4661560059
validation[          , genome  6387] predictions:    6640/  10000 (66.40%), best:    6640/10000 (66.40%), error:      9693.43848, best error:      9693.43848 on epoch:    96, epoch:   96/100, mu: 0.9884687066, learning_rate: 0.0000058486, weight_decay: 0.0000055221

epoch time: 1821.3569335938s, input_fired_time: 26.1465530396, output_fired_time: 10.8827238083, propagate_forward_time: 675.4161376953, propagate_backward_time: 1088.4404296875, weight_update_time: 0.9828869700, other_time: 20.4710693359
epoch time: 135.8426513672s, input_fired_time: 1.8952627182, output_fired_time: 0.0000000000, propagate_forward_time: 130.3485107422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5988769531
validation[          , genome  6387] predictions:    6651/  10000 (66.51%), best:    6640/10000 (66.40%), error:      9696.66602, best error:      9693.43848 on epoch:    96, epoch:   97/100, mu: 0.9885569215, learning_rate: 0.0000054936, weight_decay: 0.0000052682

epoch time: 1879.1539306641s, input_fired_time: 27.0096187592, output_fired_time: 11.3936614990, propagate_forward_time: 691.5296020508, propagate_backward_time: 1127.7062988281, weight_update_time: 1.0027594566, other_time: 21.5146484375
epoch time: 137.3051452637s, input_fired_time: 1.8919080496, output_fired_time: 0.0000000000, propagate_forward_time: 132.0601806641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3530578613
validation[          , genome  6387] predictions:    6589/  10000 (65.89%), best:    6640/10000 (66.40%), error:      9745.51855, best error:      9693.43848 on epoch:    96, epoch:   98/100, mu: 0.9886400700, learning_rate: 0.0000051600, weight_decay: 0.0000050260

epoch time: 1912.1726074219s, input_fired_time: 28.1594505310, output_fired_time: 11.9670763016, propagate_forward_time: 706.7376098633, propagate_backward_time: 1142.2652587891, weight_update_time: 1.0768636465, other_time: 23.0433349609
epoch time: 143.0235137939s, input_fired_time: 1.9716889858, output_fired_time: 0.0000000000, propagate_forward_time: 137.5016174316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5502014160
validation[          , genome  6387] predictions:    6626/  10000 (66.26%), best:    6640/10000 (66.40%), error:      9732.09473, best error:      9693.43848 on epoch:    96, epoch:   99/100, mu: 0.9887183905, learning_rate: 0.0000048467, weight_decay: 0.0000047949

epoch time: 1818.8242187500s, input_fired_time: 26.2645931244, output_fired_time: 10.6387615204, propagate_forward_time: 664.7756347656, propagate_backward_time: 1096.9095458984, weight_update_time: 0.9695119858, other_time: 20.2355957031
epoch time: 146.4727935791s, input_fired_time: 1.9264677763, output_fired_time: 0.0000000000, propagate_forward_time: 141.0166778564, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5296478271
validation[          , genome  6387] predictions:    6634/  10000 (66.34%), best:    6640/10000 (66.40%), error:      9694.08789, best error:      9693.43848 on epoch:    96, epoch:  100/100, mu: 0.9887922406, learning_rate: 0.0000045525, weight_decay: 0.0000045745

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 702.2534179688s, input_fired_time: 9.5077171326, output_fired_time: 0.0000000000, propagate_forward_time: 674.9332275391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.8125000000
best training[          , genome  6387] predictions:   37359/  50000 (74.72%), best:    6640/10000 (66.40%), error:     38819.43359, best error:      9693.43848 on epoch:    96, epoch:  101/100, mu: 0.9888617992, learning_rate: 0.0000042761, weight_decay: 0.0000043642
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 153.3391876221s, input_fired_time: 2.4187114239, output_fired_time: 0.0000000000, propagate_forward_time: 145.1434173584, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7770538330
   testing[          , genome  6387] predictions:    6640/  10000 (66.40%), best:    6640/10000 (66.40%), error:      9693.43848, best error:      9693.43848 on epoch:    96, epoch:  101/100, mu: 0.9888617992, learning_rate: 0.0000042761, weight_decay: 0.0000043642
backpropagation finished successfully!
00:26:18 (2148): called boinc_finish(0)

</stderr_txt>
]]>