Nameexact_genome_1542383404_172_6095_1
Workunit4093559
Created16 Nov 2018, 19:27:33 UTC
Sent16 Nov 2018, 20:06:40 UTC
Report deadline17 Dec 2018, 5:13:31 UTC
Received3 Dec 2018, 22:10:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID13956
Run time12 hours 16 min 22 sec
CPU time12 hours 16 min 22 sec
Validate stateValid
Credit7,594.94
Device peak FLOPS2.92 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size588.07 MB
Peak swap size607.80 MB
Peak disk usage13.68 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095_1_r1488441718_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.501246
read mu: 0.501246
read mu_delta: 0.931124
read initial_learning_rate: 0.000235667
read learning_rate: 0.000235667
read learning_rate_delta: 0.931787
read initial_weight_decay: 0.000237445
read weight_decay: 0.000237445
read weight_decay_delta: 0.939875
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0944787
read input_dropout_probability: 0.00138075
read hidden_dropout_probability: 0.0791619
read velocity_reset: 1888
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6095
read normal distribution: '1 0 0'
generator_str: '1224713281'
read generator: 1224713281
read generated_by_map:
1 crossover 1
reading 117 nodes.
reading 470 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 125357272
post shuffle 1: 44048860
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095_1_r1488441718_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_172_6095.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.501246
read mu: 0.501246
read mu_delta: 0.931124
read initial_learning_rate: 0.000235667
read learning_rate: 0.000235667
read learning_rate_delta: 0.931787
read initial_weight_decay: 0.000237445
read weight_decay: 0.000237445
read weight_decay_delta: 0.939875
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0944787
read input_dropout_probability: 0.00138075
read hidden_dropout_probability: 0.0791619
read velocity_reset: 1888
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6095
read normal distribution: '1 0 0'
generator_str: '1224713281'
read generator: 1224713281
read generated_by_map:
1 crossover 1
reading 117 nodes.
reading 470 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 125357272
post shuffle 1: 44048860
epoch time: 236474.84375s, input_fired_time: 36.9007148742676, output_fired_time: 13.5371990203857, propagate_forward_time: 32694.560546875, propagate_backward_time: 203700.28125, weight_update_time: 0.505995631217957, other_time: 29.5625
epoch time: 92.1673431396484s, input_fired_time: 2.16951036453247, output_fired_time: 0, propagate_forward_time: 84.9963455200195, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.00148773193359
validation[          , genome  6095] predictions:    6355/  10000 (63.55%), best:    6355/10000 (63.55%), error:     10508.13184, best error:     10508.13184 on epoch:     0, epoch:    0/25, mu: 0.5012463927, learning_rate: 0.0002356670, weight_decay: 0.0002374449

epoch time: 5333.1708984375s, input_fired_time: 39.5194969177, output_fired_time: 15.8304862976, propagate_forward_time: 498.2208251953, propagate_backward_time: 4743.6889648438, weight_update_time: 0.6068588495, other_time: 35.9111328125
epoch time: 128.0649566650s, input_fired_time: 3.7525832653, output_fired_time: 0.0000000000, propagate_forward_time: 116.2894439697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.0229263306
validation[          , genome  6095] predictions:    5802/  10000 (58.02%), best:    6355/10000 (63.55%), error:     12082.43848, best error:     10508.13184 on epoch:     0, epoch:    1/25, mu: 0.5349097252, learning_rate: 0.0002195913, weight_decay: 0.0002231686

epoch time: 1492.9006347656s, input_fired_time: 33.4604644775, output_fired_time: 11.3223495483, propagate_forward_time: 431.7261352539, propagate_backward_time: 989.1614990234, weight_update_time: 0.4675647914, other_time: 27.2301635742
epoch time: 94.7376632690s, input_fired_time: 2.1732716560, output_fired_time: 0.0000000000, propagate_forward_time: 87.6318435669, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9325485229
validation[          , genome  6095] predictions:    6341/  10000 (63.41%), best:    6355/10000 (63.55%), error:     10525.18945, best error:     10508.13184 on epoch:     0, epoch:    2/25, mu: 0.5662544966, learning_rate: 0.0002046123, weight_decay: 0.0002097507

epoch time: 4443.8115234375s, input_fired_time: 2828.0886230469, output_fired_time: 14.4226722717, propagate_forward_time: 494.4707641602, propagate_backward_time: 1073.1469726563, weight_update_time: 0.5642078519, other_time: 33.6823730469
epoch time: 121.5888366699s, input_fired_time: 3.8780150414, output_fired_time: 0.0000000000, propagate_forward_time: 107.2443084717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.4665145874
validation[          , genome  6095] predictions:    5945/  10000 (59.45%), best:    6355/10000 (63.55%), error:     11575.38281, best error:     10508.13184 on epoch:     0, epoch:    3/25, mu: 0.5954403281, learning_rate: 0.0001906549, weight_decay: 0.0001971395

epoch time: 14500.4130859375s, input_fired_time: 43.1011199951, output_fired_time: 21.1779594421, propagate_forward_time: 13239.5605468750, propagate_backward_time: 1142.0009765625, weight_update_time: 0.8016401529, other_time: 54.5722656250
epoch time: 134.0170898438s, input_fired_time: 3.7130508423, output_fired_time: 0.0000000000, propagate_forward_time: 121.0376968384, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2663497925
validation[          , genome  6095] predictions:    6240/  10000 (62.40%), best:    6355/10000 (63.55%), error:     10832.07031, best error:     10508.13184 on epoch:     0, epoch:    4/25, mu: 0.6226159930, learning_rate: 0.0001776497, weight_decay: 0.0001852866

epoch time: 1675.3288574219s, input_fired_time: 38.3822364807, output_fired_time: 15.5858802795, propagate_forward_time: 500.5216674805, propagate_backward_time: 1086.0146484375, weight_update_time: 0.5876004696, other_time: 34.8244628906
epoch time: 101.2631683350s, input_fired_time: 2.2154860497, output_fired_time: 0.0000000000, propagate_forward_time: 94.1501312256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8975524902
validation[          , genome  6095] predictions:    6087/  10000 (60.87%), best:    6355/10000 (63.55%), error:     11582.26074, best error:     10508.13184 on epoch:     0, epoch:    5/25, mu: 0.6479198933, learning_rate: 0.0001655316, weight_decay: 0.0001741463

epoch time: 1567.8856201172s, input_fired_time: 34.5851249695, output_fired_time: 12.4009504318, propagate_forward_time: 472.3047180176, propagate_backward_time: 1020.6264038086, weight_update_time: 0.4885305762, other_time: 27.9684448242
epoch time: 100.5557861328s, input_fired_time: 2.2009818554, output_fired_time: 0.0000000000, propagate_forward_time: 93.4702377319, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8845672607
validation[          , genome  6095] predictions:    6214/  10000 (62.14%), best:    6355/10000 (63.55%), error:     10998.51074, best error:     10508.13184 on epoch:     0, epoch:    6/25, mu: 0.6714809537, learning_rate: 0.0001542401, weight_decay: 0.0001636758

epoch time: 45616.3671875000s, input_fired_time: 35.2914428711, output_fired_time: 12.9251613617, propagate_forward_time: 33064.6093750000, propagate_backward_time: 12473.5039062500, weight_update_time: 0.5589415431, other_time: 30.0351562500
epoch time: 108.2022933960s, input_fired_time: 2.4764864445, output_fired_time: 0.0000000000, propagate_forward_time: 100.2767028809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4491043091
validation[          , genome  6095] predictions:    6177/  10000 (61.77%), best:    6355/10000 (63.55%), error:     11085.54785, best error:     10508.13184 on epoch:     0, epoch:    7/25, mu: 0.6934192181, learning_rate: 0.0001437189, weight_decay: 0.0001538349

epoch time: 1726.1613769531s, input_fired_time: 38.7048988342, output_fired_time: 15.6963205338, propagate_forward_time: 512.6918945313, propagate_backward_time: 1125.4755859375, weight_update_time: 0.5777207017, other_time: 33.5926513672
epoch time: 114.3844680786s, input_fired_time: 3.1713998318, output_fired_time: 0.0000000000, propagate_forward_time: 101.1807098389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.0323562622
validation[          , genome  6095] predictions:    6065/  10000 (60.65%), best:    6355/10000 (63.55%), error:     11519.05664, best error:     10508.13184 on epoch:     0, epoch:    8/25, mu: 0.7138465047, learning_rate: 0.0001339153, weight_decay: 0.0001445856

epoch time: 1599.2302246094s, input_fired_time: 36.1022682190, output_fired_time: 14.1542348862, propagate_forward_time: 472.8854675293, propagate_backward_time: 1042.7279052734, weight_update_time: 0.5595433116, other_time: 33.3603515625
epoch time: 107.0696868896s, input_fired_time: 2.5332508087, output_fired_time: 0.0000000000, propagate_forward_time: 98.4947204590, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0417175293
validation[          , genome  6095] predictions:    6186/  10000 (61.86%), best:    6355/10000 (63.55%), error:     10877.28125, best error:     10508.13184 on epoch:     0, epoch:    9/25, mu: 0.7328668237, learning_rate: 0.0001247805, weight_decay: 0.0001358924

epoch time: 1564.3745117188s, input_fired_time: 35.1745872498, output_fired_time: 12.3041200638, propagate_forward_time: 464.0509338379, propagate_backward_time: 1024.1989746094, weight_update_time: 0.4973936379, other_time: 28.6459960938
epoch time: 106.0451126099s, input_fired_time: 2.5008904934, output_fired_time: 0.0000000000, propagate_forward_time: 97.8944244385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6497955322
validation[          , genome  6095] predictions:    6212/  10000 (62.12%), best:    6355/10000 (63.55%), error:     10989.51270, best error:     10508.13184 on epoch:     0, epoch:   10/25, mu: 0.7505770922, learning_rate: 0.0001162688, weight_decay: 0.0001277220

epoch time: 1606.7388916016s, input_fired_time: 36.3388862610, output_fired_time: 13.6806287766, propagate_forward_time: 475.0014953613, propagate_backward_time: 1050.4323730469, weight_update_time: 0.5557222366, other_time: 31.2855224609
epoch time: 110.3641204834s, input_fired_time: 2.6404137611, output_fired_time: 0.0000000000, propagate_forward_time: 102.0473937988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6763153076
validation[          , genome  6095] predictions:    6133/  10000 (61.33%), best:    6355/10000 (63.55%), error:     11089.27539, best error:     10508.13184 on epoch:     0, epoch:   11/25, mu: 0.7670675516, learning_rate: 0.0001083377, weight_decay: 0.0001200427

epoch time: 1661.9669189453s, input_fired_time: 35.7798995972, output_fired_time: 13.5587882996, propagate_forward_time: 524.6424560547, propagate_backward_time: 1058.0909423828, weight_update_time: 0.5355276465, other_time: 29.8947753906
epoch time: 105.5567779541s, input_fired_time: 2.3732595444, output_fired_time: 0.0000000000, propagate_forward_time: 97.9526672363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2308502197
validation[          , genome  6095] predictions:    6255/  10000 (62.55%), best:    6355/10000 (63.55%), error:     10924.81934, best error:     10508.13184 on epoch:     0, epoch:   12/25, mu: 0.7824222445, learning_rate: 0.0001009476, weight_decay: 0.0001128252

epoch time: 1661.1917724609s, input_fired_time: 37.8168869019, output_fired_time: 14.8461494446, propagate_forward_time: 502.5428466797, propagate_backward_time: 1073.6583251953, weight_update_time: 0.5834590793, other_time: 32.3275146484
epoch time: 106.6879806519s, input_fired_time: 2.5669932365, output_fired_time: 0.0000000000, propagate_forward_time: 98.0585021973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0624847412
validation[          , genome  6095] predictions:    6277/  10000 (62.77%), best:    6355/10000 (63.55%), error:     10769.37109, best error:     10508.13184 on epoch:     0, epoch:   13/25, mu: 0.7967193723, learning_rate: 0.0000940616, weight_decay: 0.0001060416

epoch time: 1630.8662109375s, input_fired_time: 36.4942932129, output_fired_time: 14.0889043808, propagate_forward_time: 487.3221740723, propagate_backward_time: 1062.5964355469, weight_update_time: 0.5513988137, other_time: 30.3645019531
epoch time: 101.5208587646s, input_fired_time: 2.5589768887, output_fired_time: 0.0000000000, propagate_forward_time: 93.5657882690, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3960952759
validation[          , genome  6095] predictions:    6236/  10000 (62.36%), best:    6355/10000 (63.55%), error:     10750.39453, best error:     10508.13184 on epoch:     0, epoch:   14/25, mu: 0.8100317717, learning_rate: 0.0000876453, weight_decay: 0.0000996659

epoch time: 1585.2873535156s, input_fired_time: 35.5754890442, output_fired_time: 13.0490922928, propagate_forward_time: 468.0783996582, propagate_backward_time: 1039.0439453125, weight_update_time: 0.5222003460, other_time: 29.5405273438
epoch time: 100.9411926270s, input_fired_time: 2.3232517242, output_fired_time: 0.0000000000, propagate_forward_time: 93.3476943970, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2702484131
validation[          , genome  6095] predictions:    6195/  10000 (61.95%), best:    6355/10000 (63.55%), error:     10853.58984, best error:     10508.13184 on epoch:     0, epoch:   15/25, mu: 0.8224272728, learning_rate: 0.0000816667, weight_decay: 0.0000936735

epoch time: 2060.8649902344s, input_fired_time: 34.5529289246, output_fired_time: 12.3369255066, propagate_forward_time: 458.2799377441, propagate_backward_time: 1527.4458007813, weight_update_time: 0.5099132061, other_time: 28.2493896484
epoch time: 97.7964630127s, input_fired_time: 2.1795878410, output_fired_time: 0.0000000000, propagate_forward_time: 90.8612213135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7556533813
validation[          , genome  6095] predictions:    6204/  10000 (62.04%), best:    6355/10000 (63.55%), error:     10896.00488, best error:     10508.13184 on epoch:     0, epoch:   16/25, mu: 0.8339689970, learning_rate: 0.0000760960, weight_decay: 0.0000880415

epoch time: 1509.0046386719s, input_fired_time: 33.4931449890, output_fired_time: 11.0068674088, propagate_forward_time: 444.4504089355, propagate_backward_time: 994.3614501953, weight_update_time: 0.4688788354, other_time: 25.6927490234
epoch time: 97.3087387085s, input_fired_time: 2.2740006447, output_fired_time: 0.0000000000, propagate_forward_time: 90.0750808716, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9596557617
validation[          , genome  6095] predictions:    6139/  10000 (61.39%), best:    6355/10000 (63.55%), error:     11152.68457, best error:     10508.13184 on epoch:     0, epoch:   17/25, mu: 0.8447157741, learning_rate: 0.0000709052, weight_decay: 0.0000827480

epoch time: 1617.2999267578s, input_fired_time: 35.7192649841, output_fired_time: 13.5418033600, propagate_forward_time: 481.6934509277, propagate_backward_time: 1056.8885498047, weight_update_time: 0.5491178632, other_time: 29.4569091797
epoch time: 112.5872039795s, input_fired_time: 2.5725128651, output_fired_time: 0.0000000000, propagate_forward_time: 104.5614166260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4532775879
validation[          , genome  6095] predictions:    6251/  10000 (62.51%), best:    6355/10000 (63.55%), error:     10767.02832, best error:     10508.13184 on epoch:     0, epoch:   18/25, mu: 0.8547223806, learning_rate: 0.0000660685, weight_decay: 0.0000777728

epoch time: 1706.7993164063s, input_fired_time: 38.8130455017, output_fired_time: 14.7844266891, propagate_forward_time: 519.1512451172, propagate_backward_time: 1102.4052734375, weight_update_time: 0.5664264560, other_time: 31.6453857422
epoch time: 104.1730194092s, input_fired_time: 2.2809281349, output_fired_time: 0.0000000000, propagate_forward_time: 97.0885086060, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8035812378
validation[          , genome  6095] predictions:    6261/  10000 (62.61%), best:    6355/10000 (63.55%), error:     10776.60156, best error:     10508.13184 on epoch:     0, epoch:   19/25, mu: 0.8640397787, learning_rate: 0.0000615617, weight_decay: 0.0000730967

epoch time: 1579.0686035156s, input_fired_time: 35.4274177551, output_fired_time: 12.5574045181, propagate_forward_time: 464.4334106445, propagate_backward_time: 1037.8181152344, weight_update_time: 0.5132469535, other_time: 28.8322753906
epoch time: 125.0523376465s, input_fired_time: 3.7800362110, output_fired_time: 0.0000000000, propagate_forward_time: 109.6519851685, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.6203155518
validation[          , genome  6095] predictions:    6064/  10000 (60.64%), best:    6355/10000 (63.55%), error:     11338.76074, best error:     10508.13184 on epoch:     0, epoch:   20/25, mu: 0.8727154136, learning_rate: 0.0000573624, weight_decay: 0.0000687018

epoch time: 1689.0972900391s, input_fired_time: 38.5144157410, output_fired_time: 15.4549293518, propagate_forward_time: 502.3733520508, propagate_backward_time: 1097.4897460938, weight_update_time: 0.6356041431, other_time: 35.2648925781
epoch time: 108.4552764893s, input_fired_time: 2.4191145897, output_fired_time: 0.0000000000, propagate_forward_time: 100.7484283447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2877349854
validation[          , genome  6095] predictions:    6130/  10000 (61.30%), best:    6355/10000 (63.55%), error:     10866.57520, best error:     10508.13184 on epoch:     0, epoch:   21/25, mu: 0.8807935119, learning_rate: 0.0000534495, weight_decay: 0.0000645712

epoch time: 1703.2131347656s, input_fired_time: 38.2250404358, output_fired_time: 15.0470867157, propagate_forward_time: 509.1177062988, propagate_backward_time: 1109.3144531250, weight_update_time: 0.7048981786, other_time: 31.5087890625
epoch time: 101.0178909302s, input_fired_time: 2.3261249065, output_fired_time: 0.0000000000, propagate_forward_time: 93.5750656128, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1166992188
validation[          , genome  6095] predictions:    6332/  10000 (63.32%), best:    6355/10000 (63.55%), error:     10621.76855, best error:     10508.13184 on epoch:     0, epoch:   22/25, mu: 0.8883152008, learning_rate: 0.0000498035, weight_decay: 0.0000606888

epoch time: 1640.8347167969s, input_fired_time: 37.0759963989, output_fired_time: 13.3987598419, propagate_forward_time: 482.1802978516, propagate_backward_time: 1075.9838867188, weight_update_time: 0.5358902812, other_time: 32.1956787109
epoch time: 96.8124618530s, input_fired_time: 2.1494019032, output_fired_time: 0.0000000000, propagate_forward_time: 90.0586929321, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6043701172
validation[          , genome  6095] predictions:    6250/  10000 (62.50%), best:    6355/10000 (63.55%), error:     10799.17676, best error:     10508.13184 on epoch:     0, epoch:   23/25, mu: 0.8953188062, learning_rate: 0.0000464063, weight_decay: 0.0000570399

epoch time: 1554.0527343750s, input_fired_time: 34.4206695557, output_fired_time: 12.0657634735, propagate_forward_time: 460.9675598145, propagate_backward_time: 1019.3662109375, weight_update_time: 0.4970195889, other_time: 27.2325439453
epoch time: 102.0457534790s, input_fired_time: 2.3750996590, output_fired_time: 0.0000000000, propagate_forward_time: 94.4663696289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2042846680
validation[          , genome  6095] predictions:    6138/  10000 (61.38%), best:    6355/10000 (63.55%), error:     11194.66016, best error:     10508.13184 on epoch:     0, epoch:   24/25, mu: 0.9018400311, learning_rate: 0.0000432407, weight_decay: 0.0000536104

epoch time: 1648.1928710938s, input_fired_time: 37.3600959778, output_fired_time: 14.3158626556, propagate_forward_time: 486.8041992188, propagate_backward_time: 1078.6746826172, weight_update_time: 0.5521543622, other_time: 31.0379638672
epoch time: 100.6806716919s, input_fired_time: 2.3125042915, output_fired_time: 0.0000000000, propagate_forward_time: 93.3204040527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0477600098
validation[          , genome  6095] predictions:    6359/  10000 (63.59%), best:    6355/10000 (63.55%), error:     10577.09570, best error:     10508.13184 on epoch:     0, epoch:   25/25, mu: 0.9079121351, learning_rate: 0.0000402911, weight_decay: 0.0000503871

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 536.3916625977s, input_fired_time: 13.5024070740, output_fired_time: 0.0000000000, propagate_forward_time: 493.2277526855, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 29.6615295410
best training[          , genome  6095] predictions:   35249/  50000 (70.50%), best:    6355/10000 (63.55%), error:     44481.98438, best error:     10508.13184 on epoch:     0, epoch:   26/25, mu: 0.9135659933, learning_rate: 0.0000375427, weight_decay: 0.0000473576
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 107.5643844604s, input_fired_time: 2.7947742939, output_fired_time: 0.0000000000, propagate_forward_time: 98.4898681641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2797393799
   testing[          , genome  6095] predictions:    6355/  10000 (63.55%), best:    6355/10000 (63.55%), error:     10508.13184, best error:     10508.13184 on epoch:     0, epoch:   26/25, mu: 0.9135659933, learning_rate: 0.0000375427, weight_decay: 0.0000473576
backpropagation finished successfully!
16:09:16 (6220): called boinc_finish(0)

</stderr_txt>
]]>