Nameexact_genome_1542727803_173_6422_3
Workunit4101314
Created20 Nov 2018, 23:05:42 UTC
Sent20 Nov 2018, 23:05:59 UTC
Report deadline30 Dec 2018, 0:03:39 UTC
Received22 Nov 2018, 18:27:23 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88841
Run time10 hours 18 min 4 sec
CPU time10 hours 18 min 4 sec
Validate stateValid
Credit9,760.01
Device peak FLOPS4.24 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size596.64 MB
Peak swap size595.15 MB
Peak disk usage14.41 MB

Stderr output

<core_client_version>7.6.33</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422_3_r319873370_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549852
read mu: 0.549852
read mu_delta: 0.969923
read initial_learning_rate: 0.00161364
read learning_rate: 0.00161364
read learning_rate_delta: 0.915733
read initial_weight_decay: 0.000322808
read weight_decay: 0.000322808
read weight_decay_delta: 0.91574
read batch_size: 122
read epsilon: 1e-07
read alpha: 0.125314
read input_dropout_probability: 0.00168037
read hidden_dropout_probability: 0.0658385
read velocity_reset: 1812
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.647515892982483 1.24656867980957'
generator_str: '700611225'
read generator: 700611225
read generated_by_map:
1 split_edge 1
reading 113 nodes.
reading 466 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 520022074
post shuffle 1: 1938633477
epoch time: 3189.83447265625s, input_fired_time: 54.5609512329102, output_fired_time: 27.3700065612793, propagate_forward_time: 1252.09887695313, propagate_backward_time: 1823.533203125, weight_update_time: 3.37931180000305, other_time: 32.271240234375
epoch time: 260.167205810547s, input_fired_time: 1.28645074367523, output_fired_time: 0, propagate_forward_time: 250.561096191406, propagate_backward_time: 0, weight_update_time: 0, other_time: 8.31967163085938
validation[          , genome  6422] predictions:    5668/  10000 (56.68%), best:    5668/10000 (56.68%), error:     12504.71094, best error:     12504.71094 on epoch:     0, epoch:    0/25, mu: 0.5498522520, learning_rate: 0.0016136372, weight_decay: 0.0003228079

epoch time: 3380.9316406250s, input_fired_time: 60.0235977173, output_fired_time: 10.0262861252, propagate_forward_time: 1277.8953857422, propagate_backward_time: 1875.6293945313, weight_update_time: 0.3319987059, other_time: 157.3568115234
epoch time: 275.0461425781s, input_fired_time: 4.4217066765, output_fired_time: 0.0000000000, propagate_forward_time: 261.5104370117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.1140136719
validation[          , genome  6422] predictions:    5990/  10000 (59.90%), best:    5990/10000 (59.90%), error:     11566.64844, best error:     11566.64844 on epoch:     1, epoch:    1/25, mu: 0.5630903840, learning_rate: 0.0014776600, weight_decay: 0.0002956082

epoch time: 3446.3718261719s, input_fired_time: 52.4937019348, output_fired_time: 10.9935283661, propagate_forward_time: 1295.7508544922, propagate_backward_time: 1892.3662109375, weight_update_time: 0.3376754820, other_time: 194.7675781250
epoch time: 264.5722045898s, input_fired_time: 4.3383755684, output_fired_time: 0.0000000000, propagate_forward_time: 254.9616851807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2721405029
validation[          , genome  6422] predictions:    6070/  10000 (60.70%), best:    6070/10000 (60.70%), error:     11113.66797, best error:     11113.66797 on epoch:     2, epoch:    2/25, mu: 0.5759303570, learning_rate: 0.0013531413, weight_decay: 0.0002707004

epoch time: 3374.9267578125s, input_fired_time: 38.7186775208, output_fired_time: 21.4734649658, propagate_forward_time: 1312.9544677734, propagate_backward_time: 1922.8662109375, weight_update_time: 1.8430304527, other_time: 78.9139404297
epoch time: 271.0561523438s, input_fired_time: 4.4141445160, output_fired_time: 0.0000000000, propagate_forward_time: 262.6311340332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0108642578
validation[          , genome  6422] predictions:    6040/  10000 (60.40%), best:    6070/10000 (60.70%), error:     11406.20703, best error:     11113.66797 on epoch:     2, epoch:    3/25, mu: 0.5883841515, learning_rate: 0.0012391155, weight_decay: 0.0002478913

epoch time: 3370.3918457031s, input_fired_time: 57.4980278015, output_fired_time: 18.5677661896, propagate_forward_time: 1321.4433593750, propagate_backward_time: 1945.7470703125, weight_update_time: 0.3460739851, other_time: 27.1354980469
epoch time: 268.4280700684s, input_fired_time: 1.3527886868, output_fired_time: 0.0000000000, propagate_forward_time: 252.8274383545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 14.2478485107
validation[          , genome  6422] predictions:    6147/  10000 (61.47%), best:    6147/10000 (61.47%), error:     11036.12695, best error:     11036.12695 on epoch:     4, epoch:    4/25, mu: 0.6004633904, learning_rate: 0.0011346983, weight_decay: 0.0002270041

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422_3_r319873370_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549852
read mu: 0.612179
read mu_delta: 0.969923
read initial_learning_rate: 0.00161364
read learning_rate: 0.00103908
read learning_rate_delta: 0.915733
read initial_weight_decay: 0.000322808
read weight_decay: 0.000207877
read weight_decay_delta: 0.91574
read batch_size: 122
read epsilon: 1e-07
read alpha: 0.125314
read input_dropout_probability: 0.00168037
read hidden_dropout_probability: 0.0658385
read velocity_reset: 1812
read epoch: 5
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 4
read number_validation_images: 10000
read best_validation_predictions: 6147
read best_validation_error: 11036.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.647515892982483 1.24656867980957'
generator_str: '2076790530'
read generator: 2076790530
read generated_by_map:
1 split_edge 1
reading 113 nodes.
reading 466 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_173_6422_3_r319873370_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549852
read mu: 0.612179
read mu_delta: 0.969923
read initial_learning_rate: 0.00161364
read learning_rate: 0.00103908
read learning_rate_delta: 0.915733
read initial_weight_decay: 0.000322808
read weight_decay: 0.000207877
read weight_decay_delta: 0.91574
read batch_size: 122
read epsilon: 1e-07
read alpha: 0.125314
read input_dropout_probability: 0.00168037
read hidden_dropout_probability: 0.0658385
read velocity_reset: 1812
read epoch: 5
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 4
read number_validation_images: 10000
read best_validation_predictions: 6147
read best_validation_error: 11036.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.647515892982483 1.24656867980957'
generator_str: '2076790530'
read generator: 2076790530
read generated_by_map:
1 split_edge 1
reading 113 nodes.
reading 466 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3388.53588867188s, input_fired_time: 62.1284446716309, output_fired_time: 18.7513618469238, propagate_forward_time: 1327.72204589844, propagate_backward_time: 1942.90234375, weight_update_time: 0.346030056476593, other_time: 37.0316162109375
epoch time: 302.650299072266s, input_fired_time: 14.9783000946045, output_fired_time: 0, propagate_forward_time: 280.337982177734, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.33401489257813
validation[          , genome  6422] predictions:    6199/  10000 (61.99%), best:    6199/10000 (61.99%), error:     10892.78516, best error:     10892.78516 on epoch:     5, epoch:    5/25, mu: 0.6121793389, learning_rate: 0.0010390801, weight_decay: 0.0002078768

epoch time: 3495.1284179688s, input_fired_time: 60.5617294312, output_fired_time: 14.3715019226, propagate_forward_time: 1372.0703125000, propagate_backward_time: 2017.6027832031, weight_update_time: 0.3609464169, other_time: 30.5219726563
epoch time: 302.3327636719s, input_fired_time: 5.9640612602, output_fired_time: 0.0000000000, propagate_forward_time: 287.7504882813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.6182250977
validation[          , genome  6422] predictions:    6120/  10000 (61.20%), best:    6199/10000 (61.99%), error:     10943.28906, best error:     10892.78516 on epoch:     5, epoch:    6/25, mu: 0.6235429049, learning_rate: 0.0009515194, weight_decay: 0.0001903612

epoch time: 3248.1601562500s, input_fired_time: 55.6577262878, output_fired_time: 24.5814590454, propagate_forward_time: 1264.4967041016, propagate_backward_time: 1869.2351074219, weight_update_time: 1.8515169621, other_time: 34.1890869141
epoch time: 258.2248840332s, input_fired_time: 5.9128985405, output_fired_time: 0.0000000000, propagate_forward_time: 245.5445709229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7674102783
validation[          , genome  6422] predictions:    6348/  10000 (63.48%), best:    6348/10000 (63.48%), error:     10605.16602, best error:     10605.16602 on epoch:     7, epoch:    7/25, mu: 0.6345646977, learning_rate: 0.0008713373, weight_decay: 0.0001743214

epoch time: 3228.5942382813s, input_fired_time: 43.8316040039, output_fired_time: 9.1298055649, propagate_forward_time: 1270.8149414063, propagate_backward_time: 1880.5708007813, weight_update_time: 0.3180816472, other_time: 24.2470703125
epoch time: 269.5434570313s, input_fired_time: 6.0360527039, output_fired_time: 0.0000000000, propagate_forward_time: 258.2040405273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3033752441
validation[          , genome  6422] predictions:    6213/  10000 (62.13%), best:    6348/10000 (63.48%), error:     10849.24707, best error:     10605.16602 on epoch:     7, epoch:    8/25, mu: 0.6452549696, learning_rate: 0.0007979118, weight_decay: 0.0001596332

epoch time: 3208.6408691406s, input_fired_time: 53.4706039429, output_fired_time: 19.8293304443, propagate_forward_time: 1286.6977539063, propagate_backward_time: 1822.2008056641, weight_update_time: 0.3193342388, other_time: 26.4422607422
epoch time: 260.3830261230s, input_fired_time: 2.8704838753, output_fired_time: 0.0000000000, propagate_forward_time: 249.1475524902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3649902344
validation[          , genome  6422] predictions:    6168/  10000 (61.68%), best:    6348/10000 (63.48%), error:     10996.20605, best error:     10605.16602 on epoch:     7, epoch:    9/25, mu: 0.6556237340, learning_rate: 0.0007306738, weight_decay: 0.0001461825

epoch time: 3227.2028808594s, input_fired_time: 48.9590530396, output_fired_time: 27.4301700592, propagate_forward_time: 1262.1889648438, propagate_backward_time: 1848.7448730469, weight_update_time: 0.3239121735, other_time: 39.8798828125
epoch time: 262.7259521484s, input_fired_time: 2.8686976433, output_fired_time: 0.0000000000, propagate_forward_time: 254.5854492188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2718200684
validation[          , genome  6422] predictions:    6157/  10000 (61.57%), best:    6348/10000 (63.48%), error:     11007.75586, best error:     10605.16602 on epoch:     7, epoch:   10/25, mu: 0.6656806469, learning_rate: 0.0006691017, weight_decay: 0.0001338653

epoch time: 3199.3010253906s, input_fired_time: 32.5802001953, output_fired_time: 18.2776794434, propagate_forward_time: 1289.4281005859, propagate_backward_time: 1828.0314941406, weight_update_time: 0.3215483129, other_time: 30.9835205078
epoch time: 260.7143554688s, input_fired_time: 4.4032316208, output_fired_time: 0.0000000000, propagate_forward_time: 254.1188354492, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1922912598
validation[          , genome  6422] predictions:    6292/  10000 (62.92%), best:    6348/10000 (63.48%), error:     10642.71973, best error:     10605.16602 on epoch:     7, epoch:   11/25, mu: 0.6754350662, learning_rate: 0.0006127182, weight_decay: 0.0001225858

epoch time: 3213.8632812500s, input_fired_time: 59.2498779297, output_fired_time: 13.8206119537, propagate_forward_time: 1245.4807128906, propagate_backward_time: 1853.4484863281, weight_update_time: 1.8485256433, other_time: 41.8635253906
epoch time: 270.2401733398s, input_fired_time: 2.8853981495, output_fired_time: 0.0000000000, propagate_forward_time: 265.1607360840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1940307617
validation[          , genome  6422] predictions:    6378/  10000 (63.78%), best:    6378/10000 (63.78%), error:     10372.88184, best error:     10372.88184 on epoch:    12, epoch:   12/25, mu: 0.6848961115, learning_rate: 0.0005610859, weight_decay: 0.0001122568

epoch time: 3225.7072753906s, input_fired_time: 44.7810249329, output_fired_time: 21.4725151062, propagate_forward_time: 1273.0072021484, propagate_backward_time: 1849.5180664063, weight_update_time: 1.8455243111, other_time: 36.9285888672
epoch time: 262.2768249512s, input_fired_time: 2.8790447712, output_fired_time: 0.0000000000, propagate_forward_time: 255.3385620117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0592041016
validation[          , genome  6422] predictions:    6324/  10000 (63.24%), best:    6378/10000 (63.78%), error:     10433.01270, best error:     10372.88184 on epoch:    12, epoch:   13/25, mu: 0.6940726042, learning_rate: 0.0005138046, weight_decay: 0.0001027981

epoch time: 3139.3725585938s, input_fired_time: 70.3415832520, output_fired_time: 19.6326408386, propagate_forward_time: 1231.6126708984, propagate_backward_time: 1796.3701171875, weight_update_time: 0.3089716733, other_time: 21.4156494141
epoch time: 250.0745544434s, input_fired_time: 5.8767971992, output_fired_time: 0.0000000000, propagate_forward_time: 242.0745849609, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1231689453
validation[          , genome  6422] predictions:    6193/  10000 (61.93%), best:    6378/10000 (63.78%), error:     10976.60156, best error:     10372.88184 on epoch:    12, epoch:   14/25, mu: 0.7029730678, learning_rate: 0.0004705076, weight_decay: 0.0000941363

epoch time: 3118.5493164063s, input_fired_time: 59.4108657837, output_fired_time: 13.3993024826, propagate_forward_time: 1235.4437255859, propagate_backward_time: 1775.2103271484, weight_update_time: 0.3030244112, other_time: 35.0849609375
epoch time: 255.6577301025s, input_fired_time: 5.8880505562, output_fired_time: 0.0000000000, propagate_forward_time: 246.1275939941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6420898438
validation[          , genome  6422] predictions:    6310/  10000 (63.10%), best:    6378/10000 (63.78%), error:     10565.17773, best error:     10372.88184 on epoch:    12, epoch:   15/25, mu: 0.7116058469, learning_rate: 0.0004308591, weight_decay: 0.0000862044

epoch time: 3096.4199218750s, input_fired_time: 57.6617279053, output_fired_time: 18.1019458771, propagate_forward_time: 1256.8482666016, propagate_backward_time: 1730.3728027344, weight_update_time: 0.3031122983, other_time: 33.4351806641
epoch time: 247.6587371826s, input_fired_time: 1.2957254648, output_fired_time: 0.0000000000, propagate_forward_time: 235.0893859863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.2736206055
validation[          , genome  6422] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10166.17188, best error:     10166.17188 on epoch:    16, epoch:   16/25, mu: 0.7199789882, learning_rate: 0.0003945517, weight_decay: 0.0000789409

epoch time: 3089.2836914063s, input_fired_time: 50.3319931030, output_fired_time: 15.0220994949, propagate_forward_time: 1204.7773437500, propagate_backward_time: 1785.7260742188, weight_update_time: 3.3630630970, other_time: 33.4260253906
epoch time: 256.0047912598s, input_fired_time: 2.8339478970, output_fired_time: 0.0000000000, propagate_forward_time: 249.5254669189, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6453704834
validation[          , genome  6422] predictions:    6497/  10000 (64.97%), best:    6497/10000 (64.97%), error:     10082.85547, best error:     10082.85547 on epoch:    17, epoch:   17/25, mu: 0.7281002998, learning_rate: 0.0003613038, weight_decay: 0.0000722894

epoch time: 3093.9079589844s, input_fired_time: 55.2848167419, output_fired_time: 15.0571918488, propagate_forward_time: 1226.8168945313, propagate_backward_time: 1769.3431396484, weight_update_time: 0.3068650663, other_time: 27.4058837891
epoch time: 252.8159179688s, input_fired_time: 5.7934875488, output_fired_time: 0.0000000000, propagate_forward_time: 243.3677520752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6546783447
validation[          , genome  6422] predictions:    6452/  10000 (64.52%), best:    6497/10000 (64.97%), error:     10103.45605, best error:     10082.85547 on epoch:    17, epoch:   18/25, mu: 0.7359773517, learning_rate: 0.0003308576, weight_decay: 0.0000661983

epoch time: 3093.9907226563s, input_fired_time: 51.6597938538, output_fired_time: 24.1261730194, propagate_forward_time: 1222.4993896484, propagate_backward_time: 1769.8543701172, weight_update_time: 0.3060196638, other_time: 25.8508300781
epoch time: 254.4441528320s, input_fired_time: 2.8422367573, output_fired_time: 0.0000000000, propagate_forward_time: 246.4081878662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1937255859
validation[          , genome  6422] predictions:    6556/  10000 (65.56%), best:    6556/10000 (65.56%), error:     10040.54590, best error:     10040.54590 on epoch:    19, epoch:   19/25, mu: 0.7436174750, learning_rate: 0.0003029771, weight_decay: 0.0000606204

epoch time: 3081.0827636719s, input_fired_time: 49.1711082458, output_fired_time: 19.5216846466, propagate_forward_time: 1220.7456054688, propagate_backward_time: 1758.1433105469, weight_update_time: 1.8360472918, other_time: 33.5009765625
epoch time: 252.5988464355s, input_fired_time: 1.3192791939, output_fired_time: 0.0000000000, propagate_forward_time: 249.1787414551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1008300781
validation[          , genome  6422] predictions:    6510/  10000 (65.10%), best:    6510/10000 (65.10%), error:     10007.61035, best error:     10007.61035 on epoch:    20, epoch:   20/25, mu: 0.7510278225, learning_rate: 0.0002774459, weight_decay: 0.0000555126

epoch time: 3091.7458496094s, input_fired_time: 48.7247505188, output_fired_time: 22.5996360779, propagate_forward_time: 1224.2194824219, propagate_backward_time: 1766.0036621094, weight_update_time: 0.3032393456, other_time: 30.1982421875
epoch time: 255.9414672852s, input_fired_time: 2.8500080109, output_fired_time: 0.0000000000, propagate_forward_time: 249.4592895508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6321716309
validation[          , genome  6422] predictions:    6524/  10000 (65.24%), best:    6524/10000 (65.24%), error:      9905.63574, best error:      9905.63574 on epoch:    21, epoch:   21/25, mu: 0.7582153082, learning_rate: 0.0002540663, weight_decay: 0.0000508351

epoch time: 3112.9941406250s, input_fired_time: 52.4995117188, output_fired_time: 13.5235137939, propagate_forward_time: 1239.9864501953, propagate_backward_time: 1775.2636718750, weight_update_time: 0.3040542305, other_time: 31.7210693359
epoch time: 252.8035430908s, input_fired_time: 4.3782086372, output_fired_time: 0.0000000000, propagate_forward_time: 243.2651062012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1602325439
validation[          , genome  6422] predictions:    6456/  10000 (64.56%), best:    6524/10000 (65.24%), error:     10119.50195, best error:      9905.63574 on epoch:    21, epoch:   22/25, mu: 0.7651866078, learning_rate: 0.0002326567, weight_decay: 0.0000465518

epoch time: 3126.7387695313s, input_fired_time: 39.7974510193, output_fired_time: 11.9823608398, propagate_forward_time: 1238.4316406250, propagate_backward_time: 1807.0081787109, weight_update_time: 0.3039582074, other_time: 29.5191650391
epoch time: 260.1025390625s, input_fired_time: 2.8560836315, output_fired_time: 0.0000000000, propagate_forward_time: 252.0789184570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1675415039
validation[          , genome  6422] predictions:    6478/  10000 (64.78%), best:    6524/10000 (65.24%), error:     10083.16992, best error:      9905.63574 on epoch:    21, epoch:   23/25, mu: 0.7719482183, learning_rate: 0.0002130513, weight_decay: 0.0000426293

epoch time: 3123.1520996094s, input_fired_time: 50.7235832214, output_fired_time: 13.5443372726, propagate_forward_time: 1240.4976806641, propagate_backward_time: 1781.9465332031, weight_update_time: 0.3031524718, other_time: 36.4398193359
epoch time: 261.6384277344s, input_fired_time: 5.8757114410, output_fired_time: 0.0000000000, propagate_forward_time: 247.5491333008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2135772705
validation[          , genome  6422] predictions:    6550/  10000 (65.50%), best:    6524/10000 (65.24%), error:      9956.15527, best error:      9905.63574 on epoch:    21, epoch:   24/25, mu: 0.7785064578, learning_rate: 0.0001950980, weight_decay: 0.0000390374

epoch time: 3207.5480957031s, input_fired_time: 54.3036460876, output_fired_time: 13.8269014359, propagate_forward_time: 1259.7703857422, propagate_backward_time: 1850.3391113281, weight_update_time: 0.3247075677, other_time: 29.3079833984
epoch time: 262.7836303711s, input_fired_time: 4.4577918053, output_fired_time: 0.0000000000, propagate_forward_time: 255.9530334473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3728027344
validation[          , genome  6422] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:      9891.76953, best error:      9891.76953 on epoch:    25, epoch:   25/25, mu: 0.7848674655, learning_rate: 0.0001786576, weight_decay: 0.0000357481

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1319.5170898438s, input_fired_time: 15.5429744720, output_fired_time: 0.0000000000, propagate_forward_time: 1280.1931152344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 23.7810058594
best training[          , genome  6422] predictions:   36372/  50000 (72.74%), best:    6546/10000 (65.46%), error:     40387.73438, best error:      9891.76953 on epoch:    25, epoch:   26/25, mu: 0.7910371423, learning_rate: 0.0001636026, weight_decay: 0.0000327360
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 270.8117980957s, input_fired_time: 2.9544501305, output_fired_time: 0.0000000000, propagate_forward_time: 261.0077514648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8496093750
   testing[          , genome  6422] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:      9891.76953, best error:      9891.76953 on epoch:    25, epoch:   26/25, mu: 0.7910371423, learning_rate: 0.0001636026, weight_decay: 0.0000327360
backpropagation finished successfully!
10:39:11 (4880): called boinc_finish(0)

</stderr_txt>
]]>