Nameexact_genome_1542692703_178_6370_2
Workunit4100303
Created20 Nov 2018, 8:26:01 UTC
Sent20 Nov 2018, 9:03:12 UTC
Report deadline21 Jan 2019, 10:21:24 UTC
Received25 Nov 2018, 6:18:04 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID75834
Run time19 hours 57 min 25 sec
CPU time19 hours 56 min 9 sec
Validate stateValid
Credit15,513.58
Device peak FLOPS3.32 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34

Stderr output

<core_client_version>7.0.80</core_client_version>
<![CDATA[
<stderr_txt>
ght: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.647268
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.00033874
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000360088
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 7
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 4
read number_validation_images: 10000
read best_validation_predictions: 5878
read best_validation_error: 11958.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '502580306'
read generator: 502580306
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.647268
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.00033874
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000360088
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 7
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 4
read number_validation_images: 10000
read best_validation_predictions: 5878
read best_validation_error: 11958.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '502580306'
read generator: 502580306
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4358.92822265625s, input_fired_time: 41.432445526123, output_fired_time: 18.1713256835938, propagate_forward_time: 1747.27746582031, propagate_backward_time: 2505.33349609375, weight_update_time: 2.58269095420837, other_time: 46.71337890625
epoch time: 346.994323730469s, input_fired_time: 2.2908091545105, output_fired_time: 0, propagate_forward_time: 337.703186035156, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.00033569335938
validation[          , genome  6370] predictions:    5509/  10000 (55.09%), best:    5878/10000 (58.78%), error:     13057.38281, best error:     11958.82813 on epoch:     4, epoch:    7/50, mu: 0.6472680569, learning_rate: 0.0003387403, weight_decay: 0.0003600882

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.665634
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.000309582
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000332588
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 8
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 4
read number_validation_images: 10000
read best_validation_predictions: 5878
read best_validation_error: 11958.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '2102446344'
read generator: 2102446344
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4189.4345703125s, input_fired_time: 33.7424163818359, output_fired_time: 14.8245897293091, propagate_forward_time: 1696.05358886719, propagate_backward_time: 2398.73315429688, weight_update_time: 4.56260347366333, other_time: 46.080810546875
epoch time: 330.969299316406s, input_fired_time: 4.23800849914551, output_fired_time: 0, propagate_forward_time: 321.977783203125, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.75350952148438
validation[          , genome  6370] predictions:    5839/  10000 (58.39%), best:    5878/10000 (58.78%), error:     12263.33691, best error:     11958.82813 on epoch:     4, epoch:    8/50, mu: 0.6656336188, learning_rate: 0.0003095822, weight_decay: 0.0003325879

epoch time: 3973.2280273438s, input_fired_time: 44.2427940369, output_fired_time: 26.4139080048, propagate_forward_time: 1592.5985107422, propagate_backward_time: 2271.4997558594, weight_update_time: 1.5197752714, other_time: 38.4733886719
epoch time: 323.1374816895s, input_fired_time: 3.2208397388, output_fired_time: 0.0000000000, propagate_forward_time: 313.4270019531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4896545410
validation[          , genome  6370] predictions:    5856/  10000 (58.56%), best:    5878/10000 (58.78%), error:     12033.66309, best error:     11958.82813 on epoch:     4, epoch:    9/50, mu: 0.6830150485, learning_rate: 0.0002829339, weight_decay: 0.0003071878

epoch time: 3971.6059570313s, input_fired_time: 54.8988800049, output_fired_time: 16.4529514313, propagate_forward_time: 1584.7797851563, propagate_backward_time: 2276.1584472656, weight_update_time: 2.5262041092, other_time: 39.3159179688
epoch time: 327.1867065430s, input_fired_time: 8.2530422211, output_fired_time: 0.0000000000, propagate_forward_time: 312.4292602539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5043945313
validation[          , genome  6370] predictions:    5756/  10000 (57.56%), best:    5878/10000 (58.78%), error:     12326.87109, best error:     11958.82813 on epoch:     4, epoch:   10/50, mu: 0.6994650364, learning_rate: 0.0002585795, weight_decay: 0.0002837276

epoch time: 3947.3454589844s, input_fired_time: 40.2040290833, output_fired_time: 13.3975143433, propagate_forward_time: 1593.0120849609, propagate_backward_time: 2266.3898925781, weight_update_time: 0.5153633952, other_time: 34.3420410156
epoch time: 326.8926086426s, input_fired_time: 6.2442522049, output_fired_time: 0.0000000000, propagate_forward_time: 316.1620178223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4863281250
validation[          , genome  6370] predictions:    5870/  10000 (58.70%), best:    5878/10000 (58.78%), error:     12015.41602, best error:     11958.82813 on epoch:     4, epoch:   11/50, mu: 0.7150335312, learning_rate: 0.0002363215, weight_decay: 0.0002620591

epoch time: 3960.3908691406s, input_fired_time: 52.2151069641, output_fired_time: 16.2609634399, propagate_forward_time: 1584.2778320313, propagate_backward_time: 2268.2192382813, weight_update_time: 0.5208065510, other_time: 39.4177246094
epoch time: 327.3403930664s, input_fired_time: 6.2453746796, output_fired_time: 0.0000000000, propagate_forward_time: 309.5417785645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.5532531738
validation[          , genome  6370] predictions:    6031/  10000 (60.31%), best:    6031/10000 (60.31%), error:     11608.99512, best error:     11608.99512 on epoch:    12, epoch:   12/50, mu: 0.7297677994, learning_rate: 0.0002159794, weight_decay: 0.0002420454

epoch time: 3692.3254394531s, input_fired_time: 51.7283020020, output_fired_time: 16.8215484619, propagate_forward_time: 1482.5087890625, propagate_backward_time: 2099.1762695313, weight_update_time: 1.4949883223, other_time: 42.0905761719
epoch time: 290.3284912109s, input_fired_time: 1.1060956717, output_fired_time: 0.0000000000, propagate_forward_time: 281.7709655762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4514160156
validation[          , genome  6370] predictions:    5951/  10000 (59.51%), best:    6031/10000 (60.31%), error:     11860.06836, best error:     11608.99512 on epoch:    12, epoch:   13/50, mu: 0.7437125444, learning_rate: 0.0001973883, weight_decay: 0.0002235602

epoch time: 3558.6999511719s, input_fired_time: 33.4726562500, output_fired_time: 24.6305656433, propagate_forward_time: 1403.4792480469, propagate_backward_time: 2066.4550781250, weight_update_time: 0.4656341970, other_time: 30.6623535156
epoch time: 290.3158874512s, input_fired_time: 2.1229100227, output_fired_time: 0.0000000000, propagate_forward_time: 282.7668151855, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4261474609
validation[          , genome  6370] predictions:    5657/  10000 (56.57%), best:    6031/10000 (60.31%), error:     12482.42578, best error:     11608.99512 on epoch:    12, epoch:   14/50, mu: 0.7569100261, learning_rate: 0.0001803975, weight_decay: 0.0002064867

epoch time: 3555.9763183594s, input_fired_time: 52.5814132690, output_fired_time: 17.5965099335, propagate_forward_time: 1389.5416259766, propagate_backward_time: 2059.4333496094, weight_update_time: 0.4718811214, other_time: 36.8234863281
epoch time: 250.8625793457s, input_fired_time: 3.0180151463, output_fired_time: 0.0000000000, propagate_forward_time: 245.6066131592, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2379455566
validation[          , genome  6370] predictions:    5958/  10000 (59.58%), best:    6031/10000 (60.31%), error:     11751.99902, best error:     11608.99512 on epoch:    12, epoch:   15/50, mu: 0.7694002986, learning_rate: 0.0001648692, weight_decay: 0.0001907171

epoch time: 3126.2827148438s, input_fired_time: 36.8067703247, output_fired_time: 13.6539955139, propagate_forward_time: 1245.1618652344, propagate_backward_time: 1799.5494384766, weight_update_time: 0.4195486009, other_time: 31.1104736328
epoch time: 252.5651397705s, input_fired_time: 4.0245952606, output_fired_time: 0.0000000000, propagate_forward_time: 243.2718200684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2687225342
validation[          , genome  6370] predictions:    5968/  10000 (59.68%), best:    5968/10000 (59.68%), error:     11499.74609, best error:     11499.74609 on epoch:    16, epoch:   16/50, mu: 0.7812212706, learning_rate: 0.0001506776, weight_decay: 0.0001761519

epoch time: 3079.5161132813s, input_fired_time: 25.4906635284, output_fired_time: 20.5233268738, propagate_forward_time: 1193.0488281250, propagate_backward_time: 1799.1981201172, weight_update_time: 0.4193812013, other_time: 41.2550048828
epoch time: 247.8488616943s, input_fired_time: 1.0100251436, output_fired_time: 0.0000000000, propagate_forward_time: 242.4978179932, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3410186768
validation[          , genome  6370] predictions:    6020/  10000 (60.20%), best:    5968/10000 (59.68%), error:     11557.54688, best error:     11499.74609 on epoch:    16, epoch:   17/50, mu: 0.7924088240, learning_rate: 0.0001377075, weight_decay: 0.0001626990

epoch time: 3056.2001953125s, input_fired_time: 27.4734516144, output_fired_time: 17.4979248047, propagate_forward_time: 1207.4271240234, propagate_backward_time: 1769.7507324219, weight_update_time: 0.4167406559, other_time: 34.0511474609
epoch time: 251.9750518799s, input_fired_time: 4.9191889763, output_fired_time: 0.0000000000, propagate_forward_time: 241.7819213867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2739410400
validation[          , genome  6370] predictions:    5886/  10000 (58.86%), best:    5968/10000 (59.68%), error:     12046.28027, best error:     11499.74609 on epoch:    16, epoch:   18/50, mu: 0.8029968739, learning_rate: 0.0001258539, weight_decay: 0.0001502735

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.813018
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.000115021
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000138797
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 19
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 16
read number_validation_images: 10000
read best_validation_predictions: 5968
read best_validation_error: 11499.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '729654118'
read generator: 729654118
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.813018
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.000115021
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000138797
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 19
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 16
read number_validation_images: 10000
read best_validation_predictions: 5968
read best_validation_error: 11499.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '729654118'
read generator: 729654118
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3154.48120117188s, input_fired_time: 36.8534545898438, output_fired_time: 22.7913494110107, propagate_forward_time: 1233.40197753906, propagate_backward_time: 1827.17614746094, weight_update_time: 0.425642549991608, other_time: 34.25830078125
epoch time: 257.017639160156s, input_fired_time: 2.02762198448181, output_fired_time: 0, propagate_forward_time: 249.717727661133, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.27229309082031
validation[          , genome  6370] predictions:    5973/  10000 (59.73%), best:    5968/10000 (59.68%), error:     11699.58008, best error:     11499.74609 on epoch:    16, epoch:   19/50, mu: 0.8130175471, learning_rate: 0.0001150207, weight_decay: 0.0001387970

epoch time: 2966.2092285156s, input_fired_time: 36.5715141296, output_fired_time: 17.3508319855, propagate_forward_time: 1175.7392578125, propagate_backward_time: 1704.8112792969, weight_update_time: 0.4054704309, other_time: 31.7363281250
epoch time: 246.9610748291s, input_fired_time: 8.0480480194, output_fired_time: 0.0000000000, propagate_forward_time: 234.5364227295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3766021729
validation[          , genome  6370] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11338.86621, best error:     11338.86621 on epoch:    20, epoch:   20/50, mu: 0.8225013018, learning_rate: 0.0001051199, weight_decay: 0.0001281970

epoch time: 3097.8193359375s, input_fired_time: 42.8280334473, output_fired_time: 20.6029720306, propagate_forward_time: 1194.7557373047, propagate_backward_time: 1804.6828613281, weight_update_time: 0.4225010872, other_time: 34.9495849609
epoch time: 256.7962036133s, input_fired_time: 1.0329406261, output_fired_time: 0.0000000000, propagate_forward_time: 252.1074676514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6557922363
validation[          , genome  6370] predictions:    6146/  10000 (61.46%), best:    6146/10000 (61.46%), error:     11094.59961, best error:     11094.59961 on epoch:    21, epoch:   21/50, mu: 0.8314768672, learning_rate: 0.0000960714, weight_decay: 0.0001184064

epoch time: 3158.8715820313s, input_fired_time: 36.8818969727, output_fired_time: 26.7504329681, propagate_forward_time: 1245.1877441406, propagate_backward_time: 1817.8176269531, weight_update_time: 1.4288250208, other_time: 32.2338867188
epoch time: 250.4124450684s, input_fired_time: 1.0067036152, output_fired_time: 0.0000000000, propagate_forward_time: 243.8988342285, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5069122314
validation[          , genome  6370] predictions:    6119/  10000 (61.19%), best:    6146/10000 (61.46%), error:     11298.76465, best error:     11094.59961 on epoch:    21, epoch:   22/50, mu: 0.8399714231, learning_rate: 0.0000878017, weight_decay: 0.0001093636

epoch time: 3111.8190917969s, input_fired_time: 39.0681381226, output_fired_time: 25.4665298462, propagate_forward_time: 1203.8930664063, propagate_backward_time: 1811.2852783203, weight_update_time: 3.4552156925, other_time: 32.1060791016
epoch time: 249.6203155518s, input_fired_time: 1.0018368959, output_fired_time: 0.0000000000, propagate_forward_time: 245.2828674316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3356170654
validation[          , genome  6370] predictions:    6064/  10000 (60.64%), best:    6146/10000 (61.46%), error:     11349.50391, best error:     11094.59961 on epoch:    21, epoch:   23/50, mu: 0.8480108380, learning_rate: 0.0000802439, weight_decay: 0.0001010114

epoch time: 3060.8505859375s, input_fired_time: 46.6054573059, output_fired_time: 14.4741878510, propagate_forward_time: 1193.3901367188, propagate_backward_time: 1773.0102539063, weight_update_time: 2.4352602959, other_time: 33.3706054688
epoch time: 249.7177124023s, input_fired_time: 3.9230325222, output_fired_time: 0.0000000000, propagate_forward_time: 242.3790588379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4156188965
validation[          , genome  6370] predictions:    6164/  10000 (61.64%), best:    6146/10000 (61.46%), error:     11102.73926, best error:     11094.59961 on epoch:    21, epoch:   24/50, mu: 0.8556194305, learning_rate: 0.0000733367, weight_decay: 0.0000932971

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_2_r1579818876_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.86282
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 6.7024e-05
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 8.61719e-05
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 25
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 21
read number_validation_images: 10000
read best_validation_predictions: 6146
read best_validation_error: 11094.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '1106946617'
read generator: 1106946617
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2757.40014648438s, input_fired_time: 46.2103424072266, output_fired_time: 15.7693004608154, propagate_forward_time: 1071.86340332031, propagate_backward_time: 1586.01306152344, weight_update_time: 0.383951097726822, other_time: 37.5439453125
epoch time: 223.197067260742s, input_fired_time: 1.93024873733521, output_fired_time: 0, propagate_forward_time: 218.08674621582, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.18006896972656
validation[          , genome  6370] predictions:    6194/  10000 (61.94%), best:    6194/10000 (61.94%), error:     11048.55664, best error:     11048.55664 on epoch:    25, epoch:   25/50, mu: 0.8628203273, learning_rate: 0.0000670240, weight_decay: 0.0000861719

epoch time: 2932.3295898438s, input_fired_time: 31.5476551056, output_fired_time: 12.2167968750, propagate_forward_time: 1153.0301513672, propagate_backward_time: 1698.3889160156, weight_update_time: 0.3986062407, other_time: 37.1461181641
epoch time: 220.3916320801s, input_fired_time: 0.9339177608, output_fired_time: 0.0000000000, propagate_forward_time: 211.1205139160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3372039795
validation[          , genome  6370] predictions:    6128/  10000 (61.28%), best:    6194/10000 (61.94%), error:     11242.00977, best error:     11048.55664 on epoch:    25, epoch:   26/50, mu: 0.8696353436, learning_rate: 0.0000612547, weight_decay: 0.0000795909

epoch time: 2750.6806640625s, input_fired_time: 35.1089973450, output_fired_time: 11.7711906433, propagate_forward_time: 1062.4981689453, propagate_backward_time: 1602.7038574219, weight_update_time: 1.3932594061, other_time: 38.5985107422
epoch time: 231.6068267822s, input_fired_time: 2.9805753231, output_fired_time: 0.0000000000, propagate_forward_time: 224.3050384521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3212127686
validation[          , genome  6370] predictions:    6249/  10000 (62.49%), best:    6249/10000 (62.49%), error:     10933.91895, best error:     10933.91895 on epoch:    27, epoch:   27/50, mu: 0.8760851622, learning_rate: 0.0000559820, weight_decay: 0.0000735125

epoch time: 2829.6665039063s, input_fired_time: 28.2494220734, output_fired_time: 12.9128170013, propagate_forward_time: 1110.9205322266, propagate_backward_time: 1641.6781005859, weight_update_time: 0.3975059986, other_time: 35.9055175781
epoch time: 223.0215454102s, input_fired_time: 0.9292191267, output_fired_time: 0.0000000000, propagate_forward_time: 211.9207916260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.1715393066
validation[          , genome  6370] predictions:    6247/  10000 (62.47%), best:    6247/10000 (62.47%), error:     10818.81836, best error:     10818.81836 on epoch:    28, epoch:   28/50, mu: 0.8821893930, learning_rate: 0.0000511632, weight_decay: 0.0000678983

epoch time: 2735.6445312500s, input_fired_time: 25.0603218079, output_fired_time: 18.6284675598, propagate_forward_time: 1056.8342285156, propagate_backward_time: 1598.7900390625, weight_update_time: 0.3754286766, other_time: 36.3315429688
epoch time: 221.0479583740s, input_fired_time: 2.9460504055, output_fired_time: 0.0000000000, propagate_forward_time: 211.8582611084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2436523438
validation[          , genome  6370] predictions:    6246/  10000 (62.46%), best:    6247/10000 (62.47%), error:     10832.40430, best error:     10818.81836 on epoch:    28, epoch:   29/50, mu: 0.8879665136, learning_rate: 0.0000467592, weight_decay: 0.0000627128

epoch time: 2734.4331054688s, input_fired_time: 29.0772266388, output_fired_time: 14.7111101151, propagate_forward_time: 1056.1752929688, propagate_backward_time: 1600.0651855469, weight_update_time: 2.3897011280, other_time: 34.4042968750
epoch time: 220.3259124756s, input_fired_time: 2.9258923531, output_fired_time: 0.0000000000, propagate_forward_time: 212.3489379883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0510864258
validation[          , genome  6370] predictions:    6302/  10000 (63.02%), best:    6302/10000 (63.02%), error:     10750.92480, best error:     10750.92480 on epoch:    30, epoch:   30/50, mu: 0.8934340477, learning_rate: 0.0000427342, weight_decay: 0.0000579234

epoch time: 2729.4111328125s, input_fired_time: 33.0048675537, output_fired_time: 16.7255420685, propagate_forward_time: 1064.6790771484, propagate_backward_time: 1588.5803222656, weight_update_time: 2.3902425766, other_time: 26.4212646484
epoch time: 225.5932312012s, input_fired_time: 2.9448270798, output_fired_time: 0.0000000000, propagate_forward_time: 218.4715728760, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1768341064
validation[          , genome  6370] predictions:    6387/  10000 (63.87%), best:    6387/10000 (63.87%), error:     10551.79980, best error:     10551.79980 on epoch:    31, epoch:   31/50, mu: 0.8986086249, learning_rate: 0.0000390557, weight_decay: 0.0000534997

epoch time: 2747.4677734375s, input_fired_time: 31.1299076080, output_fired_time: 12.7225017548, propagate_forward_time: 1090.4481201172, propagate_backward_time: 1585.7456054688, weight_update_time: 0.3805676401, other_time: 27.4217529297
epoch time: 222.1866760254s, input_fired_time: 2.9400112629, output_fired_time: 0.0000000000, propagate_forward_time: 210.1803588867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.0662994385
validation[          , genome  6370] predictions:    6185/  10000 (61.85%), best:    6387/10000 (63.87%), error:     10892.11230, best error:     10551.79980 on epoch:    31, epoch:   32/50, mu: 0.9035059214, learning_rate: 0.0000356939, weight_decay: 0.0000494139

epoch time: 2747.3425292969s, input_fired_time: 29.0845127106, output_fired_time: 17.6118297577, propagate_forward_time: 1044.1519775391, propagate_backward_time: 1616.0621337891, weight_update_time: 2.3862147331, other_time: 40.4321289063
epoch time: 220.7837524414s, input_fired_time: 1.9387822151, output_fired_time: 0.0000000000, propagate_forward_time: 213.6551818848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1897888184
validation[          , genome  6370] predictions:    6306/  10000 (63.06%), best:    6387/10000 (63.87%), error:     10682.11328, best error:     10551.79980 on epoch:    31, epoch:   33/50, mu: 0.9081407785, learning_rate: 0.0000326214, weight_decay: 0.0000456401

epoch time: 2730.9682617188s, input_fired_time: 28.0937576294, output_fired_time: 17.7348613739, propagate_forward_time: 1058.3698730469, propagate_backward_time: 1588.3432617188, weight_update_time: 2.3931043148, other_time: 38.4265136719
epoch time: 221.9073333740s, input_fired_time: 1.9416071177, output_fired_time: 0.0000000000, propagate_forward_time: 216.7593383789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2063903809
validation[          , genome  6370] predictions:    6272/  10000 (62.72%), best:    6387/10000 (63.87%), error:     10764.71289, best error:     10551.79980 on epoch:    31, epoch:   34/50, mu: 0.9125272632, learning_rate: 0.0000298134, weight_decay: 0.0000421545

epoch time: 2782.3979492188s, input_fired_time: 31.0437011719, output_fired_time: 13.7471799850, propagate_forward_time: 1087.7934570313, propagate_backward_time: 1623.1950683594, weight_update_time: 0.3896980286, other_time: 26.6186523438
epoch time: 220.7541351318s, input_fired_time: 5.9548940659, output_fired_time: 0.0000000000, propagate_forward_time: 210.6809844971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1182556152
validation[          , genome  6370] predictions:    6338/  10000 (63.38%), best:    6387/10000 (63.87%), error:     10576.46289, best error:     10551.79980 on epoch:    31, epoch:   35/50, mu: 0.9166786671, learning_rate: 0.0000272471, weight_decay: 0.0000389352

epoch time: 2786.4062500000s, input_fired_time: 36.2478828430, output_fired_time: 18.8523120880, propagate_forward_time: 1093.8843994141, propagate_backward_time: 1610.1331787109, weight_update_time: 0.3836491108, other_time: 27.2885742188
epoch time: 248.5825347900s, input_fired_time: 5.0518832207, output_fired_time: 0.0000000000, propagate_forward_time: 237.2098693848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3207855225
validation[          , genome  6370] predictions:    6385/  10000 (63.85%), best:    6385/10000 (63.85%), error:     10506.85938, best error:     10506.85938 on epoch:    36, epoch:   36/50, mu: 0.9206076264, learning_rate: 0.0000249018, weight_decay: 0.0000359616

epoch time: 2914.5246582031s, input_fired_time: 42.5440406799, output_fired_time: 14.0395307541, propagate_forward_time: 1148.4326171875, propagate_backward_time: 1683.4649658203, weight_update_time: 0.3971505165, other_time: 26.0435791016
epoch time: 224.8735656738s, input_fired_time: 3.9644503593, output_fired_time: 0.0000000000, propagate_forward_time: 212.6830291748, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2260894775
validation[          , genome  6370] predictions:    6370/  10000 (63.70%), best:    6370/10000 (63.70%), error:     10476.96777, best error:     10476.96777 on epoch:    37, epoch:   37/50, mu: 0.9243260622, learning_rate: 0.0000227583, weight_decay: 0.0000332152

epoch time: 2722.5075683594s, input_fired_time: 38.1192169189, output_fired_time: 15.7047777176, propagate_forward_time: 1054.0865478516, propagate_backward_time: 1581.1676025391, weight_update_time: 3.3993179798, other_time: 33.4294433594
epoch time: 220.1634674072s, input_fired_time: 2.9510276318, output_fired_time: 0.0000000000, propagate_forward_time: 214.0503234863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1621093750
validation[          , genome  6370] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10390.01074, best error:     10390.01074 on epoch:    38, epoch:   38/50, mu: 0.9278452396, learning_rate: 0.0000207993, weight_decay: 0.0000306785

epoch time: 2680.2290039063s, input_fired_time: 27.0175876617, output_fired_time: 17.5342006683, propagate_forward_time: 1046.9844970703, propagate_backward_time: 1550.4741210938, weight_update_time: 2.3850255013, other_time: 38.2186279297
epoch time: 215.1682128906s, input_fired_time: 0.9124386311, output_fired_time: 0.0000000000, propagate_forward_time: 209.1426391602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1131286621
validation[          , genome  6370] predictions:    6403/  10000 (64.03%), best:    6388/10000 (63.88%), error:     10398.60156, best error:     10390.01074 on epoch:    38, epoch:   39/50, mu: 0.9311758280, learning_rate: 0.0000190089, weight_decay: 0.0000283356

epoch time: 2646.8298339844s, input_fired_time: 30.9030437469, output_fired_time: 11.4804439545, propagate_forward_time: 1046.7562255859, propagate_backward_time: 1528.7612304688, weight_update_time: 0.3664728701, other_time: 28.9288330078
epoch time: 215.1291961670s, input_fired_time: 2.9154543877, output_fired_time: 0.0000000000, propagate_forward_time: 207.1305847168, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0831604004
validation[          , genome  6370] predictions:    6424/  10000 (64.24%), best:    6388/10000 (63.88%), error:     10393.96094, best error:     10390.01074 on epoch:    38, epoch:   40/50, mu: 0.9343279600, learning_rate: 0.0000173727, weight_decay: 0.0000261716

epoch time: 2652.0463867188s, input_fired_time: 40.8242378235, output_fired_time: 14.4237623215, propagate_forward_time: 1036.8730468750, propagate_backward_time: 1536.8734130859, weight_update_time: 0.3670651317, other_time: 23.0518798828
epoch time: 215.2005157471s, input_fired_time: 2.9175381660, output_fired_time: 0.0000000000, propagate_forward_time: 207.1361236572, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1468505859
validation[          , genome  6370] predictions:    6410/  10000 (64.10%), best:    6388/10000 (63.88%), error:     10409.46289, best error:     10390.01074 on epoch:    38, epoch:   41/50, mu: 0.9373111725, learning_rate: 0.0000158773, weight_decay: 0.0000241728

epoch time: 2652.0144042969s, input_fired_time: 40.9132919312, output_fired_time: 11.4138755798, propagate_forward_time: 1031.9226074219, propagate_backward_time: 1533.8502197266, weight_update_time: 2.3720510006, other_time: 33.9144287109
epoch time: 217.8611907959s, input_fired_time: 1.9249292612, output_fired_time: 0.0000000000, propagate_forward_time: 211.8145599365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1217041016
validation[          , genome  6370] predictions:    6404/  10000 (64.04%), best:    6404/10000 (64.04%), error:     10353.64160, best error:     10353.64160 on epoch:    42, epoch:   42/50, mu: 0.9401345253, learning_rate: 0.0000145106, weight_decay: 0.0000223267

epoch time: 2652.4897460938s, input_fired_time: 33.9042625427, output_fired_time: 10.4005956650, propagate_forward_time: 1036.5941162109, propagate_backward_time: 1548.5568847656, weight_update_time: 0.3705902398, other_time: 23.0338134766
epoch time: 214.8051452637s, input_fired_time: 2.9194641113, output_fired_time: 0.0000000000, propagate_forward_time: 203.7850494385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1006317139
validation[          , genome  6370] predictions:    6446/  10000 (64.46%), best:    6446/10000 (64.46%), error:     10291.49805, best error:     10291.49805 on epoch:    43, epoch:   43/50, mu: 0.9428066015, learning_rate: 0.0000132615, weight_decay: 0.0000206216

epoch time: 2653.8500976563s, input_fired_time: 34.8666572571, output_fired_time: 14.5060825348, propagate_forward_time: 1018.5722656250, propagate_backward_time: 1543.7891845703, weight_update_time: 0.3672514260, other_time: 42.1158447266
epoch time: 213.9501190186s, input_fired_time: 4.9271354675, output_fired_time: 0.0000000000, propagate_forward_time: 204.9568939209, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0660858154
validation[          , genome  6370] predictions:    6394/  10000 (63.94%), best:    6446/10000 (64.46%), error:     10394.59863, best error:     10291.49805 on epoch:    43, epoch:   44/50, mu: 0.9453355074, learning_rate: 0.0000121200, weight_decay: 0.0000190467

epoch time: 2652.2187500000s, input_fired_time: 33.9493713379, output_fired_time: 9.4962759018, propagate_forward_time: 1049.5000000000, propagate_backward_time: 1535.3217773438, weight_update_time: 0.3724218309, other_time: 23.9511718750
epoch time: 211.0296325684s, input_fired_time: 1.9193408489, output_fired_time: 0.0000000000, propagate_forward_time: 204.9720306396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1382598877
validation[          , genome  6370] predictions:    6421/  10000 (64.21%), best:    6446/10000 (64.46%), error:     10296.09961, best error:     10291.49805 on epoch:    43, epoch:   45/50, mu: 0.9477288723, learning_rate: 0.0000110767, weight_decay: 0.0000175921

epoch time: 2654.6706542969s, input_fired_time: 29.8556919098, output_fired_time: 11.5236377716, propagate_forward_time: 1029.4915771484, propagate_backward_time: 1553.7198486328, weight_update_time: 2.3787198067, other_time: 30.0798339844
epoch time: 215.3548431396s, input_fired_time: 0.9209605455, output_fired_time: 0.0000000000, propagate_forward_time: 210.3002471924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1336364746
validation[          , genome  6370] predictions:    6464/  10000 (64.64%), best:    6464/10000 (64.64%), error:     10260.19531, best error:     10260.19531 on epoch:    46, epoch:   46/50, mu: 0.9499940276, learning_rate: 0.0000101233, weight_decay: 0.0000162486

epoch time: 2663.1284179688s, input_fired_time: 31.8920917511, output_fired_time: 11.5490751266, propagate_forward_time: 1036.4460449219, propagate_backward_time: 1550.2236328125, weight_update_time: 0.3647370934, other_time: 33.0175781250
epoch time: 216.5113677979s, input_fired_time: 2.9273145199, output_fired_time: 0.0000000000, propagate_forward_time: 208.4768218994, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1072387695
validation[          , genome  6370] predictions:    6417/  10000 (64.17%), best:    6464/10000 (64.64%), error:     10292.41113, best error:     10260.19531 on epoch:    46, epoch:   47/50, mu: 0.9521377683, learning_rate: 0.0000092519, weight_decay: 0.0000150077

epoch time: 2647.3364257813s, input_fired_time: 29.8777122498, output_fired_time: 8.4917192459, propagate_forward_time: 1033.3098144531, propagate_backward_time: 1543.4954833984, weight_update_time: 2.3846197128, other_time: 32.1617431641
epoch time: 215.5489349365s, input_fired_time: 0.9192357659, output_fired_time: 0.0000000000, propagate_forward_time: 210.5251312256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1045684814
validation[          , genome  6370] predictions:    6447/  10000 (64.47%), best:    6464/10000 (64.64%), error:     10365.42871, best error:     10260.19531 on epoch:    46, epoch:   48/50, mu: 0.9541666508, learning_rate: 0.0000084555, weight_decay: 0.0000138615

epoch time: 2660.5014648438s, input_fired_time: 34.0105552673, output_fired_time: 8.5083475113, propagate_forward_time: 1053.9206542969, propagate_backward_time: 1529.9624023438, weight_update_time: 0.3693230748, other_time: 34.0996093750
epoch time: 213.8167114258s, input_fired_time: 1.9218180180, output_fired_time: 0.0000000000, propagate_forward_time: 205.8159332275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0789642334
validation[          , genome  6370] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10180.34473, best error:     10180.34473 on epoch:    49, epoch:   49/50, mu: 0.9560868144, learning_rate: 0.0000077277, weight_decay: 0.0000128029

epoch time: 2645.1547851563s, input_fired_time: 32.8724670410, output_fired_time: 15.5211563110, propagate_forward_time: 1020.8491210938, propagate_backward_time: 1534.8211669922, weight_update_time: 2.3711357117, other_time: 41.0906982422
epoch time: 215.4154663086s, input_fired_time: 3.9367415905, output_fired_time: 0.0000000000, propagate_forward_time: 204.3499908447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1287384033
validation[          , genome  6370] predictions:    6502/  10000 (65.02%), best:    6485/10000 (64.85%), error:     10215.21875, best error:     10180.34473 on epoch:    49, epoch:   50/50, mu: 0.9579041004, learning_rate: 0.0000070625, weight_decay: 0.0000118251

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1075.6229248047s, input_fired_time: 13.6458301544, output_fired_time: 0.0000000000, propagate_forward_time: 1037.3061523438, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 24.6708984375
best training[          , genome  6370] predictions:   34765/  50000 (69.53%), best:    6485/10000 (64.85%), error:     46037.83594, best error:     10180.34473 on epoch:    49, epoch:   51/50, mu: 0.9596239924, learning_rate: 0.0000064546, weight_decay: 0.0000109220
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 215.0277252197s, input_fired_time: 1.9210044146, output_fired_time: 0.0000000000, propagate_forward_time: 208.7957763672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3109436035
   testing[          , genome  6370] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10180.34473, best error:     10180.34473 on epoch:    49, epoch:   51/50, mu: 0.9596239924, learning_rate: 0.0000064546, weight_decay: 0.0000109220
backpropagation finished successfully!
07:17:09 (15024): called boinc_finish(0)

</stderr_txt>
]]>