Nameexact_genome_1542031203_178_5669_2
Workunit4083342
Created24 Jan 2019, 15:07:41 UTC
Sent24 Jan 2019, 15:07:45 UTC
Report deadline7 Apr 2019, 5:37:01 UTC
Received28 Jan 2019, 14:25:08 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID96197
Run time3 days 11 hours 19 min 5 sec
CPU time2 days 15 hours 13 min 46 sec
Validate stateValid
Credit18,150.92
Device peak FLOPS3.03 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size557.86 MB
Peak swap size556.53 MB
Peak disk usage15.72 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669_2_r892497645_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.467552
read mu: 0.467552
read mu_delta: 0.963797
read initial_learning_rate: 0.000884987
read learning_rate: 0.000884987
read learning_rate_delta: 0.938435
read initial_weight_decay: 0.000383132
read weight_decay: 0.000383132
read weight_decay_delta: 0.927642
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.091529
read input_dropout_probability: 0.00104365
read hidden_dropout_probability: 0.0990035
read velocity_reset: 2444
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5669
read normal distribution: '0 0.863694846630096 -1.16751050949097'
generator_str: '1828302446'
read generator: 1828302446
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 589 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2083188646
post shuffle 1: 1597898051
epoch time: 17215.970703125s, input_fired_time: 172.976669311523, output_fired_time: 101.540046691895, propagate_forward_time: 6740.5380859375, propagate_backward_time: 10069.3984375, weight_update_time: 4.68857002258301, other_time: 131.5166015625
epoch time: 3009.41870117188s, input_fired_time: 37.2373046875, output_fired_time: 0, propagate_forward_time: 2914.82592773438, propagate_backward_time: 0, weight_update_time: 0, other_time: 57.35546875
validation[          , genome  5669] predictions:    5222/  10000 (52.22%), best:    5222/10000 (52.22%), error:     15761.10938, best error:     15761.10938 on epoch:     0, epoch:    0/50, mu: 0.4675524831, learning_rate: 0.0008849869, weight_decay: 0.0003831323

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669_2_r892497645_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.467552
read mu: 0.486467
read mu_delta: 0.963797
read initial_learning_rate: 0.000884987
read learning_rate: 0.000830503
read learning_rate_delta: 0.938435
read initial_weight_decay: 0.000383132
read weight_decay: 0.00035541
read weight_decay_delta: 0.927642
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.091529
read input_dropout_probability: 0.00104365
read hidden_dropout_probability: 0.0990035
read velocity_reset: 2444
read epoch: 1
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 5222
read best_validation_error: 15761.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5669
read normal distribution: '0 0.863694846630096 -1.16751050949097'
generator_str: '597726856'
read generator: 597726856
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 589 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 15926.8193359375s, input_fired_time: 149.497985839844, output_fired_time: 90.0043411254883, propagate_forward_time: 6346.201171875, propagate_backward_time: 9185.2705078125, weight_update_time: 3.55934619903564, other_time: 155.845703125
epoch time: 1200.98864746094s, input_fired_time: 9.36031723022461, output_fired_time: 0, propagate_forward_time: 1164.53576660156, propagate_backward_time: 0, weight_update_time: 0, other_time: 27.092529296875
validation[          , genome  5669] predictions:    5773/  10000 (57.73%), best:    5773/10000 (57.73%), error:     12171.25781, best error:     12171.25781 on epoch:     1, epoch:    1/50, mu: 0.4864666760, learning_rate: 0.0008305027, weight_decay: 0.0003554096

epoch time: 11066.6601562500s, input_fired_time: 117.2006912231, output_fired_time: 59.0063285828, propagate_forward_time: 4416.6718750000, propagate_backward_time: 6365.4604492188, weight_update_time: 3.8867275715, other_time: 108.3208007813
epoch time: 1774.5467529297s, input_fired_time: 18.1066722870, output_fired_time: 0.0000000000, propagate_forward_time: 1723.1949462891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 33.2451171875
validation[          , genome  5669] predictions:    5823/  10000 (58.23%), best:    5823/10000 (58.23%), error:     12100.38281, best error:     12100.38281 on epoch:     2, epoch:    2/50, mu: 0.5046961308, learning_rate: 0.0007793727, weight_decay: 0.0003296928

epoch time: 8399.5703125000s, input_fired_time: 70.2000808716, output_fired_time: 39.4469032288, propagate_forward_time: 3336.5334472656, propagate_backward_time: 4864.2841796875, weight_update_time: 2.4077274799, other_time: 89.1054687500
epoch time: 793.5977783203s, input_fired_time: 6.7961864471, output_fired_time: 0.0000000000, propagate_forward_time: 769.0607910156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.7407836914
validation[          , genome  5669] predictions:    5880/  10000 (58.80%), best:    5880/10000 (58.80%), error:     11951.30078, best error:     11951.30078 on epoch:     3, epoch:    3/50, mu: 0.5222656131, learning_rate: 0.0007313906, weight_decay: 0.0003058369

epoch time: 16371.6542968750s, input_fired_time: 154.6073608398, output_fired_time: 83.3980636597, propagate_forward_time: 6468.3476562500, propagate_backward_time: 9489.7294921875, weight_update_time: 5.7150812149, other_time: 175.5712890625
epoch time: 1216.1533203125s, input_fired_time: 10.3951005936, output_fired_time: 0.0000000000, propagate_forward_time: 1180.8551025391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 24.9030761719
validation[          , genome  5669] predictions:    5773/  10000 (57.73%), best:    5880/10000 (58.80%), error:     12449.18555, best error:     11951.30078 on epoch:     3, epoch:    4/50, mu: 0.5391989946, learning_rate: 0.0006863624, weight_decay: 0.0002837071

epoch time: 7079.4267578125s, input_fired_time: 67.0675277710, output_fired_time: 34.1212844849, propagate_forward_time: 2792.6074218750, propagate_backward_time: 4116.5063476563, weight_update_time: 2.0233695507, other_time: 69.1245117188
epoch time: 546.8583984375s, input_fired_time: 5.7258253098, output_fired_time: 0.0000000000, propagate_forward_time: 528.5302124023, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.6023559570
validation[          , genome  5669] predictions:    5861/  10000 (58.61%), best:    5880/10000 (58.80%), error:     12021.74609, best error:     11951.30078 on epoch:     3, epoch:    5/50, mu: 0.5555193424, learning_rate: 0.0006441065, weight_decay: 0.0002631786

epoch time: 6089.4975585938s, input_fired_time: 64.2974166870, output_fired_time: 31.1293582916, propagate_forward_time: 2391.2426757813, propagate_backward_time: 3542.0556640625, weight_update_time: 1.9698468447, other_time: 60.7724609375
epoch time: 486.1198425293s, input_fired_time: 4.3368725777, output_fired_time: 0.0000000000, propagate_forward_time: 460.5644531250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 21.2185058594
validation[          , genome  5669] predictions:    6017/  10000 (60.17%), best:    6017/10000 (60.17%), error:     11491.03906, best error:     11491.03906 on epoch:     6, epoch:    6/50, mu: 0.5712488294, learning_rate: 0.0006044520, weight_decay: 0.0002441355

epoch time: 7415.7255859375s, input_fired_time: 68.0772171021, output_fired_time: 41.0148124695, propagate_forward_time: 2928.1166992188, propagate_backward_time: 4288.1728515625, weight_update_time: 2.2067852020, other_time: 90.3442382813
epoch time: 630.0026855469s, input_fired_time: 5.2776989937, output_fired_time: 0.0000000000, propagate_forward_time: 612.1533813477, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.5715942383
validation[          , genome  5669] predictions:    6092/  10000 (60.92%), best:    6092/10000 (60.92%), error:     11244.06055, best error:     11244.06055 on epoch:     7, epoch:    7/50, mu: 0.5864089131, learning_rate: 0.0005672388, weight_decay: 0.0002264703

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669_2_r892497645_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.467552
read mu: 0.60102
read mu_delta: 0.963797
read initial_learning_rate: 0.000884987
read learning_rate: 0.000532317
read learning_rate_delta: 0.938435
read initial_weight_decay: 0.000383132
read weight_decay: 0.000210083
read weight_decay_delta: 0.927642
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.091529
read input_dropout_probability: 0.00104365
read hidden_dropout_probability: 0.0990035
read velocity_reset: 2444
read epoch: 8
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 7
read number_validation_images: 10000
read best_validation_predictions: 6092
read best_validation_error: 11244.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5669
read normal distribution: '0 0.863694846630096 -1.16751050949097'
generator_str: '1125185654'
read generator: 1125185654
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 589 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4130.6982421875s, input_fired_time: 43.2200164794922, output_fired_time: 23.1854343414307, propagate_forward_time: 1627.765625, propagate_backward_time: 2390.1748046875, weight_update_time: 1.22172284126282, other_time: 46.352294921875
epoch time: 354.087036132813s, input_fired_time: 4.21857643127441, output_fired_time: 0, propagate_forward_time: 335.660797119141, propagate_backward_time: 0, weight_update_time: 0, other_time: 14.2076721191406
validation[          , genome  5669] predictions:    5963/  10000 (59.63%), best:    6092/10000 (60.92%), error:     11725.69922, best error:     11244.06055 on epoch:     7, epoch:    8/50, mu: 0.6010201573, learning_rate: 0.0005323167, weight_decay: 0.0002100833

epoch time: 4151.3398437500s, input_fired_time: 37.6913185120, output_fired_time: 28.4390392303, propagate_forward_time: 1607.6811523438, propagate_backward_time: 2434.1242675781, weight_update_time: 1.3877983093, other_time: 43.4040527344
epoch time: 318.6359252930s, input_fired_time: 2.8850696087, output_fired_time: 0.0000000000, propagate_forward_time: 308.2542419434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4966125488
validation[          , genome  5669] predictions:    5956/  10000 (59.56%), best:    6092/10000 (60.92%), error:     11762.21484, best error:     11244.06055 on epoch:     7, epoch:    9/50, mu: 0.6151024103, learning_rate: 0.0004995446, weight_decay: 0.0001948821

epoch time: 3805.3574218750s, input_fired_time: 35.4834098816, output_fired_time: 20.5511360168, propagate_forward_time: 1499.8662109375, propagate_backward_time: 2209.3776855469, weight_update_time: 1.3534168005, other_time: 40.0791015625
epoch time: 324.6361083984s, input_fired_time: 2.9029958248, output_fired_time: 0.0000000000, propagate_forward_time: 314.1096801758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.6234436035
validation[          , genome  5669] predictions:    6216/  10000 (62.16%), best:    6216/10000 (62.16%), error:     11018.10254, best error:     11018.10254 on epoch:    10, epoch:   10/50, mu: 0.6286748648, learning_rate: 0.0004687901, weight_decay: 0.0001807808

epoch time: 3647.1428222656s, input_fired_time: 36.8259582520, output_fired_time: 19.8408832550, propagate_forward_time: 1410.5178222656, propagate_backward_time: 2140.4479980469, weight_update_time: 1.1982495785, other_time: 39.5102539063
epoch time: 418.0205993652s, input_fired_time: 4.0068545341, output_fired_time: 0.0000000000, propagate_forward_time: 404.1345214844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.8792114258
validation[          , genome  5669] predictions:    6156/  10000 (61.56%), best:    6216/10000 (62.16%), error:     11188.24121, best error:     11018.10254 on epoch:    10, epoch:   11/50, mu: 0.6417559385, learning_rate: 0.0004399290, weight_decay: 0.0001676998

epoch time: 3874.9987792969s, input_fired_time: 36.6498298645, output_fired_time: 21.3492374420, propagate_forward_time: 1533.8559570313, propagate_backward_time: 2238.9414062500, weight_update_time: 1.3452827930, other_time: 44.2023925781
epoch time: 297.8036193848s, input_fired_time: 2.9286377430, output_fired_time: 0.0000000000, propagate_forward_time: 287.5099182129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3650512695
validation[          , genome  5669] predictions:    6045/  10000 (60.45%), best:    6216/10000 (62.16%), error:     11386.93457, best error:     11018.10254 on epoch:    10, epoch:   12/50, mu: 0.6543634534, learning_rate: 0.0004128447, weight_decay: 0.0001555654

epoch time: 3663.0297851563s, input_fired_time: 35.7021522522, output_fired_time: 20.3183631897, propagate_forward_time: 1437.2430419922, propagate_backward_time: 2129.6359863281, weight_update_time: 1.3858354092, other_time: 40.1301269531
epoch time: 298.2994079590s, input_fired_time: 2.9429299831, output_fired_time: 0.0000000000, propagate_forward_time: 287.7565307617, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5999450684
validation[          , genome  5669] predictions:    6218/  10000 (62.18%), best:    6218/10000 (62.18%), error:     11008.47754, best error:     11008.47754 on epoch:    13, epoch:   13/50, mu: 0.6665145159, learning_rate: 0.0003874279, weight_decay: 0.0001443089

epoch time: 4010.2236328125s, input_fired_time: 38.8788108826, output_fired_time: 23.3342876434, propagate_forward_time: 1587.3785400391, propagate_backward_time: 2316.8217773438, weight_update_time: 1.5011593103, other_time: 43.8100585938
epoch time: 359.5688476563s, input_fired_time: 3.4633948803, output_fired_time: 0.0000000000, propagate_forward_time: 347.7572631836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3481750488
validation[          , genome  5669] predictions:    6365/  10000 (63.65%), best:    6365/10000 (63.65%), error:     10634.20996, best error:     10634.20996 on epoch:    14, epoch:   14/50, mu: 0.6782256961, learning_rate: 0.0003635758, weight_decay: 0.0001338670

epoch time: 4447.4707031250s, input_fired_time: 43.0750656128, output_fired_time: 24.8770217896, propagate_forward_time: 1754.0300292969, propagate_backward_time: 2577.1027832031, weight_update_time: 1.4869732857, other_time: 48.3857421875
epoch time: 381.9531860352s, input_fired_time: 3.5434923172, output_fired_time: 0.0000000000, propagate_forward_time: 367.4809265137, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.9287719727
validation[          , genome  5669] predictions:    6287/  10000 (62.87%), best:    6365/10000 (63.65%), error:     10755.51855, best error:     10634.20996 on epoch:    14, epoch:   15/50, mu: 0.6895128489, learning_rate: 0.0003411923, weight_decay: 0.0001241806

epoch time: 4308.9077148438s, input_fired_time: 42.6641273499, output_fired_time: 24.1902294159, propagate_forward_time: 1690.1424560547, propagate_backward_time: 2504.4758300781, weight_update_time: 1.6166918278, other_time: 47.4348144531
epoch time: 315.5880126953s, input_fired_time: 3.2679972649, output_fired_time: 0.0000000000, propagate_forward_time: 304.7249450684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5950622559
validation[          , genome  5669] predictions:    6204/  10000 (62.04%), best:    6365/10000 (63.65%), error:     10960.90527, best error:     10634.20996 on epoch:    14, epoch:   16/50, mu: 0.7003914118, learning_rate: 0.0003201867, weight_decay: 0.0001151952

epoch time: 3879.5380859375s, input_fired_time: 37.7097473145, output_fired_time: 22.0612831116, propagate_forward_time: 1527.8994140625, propagate_backward_time: 2248.5708007813, weight_update_time: 1.3159140348, other_time: 43.2968750000
epoch time: 320.4644775391s, input_fired_time: 3.1493799686, output_fired_time: 0.0000000000, propagate_forward_time: 309.2844238281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.0306701660
validation[          , genome  5669] predictions:    6284/  10000 (62.84%), best:    6365/10000 (63.65%), error:     10839.54883, best error:     10634.20996 on epoch:    14, epoch:   17/50, mu: 0.7108761072, learning_rate: 0.0003004744, weight_decay: 0.0001068599

epoch time: 3825.8686523438s, input_fired_time: 37.4481048584, output_fired_time: 20.9534530640, propagate_forward_time: 1510.0244140625, propagate_backward_time: 2216.2495117188, weight_update_time: 1.3078777790, other_time: 41.1933593750
epoch time: 304.9510192871s, input_fired_time: 3.0199847221, output_fired_time: 0.0000000000, propagate_forward_time: 295.1478881836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7831420898
validation[          , genome  5669] predictions:    6231/  10000 (62.31%), best:    6365/10000 (63.65%), error:     10849.23242, best error:     10634.20996 on epoch:    14, epoch:   18/50, mu: 0.7209812403, learning_rate: 0.0002819757, weight_decay: 0.0000991277

epoch time: 3803.3603515625s, input_fired_time: 36.8421821594, output_fired_time: 21.4110221863, propagate_forward_time: 1506.3007812500, propagate_backward_time: 2198.5280761719, weight_update_time: 1.2648148537, other_time: 40.2780761719
epoch time: 306.7979736328s, input_fired_time: 3.1073603630, output_fired_time: 0.0000000000, propagate_forward_time: 296.4821472168, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2084655762
validation[          , genome  5669] predictions:    6447/  10000 (64.47%), best:    6447/10000 (64.47%), error:     10499.66406, best error:     10499.66406 on epoch:    19, epoch:   19/50, mu: 0.7307205200, learning_rate: 0.0002646158, weight_decay: 0.0000919550

epoch time: 3790.9443359375s, input_fired_time: 36.9021377563, output_fired_time: 21.1630840302, propagate_forward_time: 1491.7471923828, propagate_backward_time: 2200.4353027344, weight_update_time: 1.3364738226, other_time: 40.6965332031
epoch time: 312.5532226563s, input_fired_time: 3.0574309826, output_fired_time: 0.0000000000, propagate_forward_time: 302.6794128418, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8163757324
validation[          , genome  5669] predictions:    6160/  10000 (61.60%), best:    6447/10000 (64.47%), error:     11044.64160, best error:     10499.66406 on epoch:    19, epoch:   20/50, mu: 0.7401072383, learning_rate: 0.0002483247, weight_decay: 0.0000853013

epoch time: 4260.8193359375s, input_fired_time: 43.4461364746, output_fired_time: 23.8441066742, propagate_forward_time: 1678.7595214844, propagate_backward_time: 2470.0029296875, weight_update_time: 1.4020260572, other_time: 44.7663574219
epoch time: 376.4173889160s, input_fired_time: 3.7884247303, output_fired_time: 0.0000000000, propagate_forward_time: 362.2997741699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.3291931152
validation[          , genome  5669] predictions:    6431/  10000 (64.31%), best:    6431/10000 (64.31%), error:     10328.31641, best error:     10328.31641 on epoch:    21, epoch:   21/50, mu: 0.7491540909, learning_rate: 0.0002330366, weight_decay: 0.0000791291

epoch time: 4496.8115234375s, input_fired_time: 43.8652343750, output_fired_time: 25.4085407257, propagate_forward_time: 1777.6960449219, propagate_backward_time: 2601.5395507813, weight_update_time: 1.6892286539, other_time: 48.3020019531
epoch time: 362.7984008789s, input_fired_time: 3.5755598545, output_fired_time: 0.0000000000, propagate_forward_time: 350.6230163574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5998229980
validation[          , genome  5669] predictions:    6331/  10000 (63.31%), best:    6431/10000 (64.31%), error:     10668.77539, best error:     10328.31641 on epoch:    21, epoch:   22/50, mu: 0.7578734159, learning_rate: 0.0002186896, weight_decay: 0.0000734034

epoch time: 4450.9194335938s, input_fired_time: 43.0230293274, output_fired_time: 25.6482677460, propagate_forward_time: 1762.6589355469, propagate_backward_time: 2569.5935058594, weight_update_time: 1.4929596186, other_time: 49.9956054688
epoch time: 367.9479064941s, input_fired_time: 3.5365526676, output_fired_time: 0.0000000000, propagate_forward_time: 355.3050231934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.1063232422
validation[          , genome  5669] predictions:    6421/  10000 (64.21%), best:    6431/10000 (64.31%), error:     10427.57715, best error:     10328.31641 on epoch:    21, epoch:   23/50, mu: 0.7662770748, learning_rate: 0.0002052260, weight_decay: 0.0000680921

epoch time: 4435.5112304688s, input_fired_time: 43.0753059387, output_fired_time: 25.0760631561, propagate_forward_time: 1752.3927001953, propagate_backward_time: 2565.0773925781, weight_update_time: 1.6106780767, other_time: 49.8898925781
epoch time: 366.0046997070s, input_fired_time: 3.5546743870, output_fired_time: 0.0000000000, propagate_forward_time: 353.6801452637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.7698669434
validation[          , genome  5669] predictions:    6441/  10000 (64.41%), best:    6431/10000 (64.31%), error:     10389.07910, best error:     10328.31641 on epoch:    21, epoch:   24/50, mu: 0.7743765116, learning_rate: 0.0001925912, weight_decay: 0.0000631651

epoch time: 4441.4291992188s, input_fired_time: 44.2917671204, output_fired_time: 25.0473346710, propagate_forward_time: 1755.5708007813, propagate_backward_time: 2569.9060058594, weight_update_time: 1.5395936966, other_time: 46.6130371094
epoch time: 361.9552612305s, input_fired_time: 3.5936300755, output_fired_time: 0.0000000000, propagate_forward_time: 350.1360168457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2256164551
validation[          , genome  5669] predictions:    6439/  10000 (64.39%), best:    6439/10000 (64.39%), error:     10295.35352, best error:     10295.35352 on epoch:    25, epoch:   25/50, mu: 0.7821827531, learning_rate: 0.0001807343, weight_decay: 0.0000585946

epoch time: 4389.8125000000s, input_fired_time: 42.8358421326, output_fired_time: 25.9482402802, propagate_forward_time: 1728.2363281250, propagate_backward_time: 2546.6647949219, weight_update_time: 1.4292434454, other_time: 46.1271972656
epoch time: 361.3060302734s, input_fired_time: 3.5964975357, output_fired_time: 0.0000000000, propagate_forward_time: 349.2998352051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4096984863
validation[          , genome  5669] predictions:    6525/  10000 (65.25%), best:    6525/10000 (65.25%), error:     10201.62598, best error:     10201.62598 on epoch:    26, epoch:   26/50, mu: 0.7897063494, learning_rate: 0.0001696074, weight_decay: 0.0000543548

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669_2_r892497645_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.467552
read mu: 0.796958
read mu_delta: 0.963797
read initial_learning_rate: 0.000884987
read learning_rate: 0.000159166
read learning_rate_delta: 0.938435
read initial_weight_decay: 0.000383132
read weight_decay: 5.04218e-05
read weight_decay_delta: 0.927642
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.091529
read input_dropout_probability: 0.00104365
read hidden_dropout_probability: 0.0990035
read velocity_reset: 2444
read epoch: 27
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 26
read number_validation_images: 10000
read best_validation_predictions: 6525
read best_validation_error: 10201.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5669
read normal distribution: '0 0.863694846630096 -1.16751050949097'
generator_str: '693215658'
read generator: 693215658
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 589 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4168.6279296875s, input_fired_time: 40.9574890136719, output_fired_time: 23.1952171325684, propagate_forward_time: 1651.15576171875, propagate_backward_time: 2408.861328125, weight_update_time: 1.42148804664612, other_time: 44.4580078125
epoch time: 343.626342773438s, input_fired_time: 3.50208377838135, output_fired_time: 0, propagate_forward_time: 331.910217285156, propagate_backward_time: 0, weight_update_time: 0, other_time: 8.21405029296875
validation[          , genome  5669] predictions:    6383/  10000 (63.83%), best:    6525/10000 (65.25%), error:     10378.87598, best error:     10201.62598 on epoch:    26, epoch:   27/50, mu: 0.7969576120, learning_rate: 0.0001591655, weight_decay: 0.0000504218

epoch time: 4176.9853515625s, input_fired_time: 40.7157363892, output_fired_time: 23.3989944458, propagate_forward_time: 1651.9155273438, propagate_backward_time: 2416.5991210938, weight_update_time: 1.4153977633, other_time: 44.3559570313
epoch time: 343.9319763184s, input_fired_time: 3.4113662243, output_fired_time: 0.0000000000, propagate_forward_time: 332.2721862793, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2484130859
validation[          , genome  5669] predictions:    6401/  10000 (64.01%), best:    6525/10000 (65.25%), error:     10596.74219, best error:     10201.62598 on epoch:    26, epoch:   28/50, mu: 0.8039463162, learning_rate: 0.0001493665, weight_decay: 0.0000467733

epoch time: 4166.8383789063s, input_fired_time: 41.1372337341, output_fired_time: 23.3577766418, propagate_forward_time: 1645.3718261719, propagate_backward_time: 2412.4787597656, weight_update_time: 1.6240959167, other_time: 44.4926757813
epoch time: 342.9732666016s, input_fired_time: 3.4517314434, output_fired_time: 0.0000000000, propagate_forward_time: 331.8476257324, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.6739196777
validation[          , genome  5669] predictions:    6474/  10000 (64.74%), best:    6525/10000 (65.25%), error:     10368.76270, best error:     10201.62598 on epoch:    26, epoch:   29/50, mu: 0.8106819987, learning_rate: 0.0001401707, weight_decay: 0.0000433889

epoch time: 4150.1127929688s, input_fired_time: 40.6402969360, output_fired_time: 23.1764373779, propagate_forward_time: 1637.4777832031, propagate_backward_time: 2404.0129394531, weight_update_time: 1.4331519604, other_time: 44.8054199219
epoch time: 339.0971984863s, input_fired_time: 3.4584167004, output_fired_time: 0.0000000000, propagate_forward_time: 327.7483215332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8904724121
validation[          , genome  5669] predictions:    6406/  10000 (64.06%), best:    6525/10000 (65.25%), error:     10501.96387, best error:     10201.62598 on epoch:    26, epoch:   30/50, mu: 0.8171738386, learning_rate: 0.0001315411, weight_decay: 0.0000402494

epoch time: 4153.9448242188s, input_fired_time: 40.7919425964, output_fired_time: 23.2124195099, propagate_forward_time: 1642.4934082031, propagate_backward_time: 2402.7551269531, weight_update_time: 1.4422198534, other_time: 44.6918945313
epoch time: 340.0072021484s, input_fired_time: 3.3659315109, output_fired_time: 0.0000000000, propagate_forward_time: 328.7569580078, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8843078613
validation[          , genome  5669] predictions:    6497/  10000 (64.97%), best:    6525/10000 (65.25%), error:     10241.68457, best error:     10201.62598 on epoch:    26, epoch:   31/50, mu: 0.8234306574, learning_rate: 0.0001234427, weight_decay: 0.0000373370

epoch time: 4397.3955078125s, input_fired_time: 43.6873474121, output_fired_time: 24.8662338257, propagate_forward_time: 1743.0447998047, propagate_backward_time: 2539.5002441406, weight_update_time: 1.5753828287, other_time: 46.2966308594
epoch time: 370.8008728027s, input_fired_time: 3.7082822323, output_fired_time: 0.0000000000, propagate_forward_time: 358.5791320801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5134582520
validation[          , genome  5669] predictions:    6589/  10000 (65.89%), best:    6589/10000 (65.89%), error:     10011.77930, best error:     10011.77930 on epoch:    32, epoch:   32/50, mu: 0.8294609785, learning_rate: 0.0001158430, weight_decay: 0.0000346354

epoch time: 4470.7680664063s, input_fired_time: 43.7258262634, output_fired_time: 28.6113491058, propagate_forward_time: 1765.8703613281, propagate_backward_time: 2586.9521484375, weight_update_time: 1.6170201302, other_time: 45.6086425781
epoch time: 371.8356933594s, input_fired_time: 3.8746333122, output_fired_time: 0.0000000000, propagate_forward_time: 359.2306518555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.7304077148
validation[          , genome  5669] predictions:    6519/  10000 (65.19%), best:    6589/10000 (65.89%), error:     10140.26367, best error:     10011.77930 on epoch:    32, epoch:   33/50, mu: 0.8352729678, learning_rate: 0.0001087111, weight_decay: 0.0000321292

epoch time: 4468.0756835938s, input_fired_time: 42.8443984985, output_fired_time: 24.5694427490, propagate_forward_time: 1765.0000000000, propagate_backward_time: 2590.4941406250, weight_update_time: 1.5142045021, other_time: 45.1679687500
epoch time: 366.7167663574s, input_fired_time: 3.5667917728, output_fired_time: 0.0000000000, propagate_forward_time: 354.7852172852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3647460938
validation[          , genome  5669] predictions:    6523/  10000 (65.23%), best:    6589/10000 (65.89%), error:     10167.99316, best error:     10011.77930 on epoch:    32, epoch:   34/50, mu: 0.8408745527, learning_rate: 0.0001020183, weight_decay: 0.0000298044

epoch time: 4463.5693359375s, input_fired_time: 43.2651405334, output_fired_time: 25.0420246124, propagate_forward_time: 1774.0705566406, propagate_backward_time: 2574.9907226563, weight_update_time: 1.5263221264, other_time: 46.2009277344
epoch time: 359.3238830566s, input_fired_time: 3.5429751873, output_fired_time: 0.0000000000, propagate_forward_time: 347.6106872559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1702270508
validation[          , genome  5669] predictions:    6541/  10000 (65.41%), best:    6589/10000 (65.89%), error:     10049.99512, best error:     10011.77930 on epoch:    32, epoch:   35/50, mu: 0.8462733626, learning_rate: 0.0000957375, weight_decay: 0.0000276478

epoch time: 4443.1005859375s, input_fired_time: 43.4573707581, output_fired_time: 24.9291934967, propagate_forward_time: 1759.8344726563, propagate_backward_time: 2569.9221191406, weight_update_time: 1.5273722410, other_time: 44.9572753906
epoch time: 368.8713378906s, input_fired_time: 3.7824921608, output_fired_time: 0.0000000000, propagate_forward_time: 356.7136535645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3751831055
validation[          , genome  5669] predictions:    6541/  10000 (65.41%), best:    6589/10000 (65.89%), error:     10096.77930, best error:     10011.77930 on epoch:    32, epoch:   36/50, mu: 0.8514767289, learning_rate: 0.0000898434, weight_decay: 0.0000256473

epoch time: 4485.0278320313s, input_fired_time: 44.3153038025, output_fired_time: 24.8222846985, propagate_forward_time: 1775.9777832031, propagate_backward_time: 2593.3298339844, weight_update_time: 1.5780082941, other_time: 46.5825195313
epoch time: 367.6761169434s, input_fired_time: 3.6458163261, output_fired_time: 0.0000000000, propagate_forward_time: 355.7298583984, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3004455566
validation[          , genome  5669] predictions:    6692/  10000 (66.92%), best:    6692/10000 (66.92%), error:      9790.62402, best error:      9790.62402 on epoch:    37, epoch:   37/50, mu: 0.8564916849, learning_rate: 0.0000843122, weight_decay: 0.0000237915

epoch time: 4456.0922851563s, input_fired_time: 43.4583053589, output_fired_time: 24.6906757355, propagate_forward_time: 1766.0277099609, propagate_backward_time: 2576.0917968750, weight_update_time: 1.6617203951, other_time: 45.8232421875
epoch time: 363.1109008789s, input_fired_time: 3.7458024025, output_fired_time: 0.0000000000, propagate_forward_time: 350.8496398926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5154724121
validation[          , genome  5669] predictions:    6578/  10000 (65.78%), best:    6692/10000 (66.92%), error:      9947.03906, best error:      9790.62402 on epoch:    37, epoch:   38/50, mu: 0.8613250852, learning_rate: 0.0000791215, weight_decay: 0.0000220700

epoch time: 4485.9526367188s, input_fired_time: 44.4881935120, output_fired_time: 25.0164165497, propagate_forward_time: 1781.6842041016, propagate_backward_time: 2587.8305664063, weight_update_time: 1.5593597889, other_time: 46.9331054688
epoch time: 370.4318847656s, input_fired_time: 3.7652790546, output_fired_time: 0.0000000000, propagate_forward_time: 358.2717285156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3948669434
validation[          , genome  5669] predictions:    6668/  10000 (66.68%), best:    6668/10000 (66.68%), error:      9770.53711, best error:      9770.53711 on epoch:    39, epoch:   39/50, mu: 0.8659834862, learning_rate: 0.0000742504, weight_decay: 0.0000204730

epoch time: 4421.6630859375s, input_fired_time: 43.1826972961, output_fired_time: 25.1472949982, propagate_forward_time: 1746.3807373047, propagate_backward_time: 2562.2534179688, weight_update_time: 1.3801413774, other_time: 44.6987304688
epoch time: 361.5793151855s, input_fired_time: 3.5534508228, output_fired_time: 0.0000000000, propagate_forward_time: 349.3399047852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.6859741211
validation[          , genome  5669] predictions:    6679/  10000 (66.79%), best:    6668/10000 (66.68%), error:      9783.93555, best error:      9770.53711 on epoch:    39, epoch:   40/50, mu: 0.8704732656, learning_rate: 0.0000696792, weight_decay: 0.0000189916

epoch time: 4385.9721679688s, input_fired_time: 43.1213417053, output_fired_time: 24.0703430176, propagate_forward_time: 1734.3592529297, propagate_backward_time: 2539.5100097656, weight_update_time: 1.4525907040, other_time: 44.9108886719
epoch time: 354.8624267578s, input_fired_time: 3.5424089432, output_fired_time: 0.0000000000, propagate_forward_time: 343.0718688965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2481384277
validation[          , genome  5669] predictions:    6598/  10000 (65.98%), best:    6668/10000 (66.68%), error:      9936.15430, best error:      9770.53711 on epoch:    39, epoch:   41/50, mu: 0.8748005033, learning_rate: 0.0000653894, weight_decay: 0.0000176174

epoch time: 4379.4736328125s, input_fired_time: 43.5996513367, output_fired_time: 24.6508121490, propagate_forward_time: 1734.1406250000, propagate_backward_time: 2531.0637207031, weight_update_time: 1.6421562433, other_time: 46.0187988281
epoch time: 358.8861083984s, input_fired_time: 3.5718336105, output_fired_time: 0.0000000000, propagate_forward_time: 346.8719787598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4422912598
validation[          , genome  5669] predictions:    6647/  10000 (66.47%), best:    6668/10000 (66.68%), error:      9825.84863, best error:      9770.53711 on epoch:    39, epoch:   42/50, mu: 0.8789710999, learning_rate: 0.0000613637, weight_decay: 0.0000163427

epoch time: 4389.8330078125s, input_fired_time: 43.1154861450, output_fired_time: 24.2758274078, propagate_forward_time: 1736.3083496094, propagate_backward_time: 2540.7719726563, weight_update_time: 1.4845809937, other_time: 45.3610839844
epoch time: 355.9580383301s, input_fired_time: 3.5417690277, output_fired_time: 0.0000000000, propagate_forward_time: 344.2011108398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2151489258
validation[          , genome  5669] predictions:    6626/  10000 (66.26%), best:    6668/10000 (66.68%), error:      9905.89746, best error:      9770.53711 on epoch:    39, epoch:   43/50, mu: 0.8829906583, learning_rate: 0.0000575858, weight_decay: 0.0000151601

epoch time: 4385.6328125000s, input_fired_time: 43.4693069458, output_fired_time: 24.3969230652, propagate_forward_time: 1731.4613037109, propagate_backward_time: 2540.5927734375, weight_update_time: 1.5015149117, other_time: 45.7124023438
epoch time: 362.5892639160s, input_fired_time: 3.6687750816, output_fired_time: 0.0000000000, propagate_forward_time: 349.6144409180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3060607910
validation[          , genome  5669] predictions:    6598/  10000 (65.98%), best:    6668/10000 (66.68%), error:      9951.24707, best error:      9770.53711 on epoch:    39, epoch:   44/50, mu: 0.8868647218, learning_rate: 0.0000540405, weight_decay: 0.0000140632

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_178_5669_2_r892497645_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.467552
read mu: 0.890599
read mu_delta: 0.963797
read initial_learning_rate: 0.000884987
read learning_rate: 5.07135e-05
read learning_rate_delta: 0.938435
read initial_weight_decay: 0.000383132
read weight_decay: 1.30456e-05
read weight_decay_delta: 0.927642
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.091529
read input_dropout_probability: 0.00104365
read hidden_dropout_probability: 0.0990035
read velocity_reset: 2444
read epoch: 45
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 39
read number_validation_images: 10000
read best_validation_predictions: 6668
read best_validation_error: 9770.54
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5669
read normal distribution: '0 0.863694846630096 -1.16751050949097'
generator_str: '2042998847'
read generator: 2042998847
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 589 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4406.17822265625s, input_fired_time: 43.1843490600586, output_fired_time: 24.2191886901855, propagate_forward_time: 1739.72595214844, propagate_backward_time: 2553.53100585938, weight_update_time: 1.51999580860138, other_time: 45.517333984375
epoch time: 354.889465332031s, input_fired_time: 3.53546929359436, output_fired_time: 0, propagate_forward_time: 343.107421875, propagate_backward_time: 0, weight_update_time: 0, other_time: 8.24658203125
validation[          , genome  5669] predictions:    6565/  10000 (65.65%), best:    6668/10000 (66.68%), error:      9877.85938, best error:      9770.53711 on epoch:    39, epoch:   45/50, mu: 0.8905985355, learning_rate: 0.0000507135, weight_decay: 0.0000130456

epoch time: 4234.6391601563s, input_fired_time: 41.9266510010, output_fired_time: 23.8780441284, propagate_forward_time: 1672.6156005859, propagate_backward_time: 2451.5051269531, weight_update_time: 1.4521740675, other_time: 44.7136230469
epoch time: 365.3892822266s, input_fired_time: 3.9443931580, output_fired_time: 0.0000000000, propagate_forward_time: 352.1925354004, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2523498535
validation[          , genome  5669] predictions:    6634/  10000 (66.34%), best:    6668/10000 (66.68%), error:      9773.14941, best error:      9770.53711 on epoch:    39, epoch:   46/50, mu: 0.8941971660, learning_rate: 0.0000475913, weight_decay: 0.0000121016

epoch time: 4138.1010742188s, input_fired_time: 40.6349906921, output_fired_time: 23.2647838593, propagate_forward_time: 1627.9239501953, propagate_backward_time: 2402.9033203125, weight_update_time: 1.4706023932, other_time: 43.3740234375
epoch time: 343.4657897949s, input_fired_time: 3.5043818951, output_fired_time: 0.0000000000, propagate_forward_time: 332.0783081055, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8830871582
validation[          , genome  5669] predictions:    6652/  10000 (66.52%), best:    6652/10000 (66.52%), error:      9730.10449, best error:      9730.10449 on epoch:    47, epoch:   47/50, mu: 0.8976655006, learning_rate: 0.0000446614, weight_decay: 0.0000112260

epoch time: 4165.6000976563s, input_fired_time: 41.5464897156, output_fired_time: 23.4768695831, propagate_forward_time: 1636.1187744141, propagate_backward_time: 2419.8444824219, weight_update_time: 1.4445089102, other_time: 44.6135253906
epoch time: 343.8731689453s, input_fired_time: 3.4301826954, output_fired_time: 0.0000000000, propagate_forward_time: 332.3150024414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1279907227
validation[          , genome  5669] predictions:    6684/  10000 (66.84%), best:    6684/10000 (66.84%), error:      9709.47461, best error:      9709.47461 on epoch:    48, epoch:   48/50, mu: 0.9010083079, learning_rate: 0.0000419118, weight_decay: 0.0000104137

epoch time: 4172.8583984375s, input_fired_time: 41.0916633606, output_fired_time: 23.2400321960, propagate_forward_time: 1637.9283447266, propagate_backward_time: 2425.2402343750, weight_update_time: 1.4728121758, other_time: 45.3579101563
epoch time: 339.1523742676s, input_fired_time: 3.4065768719, output_fired_time: 0.0000000000, propagate_forward_time: 326.9196777344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.8261108398
validation[          , genome  5669] predictions:    6664/  10000 (66.64%), best:    6664/10000 (66.64%), error:      9699.33398, best error:      9699.33398 on epoch:    49, epoch:   49/50, mu: 0.9042300582, learning_rate: 0.0000393315, weight_decay: 0.0000096602

epoch time: 4148.2963867188s, input_fired_time: 41.2444572449, output_fired_time: 23.1186866760, propagate_forward_time: 1622.5809326172, propagate_backward_time: 2415.0346679688, weight_update_time: 1.4081867933, other_time: 46.3173828125
epoch time: 336.8298950195s, input_fired_time: 3.6133444309, output_fired_time: 0.0000000000, propagate_forward_time: 325.3853149414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8312377930
validation[          , genome  5669] predictions:    6708/  10000 (67.08%), best:    6708/10000 (67.08%), error:      9674.87305, best error:      9674.87305 on epoch:    50, epoch:   50/50, mu: 0.9073351622, learning_rate: 0.0000369100, weight_decay: 0.0000089612

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1685.2180175781s, input_fired_time: 17.7724895477, output_fired_time: 0.0000000000, propagate_forward_time: 1625.5772705078, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 41.8682861328
best training[          , genome  5669] predictions:   37357/  50000 (74.71%), best:    6708/10000 (67.08%), error:     38445.65625, best error:      9674.87305 on epoch:    50, epoch:   51/50, mu: 0.9103278518, learning_rate: 0.0000346377, weight_decay: 0.0000083128
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 338.8752441406s, input_fired_time: 3.5311369896, output_fired_time: 0.0000000000, propagate_forward_time: 327.1964111328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1477050781
   testing[          , genome  5669] predictions:    6708/  10000 (67.08%), best:    6708/10000 (67.08%), error:      9674.87305, best error:      9674.87305 on epoch:    50, epoch:   51/50, mu: 0.9103278518, learning_rate: 0.0000346377, weight_decay: 0.0000083128
backpropagation finished successfully!
08:23:46 (1660): called boinc_finish(0)

</stderr_txt>
]]>