Nameexact_genome_1542152103_177_5834_1
Workunit4087341
Created14 Nov 2018, 0:19:41 UTC
Sent14 Nov 2018, 0:30:11 UTC
Report deadline30 Dec 2018, 0:44:55 UTC
Received15 Nov 2018, 21:24:19 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID73073
Run time13 hours 47 min 59 sec
CPU time10 hours 49 min 35 sec
Validate stateValid
Credit11,502.56
Device peak FLOPS4.57 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size527.38 MB
Peak swap size547.20 MB
Peak disk usage11.04 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834_1_r1576638797_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.5
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0125
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.0005
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5834
read normal distribution: '0 -0.293411165475845 0.638238370418549'
generator_str: '684147702'
read generator: 684147702
read generated_by_map:
1 change_size 1
reading 87 nodes.
reading 339 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 842981476
post shuffle 1: 801321644
epoch time: 1005.16363525391s, input_fired_time: 9.40628147125244, output_fired_time: 8.48221397399902, propagate_forward_time: 370.17138671875, propagate_backward_time: 603.873779296875, weight_update_time: 1.2181636095047, other_time: 13.2299194335938
epoch time: 75.7871017456055s, input_fired_time: 0.953811168670654, output_fired_time: 0, propagate_forward_time: 71.6951217651367, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.13816833496094
validation[          , genome  5834] predictions:    4860/  10000 (48.60%), best:    4860/10000 (48.60%), error:     18542.11914, best error:     18542.11914 on epoch:     0, epoch:    0/50, mu: 0.5000000000, learning_rate: 0.0125000002, weight_decay: 0.0005000000

epoch time: 1016.3884887695s, input_fired_time: 9.4856719971, output_fired_time: 7.4001445770, propagate_forward_time: 364.7005920410, propagate_backward_time: 620.5789794922, weight_update_time: 1.0900292397, other_time: 14.2231445313
epoch time: 76.5813140869s, input_fired_time: 1.0191563368, output_fired_time: 0.0000000000, propagate_forward_time: 73.1006011963, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4615554810
validation[          , genome  5834] predictions:    4989/  10000 (49.89%), best:    4989/10000 (49.89%), error:     17581.59766, best error:     17581.59766 on epoch:     1, epoch:    1/50, mu: 0.5245000124, learning_rate: 0.0118749999, weight_decay: 0.0004750000

epoch time: 1005.6183471680s, input_fired_time: 9.7789983749, output_fired_time: 7.6538882256, propagate_forward_time: 359.0255432129, propagate_backward_time: 616.8786621094, weight_update_time: 0.9957481027, other_time: 12.2812500000
epoch time: 75.7071990967s, input_fired_time: 1.0561313629, output_fired_time: 0.0000000000, propagate_forward_time: 72.3519134521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2991561890
validation[          , genome  5834] predictions:    4905/  10000 (49.05%), best:    4905/10000 (49.05%), error:     17006.93555, best error:     17006.93555 on epoch:     2, epoch:    2/50, mu: 0.5477750301, learning_rate: 0.0112812500, weight_decay: 0.0004512500

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834_1_r1576638797_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.569886
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0107172
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.000428687
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 3
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 2
read number_validation_images: 10000
read best_validation_predictions: 4905
read best_validation_error: 17006.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5834
read normal distribution: '0 -0.293411165475845 0.638238370418549'
generator_str: '249660513'
read generator: 249660513
read generated_by_map:
1 change_size 1
reading 87 nodes.
reading 339 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1039.61938476563s, input_fired_time: 10.3511190414429, output_fired_time: 8.04761219024658, propagate_forward_time: 373.066925048828, propagate_backward_time: 633.61376953125, weight_update_time: 1.08979105949402, other_time: 14.5400390625
epoch time: 75.5235366821289s, input_fired_time: 1.06172227859497, output_fired_time: 0, propagate_forward_time: 72.0496215820313, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.41219329833984
validation[          , genome  5834] predictions:    4865/  10000 (48.65%), best:    4905/10000 (49.05%), error:     17411.74609, best error:     17006.93555 on epoch:     2, epoch:    3/50, mu: 0.5698862672, learning_rate: 0.0107171871, weight_decay: 0.0004286875

epoch time: 1011.9749145508s, input_fired_time: 10.0977945328, output_fired_time: 8.6863784790, propagate_forward_time: 363.0000000000, propagate_backward_time: 615.8758544922, weight_update_time: 1.1135661602, other_time: 14.3148803711
epoch time: 76.8021163940s, input_fired_time: 1.1471167803, output_fired_time: 0.0000000000, propagate_forward_time: 73.3106613159, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3443374634
validation[          , genome  5834] predictions:    5000/  10000 (50.00%), best:    5000/10000 (50.00%), error:     15831.11426, best error:     15831.11426 on epoch:     4, epoch:    4/50, mu: 0.5908919573, learning_rate: 0.0101813274, weight_decay: 0.0004072531

epoch time: 1021.8287353516s, input_fired_time: 9.5239105225, output_fired_time: 8.8276910782, propagate_forward_time: 366.7327575684, propagate_backward_time: 623.4364624023, weight_update_time: 1.2333561182, other_time: 13.3079223633
epoch time: 74.1601791382s, input_fired_time: 1.0692348480, output_fired_time: 0.0000000000, propagate_forward_time: 70.8760986328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2148437500
validation[          , genome  5834] predictions:    5161/  10000 (51.61%), best:    5161/10000 (51.61%), error:     15439.31543, best error:     15439.31543 on epoch:     5, epoch:    5/50, mu: 0.6108473539, learning_rate: 0.0096722608, weight_decay: 0.0003868904

epoch time: 1005.4924926758s, input_fired_time: 10.4802560806, output_fired_time: 7.7485990524, propagate_forward_time: 358.3549499512, propagate_backward_time: 614.0732421875, weight_update_time: 2.0439038277, other_time: 14.8354492188
epoch time: 76.0302810669s, input_fired_time: 1.0089800358, output_fired_time: 0.0000000000, propagate_forward_time: 72.8741912842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1471099854
validation[          , genome  5834] predictions:    5381/  10000 (53.81%), best:    5381/10000 (53.81%), error:     14229.82422, best error:     14229.82422 on epoch:     6, epoch:    6/50, mu: 0.6298049688, learning_rate: 0.0091886474, weight_decay: 0.0003675459

epoch time: 1003.0959472656s, input_fired_time: 9.9056768417, output_fired_time: 8.6095943451, propagate_forward_time: 365.4265441895, propagate_backward_time: 602.8084716797, weight_update_time: 1.1726955175, other_time: 16.3455810547
epoch time: 76.1426010132s, input_fired_time: 1.0695863962, output_fired_time: 0.0000000000, propagate_forward_time: 71.7957611084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2772521973
validation[          , genome  5834] predictions:    5119/  10000 (51.19%), best:    5381/10000 (53.81%), error:     15554.48926, best error:     14229.82422 on epoch:     6, epoch:    7/50, mu: 0.6478147507, learning_rate: 0.0087292148, weight_decay: 0.0003491686

epoch time: 998.6789550781s, input_fired_time: 10.3296575546, output_fired_time: 8.7166767120, propagate_forward_time: 361.7097473145, propagate_backward_time: 605.3862915039, weight_update_time: 1.0672473907, other_time: 12.5365600586
epoch time: 74.6314773560s, input_fired_time: 0.9736201167, output_fired_time: 0.0000000000, propagate_forward_time: 71.4505538940, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2073059082
validation[          , genome  5834] predictions:    5162/  10000 (51.62%), best:    5381/10000 (53.81%), error:     15379.76758, best error:     14229.82422 on epoch:     6, epoch:    8/50, mu: 0.6649240255, learning_rate: 0.0082927542, weight_decay: 0.0003317102

epoch time: 1003.1876831055s, input_fired_time: 9.1930952072, output_fired_time: 8.1696252823, propagate_forward_time: 363.6741638184, propagate_backward_time: 609.4506225586, weight_update_time: 1.1565310955, other_time: 12.7001342773
epoch time: 75.0908279419s, input_fired_time: 1.0276274681, output_fired_time: 0.0000000000, propagate_forward_time: 71.9315261841, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1316757202
validation[          , genome  5834] predictions:    5141/  10000 (51.41%), best:    5381/10000 (53.81%), error:     15350.07422, best error:     14229.82422 on epoch:     6, epoch:    9/50, mu: 0.6811778545, learning_rate: 0.0078781163, weight_decay: 0.0003151247

epoch time: 1016.0700073242s, input_fired_time: 10.2112312317, output_fired_time: 7.5541315079, propagate_forward_time: 363.6092529297, propagate_backward_time: 622.4148559570, weight_update_time: 1.1475884914, other_time: 12.2805175781
epoch time: 75.8572998047s, input_fired_time: 1.1005386114, output_fired_time: 0.0000000000, propagate_forward_time: 71.5855026245, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1712570190
validation[          , genome  5834] predictions:    5429/  10000 (54.29%), best:    5381/10000 (53.81%), error:     14562.41797, best error:     14229.82422 on epoch:     6, epoch:   10/50, mu: 0.6966189742, learning_rate: 0.0074842102, weight_decay: 0.0002993684

epoch time: 1001.0264892578s, input_fired_time: 8.8773460388, output_fired_time: 7.3202638626, propagate_forward_time: 357.9765930176, propagate_backward_time: 613.4919433594, weight_update_time: 1.1412656307, other_time: 13.3603515625
epoch time: 75.6480484009s, input_fired_time: 1.0215567350, output_fired_time: 0.0000000000, propagate_forward_time: 72.4353561401, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1911392212
validation[          , genome  5834] predictions:    5330/  10000 (53.30%), best:    5381/10000 (53.81%), error:     14520.87695, best error:     14229.82422 on epoch:     6, epoch:   11/50, mu: 0.7112880349, learning_rate: 0.0071099997, weight_decay: 0.0002844000

epoch time: 1002.9008789063s, input_fired_time: 10.3084182739, output_fired_time: 7.4000573158, propagate_forward_time: 355.3484802246, propagate_backward_time: 614.0997924805, weight_update_time: 1.2325220108, other_time: 15.7440795898
epoch time: 75.5777206421s, input_fired_time: 0.9890281558, output_fired_time: 0.0000000000, propagate_forward_time: 72.1967468262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3919448853
validation[          , genome  5834] predictions:    5395/  10000 (53.95%), best:    5395/10000 (53.95%), error:     13808.08984, best error:     13808.08984 on epoch:    12, epoch:   12/50, mu: 0.7252236605, learning_rate: 0.0067544994, weight_decay: 0.0002701800

epoch time: 1007.1105957031s, input_fired_time: 9.2810411453, output_fired_time: 7.6385550499, propagate_forward_time: 358.7309875488, propagate_backward_time: 617.5757446289, weight_update_time: 1.0954949856, other_time: 13.8842163086
epoch time: 76.2606735229s, input_fired_time: 0.9895404577, output_fired_time: 0.0000000000, propagate_forward_time: 73.0531997681, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2179336548
validation[          , genome  5834] predictions:    5435/  10000 (54.35%), best:    5395/10000 (53.95%), error:     14559.86035, best error:     13808.08984 on epoch:    12, epoch:   13/50, mu: 0.7384625077, learning_rate: 0.0064167744, weight_decay: 0.0002566710

epoch time: 1004.1473999023s, input_fired_time: 10.3081169128, output_fired_time: 8.7821531296, propagate_forward_time: 361.1352844238, propagate_backward_time: 611.2177734375, weight_update_time: 1.1127241850, other_time: 12.7041015625
epoch time: 76.2692947388s, input_fired_time: 0.9444023371, output_fired_time: 0.0000000000, propagate_forward_time: 72.9583053589, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3665847778
validation[          , genome  5834] predictions:    5463/  10000 (54.63%), best:    5395/10000 (53.95%), error:     14001.31836, best error:     13808.08984 on epoch:    12, epoch:   14/50, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 1007.7040405273s, input_fired_time: 9.5382070541, output_fired_time: 7.7600450516, propagate_forward_time: 363.2932128906, propagate_backward_time: 614.2536010742, weight_update_time: 1.1810678244, other_time: 12.8589477539
epoch time: 76.1153793335s, input_fired_time: 1.1115976572, output_fired_time: 0.0000000000, propagate_forward_time: 72.7352142334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2685699463
validation[          , genome  5834] predictions:    5450/  10000 (54.50%), best:    5395/10000 (53.95%), error:     14196.17773, best error:     13808.08984 on epoch:    12, epoch:   15/50, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 1004.4784545898s, input_fired_time: 10.5324735641, output_fired_time: 7.4095950127, propagate_forward_time: 364.5285339355, propagate_backward_time: 608.4349975586, weight_update_time: 1.2805711031, other_time: 13.5728149414
epoch time: 76.4654464722s, input_fired_time: 1.1748831272, output_fired_time: 0.0000000000, propagate_forward_time: 73.0157928467, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2747726440
validation[          , genome  5834] predictions:    5545/  10000 (55.45%), best:    5395/10000 (53.95%), error:     13826.48438, best error:     13808.08984 on epoch:    12, epoch:   16/50, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 1007.5349731445s, input_fired_time: 8.9485006332, output_fired_time: 8.6111803055, propagate_forward_time: 361.3916320801, propagate_backward_time: 614.9898681641, weight_update_time: 1.1407494545, other_time: 13.5937500000
epoch time: 75.6077423096s, input_fired_time: 1.1265566349, output_fired_time: 0.0000000000, propagate_forward_time: 72.2871170044, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1940689087
validation[          , genome  5834] predictions:    5477/  10000 (54.77%), best:    5395/10000 (53.95%), error:     13951.30371, best error:     13808.08984 on epoch:    12, epoch:   17/50, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 1005.8054809570s, input_fired_time: 9.3998498917, output_fired_time: 8.2383127213, propagate_forward_time: 363.2817687988, propagate_backward_time: 610.1180419922, weight_update_time: 1.0914896727, other_time: 14.7674560547
epoch time: 75.4478073120s, input_fired_time: 1.1142730713, output_fired_time: 0.0000000000, propagate_forward_time: 72.1462478638, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1872863770
validation[          , genome  5834] predictions:    5593/  10000 (55.93%), best:    5593/10000 (55.93%), error:     13428.32324, best error:     13428.32324 on epoch:    18, epoch:   18/50, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 1003.8812866211s, input_fired_time: 10.8254261017, output_fired_time: 7.7085452080, propagate_forward_time: 367.4922485352, propagate_backward_time: 605.0576171875, weight_update_time: 1.2902905941, other_time: 12.7974243164
epoch time: 74.5978775024s, input_fired_time: 0.9920759201, output_fired_time: 0.0000000000, propagate_forward_time: 71.3995819092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2062225342
validation[          , genome  5834] predictions:    5721/  10000 (57.21%), best:    5721/10000 (57.21%), error:     13318.64551, best error:     13318.64551 on epoch:    19, epoch:   19/50, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

epoch time: 1007.2634277344s, input_fired_time: 9.1498947144, output_fired_time: 7.2100729942, propagate_forward_time: 366.2457275391, propagate_backward_time: 610.9669189453, weight_update_time: 0.9900125861, other_time: 13.6907958984
epoch time: 75.8469314575s, input_fired_time: 1.0474101305, output_fired_time: 0.0000000000, propagate_forward_time: 72.6201477051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1793746948
validation[          , genome  5834] predictions:    5515/  10000 (55.15%), best:    5721/10000 (57.21%), error:     13859.16602, best error:     13318.64551 on epoch:    19, epoch:   20/50, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 1005.4932250977s, input_fired_time: 11.4742555618, output_fired_time: 7.2226643562, propagate_forward_time: 359.9342956543, propagate_backward_time: 614.0776977539, weight_update_time: 1.1855669022, other_time: 12.7843627930
epoch time: 76.2356796265s, input_fired_time: 1.1576873064, output_fired_time: 0.0000000000, propagate_forward_time: 72.8084487915, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2695465088
validation[          , genome  5834] predictions:    5442/  10000 (54.42%), best:    5721/10000 (57.21%), error:     13915.23242, best error:     13318.64551 on epoch:    19, epoch:   21/50, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 1038.2977294922s, input_fired_time: 10.5453424454, output_fired_time: 8.5253772736, propagate_forward_time: 374.1345214844, propagate_backward_time: 631.2573852539, weight_update_time: 1.2267606258, other_time: 13.8351440430
epoch time: 75.4302215576s, input_fired_time: 1.1144742966, output_fired_time: 0.0000000000, propagate_forward_time: 71.0124206543, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3033294678
validation[          , genome  5834] predictions:    5616/  10000 (56.16%), best:    5616/10000 (56.16%), error:     13316.75781, best error:     13316.75781 on epoch:    22, epoch:   22/50, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 1007.7736206055s, input_fired_time: 10.3482227325, output_fired_time: 7.3788876534, propagate_forward_time: 364.9119567871, propagate_backward_time: 612.7678833008, weight_update_time: 1.0229809284, other_time: 12.3666381836
epoch time: 76.3355636597s, input_fired_time: 1.0447554588, output_fired_time: 0.0000000000, propagate_forward_time: 73.0787353516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2120742798
validation[          , genome  5834] predictions:    5661/  10000 (56.61%), best:    5616/10000 (56.16%), error:     13420.85254, best error:     13316.75781 on epoch:    22, epoch:   23/50, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 1006.3823852539s, input_fired_time: 9.5010061264, output_fired_time: 7.4719033241, propagate_forward_time: 365.6232910156, propagate_backward_time: 611.5938110352, weight_update_time: 1.0206826925, other_time: 12.1923828125
epoch time: 75.7294769287s, input_fired_time: 1.1075811386, output_fired_time: 0.0000000000, propagate_forward_time: 72.4384231567, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1834716797
validation[          , genome  5834] predictions:    5694/  10000 (56.94%), best:    5694/10000 (56.94%), error:     13229.09277, best error:     13229.09277 on epoch:    24, epoch:   24/50, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 1006.4250488281s, input_fired_time: 9.3241958618, output_fired_time: 7.6366786957, propagate_forward_time: 365.1944580078, propagate_backward_time: 611.9979858398, weight_update_time: 1.1195893288, other_time: 12.2717285156
epoch time: 76.5163650513s, input_fired_time: 1.0751039982, output_fired_time: 0.0000000000, propagate_forward_time: 73.1905517578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2507095337
validation[          , genome  5834] predictions:    5745/  10000 (57.45%), best:    5694/10000 (56.94%), error:     13332.48828, best error:     13229.09277 on epoch:    24, epoch:   25/50, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 1001.5011596680s, input_fired_time: 10.8795394897, output_fired_time: 7.4501533508, propagate_forward_time: 363.8029785156, propagate_backward_time: 605.5125122070, weight_update_time: 1.2415103912, other_time: 13.8560180664
epoch time: 76.7981185913s, input_fired_time: 1.0911378860, output_fired_time: 0.0000000000, propagate_forward_time: 73.3609924316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3459854126
validation[          , genome  5834] predictions:    5754/  10000 (57.54%), best:    5754/10000 (57.54%), error:     12833.58887, best error:     12833.58887 on epoch:    26, epoch:   26/50, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 1004.0040893555s, input_fired_time: 9.3063182831, output_fired_time: 7.5742373466, propagate_forward_time: 362.0757751465, propagate_backward_time: 611.3516845703, weight_update_time: 1.1605596542, other_time: 13.6960449219
epoch time: 76.4352722168s, input_fired_time: 1.1619637012, output_fired_time: 0.0000000000, propagate_forward_time: 72.9229965210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3503112793
validation[          , genome  5834] predictions:    5738/  10000 (57.38%), best:    5754/10000 (57.54%), error:     12981.00000, best error:     12833.58887 on epoch:    26, epoch:   27/50, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 1009.7028808594s, input_fired_time: 9.6392135620, output_fired_time: 8.2358341217, propagate_forward_time: 367.0932617188, propagate_backward_time: 610.8363037109, weight_update_time: 1.0401090384, other_time: 13.8982543945
epoch time: 76.9235458374s, input_fired_time: 1.1175143719, output_fired_time: 0.0000000000, propagate_forward_time: 73.5384445190, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2675857544
validation[          , genome  5834] predictions:    5702/  10000 (57.02%), best:    5754/10000 (57.54%), error:     13196.71777, best error:     12833.58887 on epoch:    26, epoch:   28/50, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1010.4149169922s, input_fired_time: 9.3311767578, output_fired_time: 7.1436276436, propagate_forward_time: 365.8450622559, propagate_backward_time: 615.4941406250, weight_update_time: 1.1622464657, other_time: 12.6009521484
epoch time: 74.4314880371s, input_fired_time: 1.0751775503, output_fired_time: 0.0000000000, propagate_forward_time: 71.1092147827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2470932007
validation[          , genome  5834] predictions:    5810/  10000 (58.10%), best:    5754/10000 (57.54%), error:     12895.47949, best error:     12833.58887 on epoch:    26, epoch:   29/50, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 1053.0491943359s, input_fired_time: 9.6826400757, output_fired_time: 8.8980102539, propagate_forward_time: 378.7024536133, propagate_backward_time: 639.2316894531, weight_update_time: 1.2347750664, other_time: 16.5343627930
epoch time: 114.2620620728s, input_fired_time: 1.6071430445, output_fired_time: 0.0000000000, propagate_forward_time: 109.4140625000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2408599854
validation[          , genome  5834] predictions:    5673/  10000 (56.73%), best:    5754/10000 (57.54%), error:     13412.39941, best error:     12833.58887 on epoch:    26, epoch:   30/50, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834_1_r1576638797_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.890086
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00254883
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.000101953
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 31
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 26
read number_validation_images: 10000
read best_validation_predictions: 5754
read best_validation_error: 12833.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5834
read normal distribution: '0 -0.293411165475845 0.638238370418549'
generator_str: '1926871990'
read generator: 1926871990
read generated_by_map:
1 change_size 1
reading 87 nodes.
reading 339 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1035.95227050781s, input_fired_time: 9.60557460784912, output_fired_time: 8.66279697418213, propagate_forward_time: 370.561279296875, propagate_backward_time: 631.785095214844, weight_update_time: 1.10460925102234, other_time: 15.3375244140625
epoch time: 77.2597198486328s, input_fired_time: 1.03480064868927, output_fired_time: 0, propagate_forward_time: 73.8160018920898, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.40892028808594
validation[          , genome  5834] predictions:    5775/  10000 (57.75%), best:    5754/10000 (57.54%), error:     13164.86523, best error:     12833.58887 on epoch:    26, epoch:   31/50, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 851.9301147461s, input_fired_time: 8.0238971710, output_fired_time: 6.3260130882, propagate_forward_time: 307.5333862305, propagate_backward_time: 518.0895385742, weight_update_time: 0.9115265608, other_time: 11.9572753906
epoch time: 60.8684005737s, input_fired_time: 0.8555240631, output_fired_time: 0.0000000000, propagate_forward_time: 58.1111984253, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9016799927
validation[          , genome  5834] predictions:    5709/  10000 (57.09%), best:    5754/10000 (57.54%), error:     13217.27344, best error:     12833.58887 on epoch:    26, epoch:   32/50, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 804.1071777344s, input_fired_time: 8.6193895340, output_fired_time: 6.0033845901, propagate_forward_time: 289.1070251465, propagate_backward_time: 489.4989013672, weight_update_time: 0.8760014772, other_time: 10.8785095215
epoch time: 62.0993347168s, input_fired_time: 0.8794143200, output_fired_time: 0.0000000000, propagate_forward_time: 59.3845405579, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8353805542
validation[          , genome  5834] predictions:    5690/  10000 (56.90%), best:    5754/10000 (57.54%), error:     13395.30957, best error:     12833.58887 on epoch:    26, epoch:   33/50, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 809.9906005859s, input_fired_time: 7.6548385620, output_fired_time: 5.9764399529, propagate_forward_time: 288.9759521484, propagate_backward_time: 496.9963378906, weight_update_time: 0.9172230363, other_time: 10.3870239258
epoch time: 61.3572616577s, input_fired_time: 0.8706942797, output_fired_time: 0.0000000000, propagate_forward_time: 58.6699409485, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8166275024
validation[          , genome  5834] predictions:    5801/  10000 (58.01%), best:    5754/10000 (57.54%), error:     13208.50977, best error:     12833.58887 on epoch:    26, epoch:   34/50, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 811.5255737305s, input_fired_time: 8.6292829514, output_fired_time: 8.0256748199, propagate_forward_time: 289.5915222168, propagate_backward_time: 493.8837585449, weight_update_time: 0.8863738775, other_time: 11.3953247070
epoch time: 60.4498443604s, input_fired_time: 0.8959057927, output_fired_time: 0.0000000000, propagate_forward_time: 57.7330513000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8208885193
validation[          , genome  5834] predictions:    5755/  10000 (57.55%), best:    5754/10000 (57.54%), error:     13224.17383, best error:     12833.58887 on epoch:    26, epoch:   35/50, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 804.6660766602s, input_fired_time: 7.6308279037, output_fired_time: 5.9473319054, propagate_forward_time: 287.7449645996, propagate_backward_time: 490.9570922852, weight_update_time: 1.9353380203, other_time: 12.3858947754
epoch time: 70.4783401489s, input_fired_time: 1.0047229528, output_fired_time: 0.0000000000, propagate_forward_time: 67.3719329834, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1016845703
validation[          , genome  5834] predictions:    5936/  10000 (59.36%), best:    5936/10000 (59.36%), error:     12351.61523, best error:     12351.61523 on epoch:    36, epoch:   36/50, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 812.2244873047s, input_fired_time: 7.6259245872, output_fired_time: 5.9581060410, propagate_forward_time: 297.0626525879, propagate_backward_time: 490.0050354004, weight_update_time: 0.8917860389, other_time: 11.5727539063
epoch time: 60.3013076782s, input_fired_time: 0.9103305936, output_fired_time: 0.0000000000, propagate_forward_time: 57.6010475159, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7899284363
validation[          , genome  5834] predictions:    5892/  10000 (58.92%), best:    5936/10000 (59.36%), error:     12745.40332, best error:     12351.61523 on epoch:    36, epoch:   37/50, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 802.8122558594s, input_fired_time: 7.5541958809, output_fired_time: 6.9772601128, propagate_forward_time: 283.9367065430, propagate_backward_time: 494.1305847168, weight_update_time: 0.9248341322, other_time: 10.2135314941
epoch time: 61.4068908691s, input_fired_time: 0.8636738658, output_fired_time: 0.0000000000, propagate_forward_time: 58.6743125916, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8689041138
validation[          , genome  5834] predictions:    6013/  10000 (60.13%), best:    5936/10000 (59.36%), error:     12407.25879, best error:     12351.61523 on epoch:    36, epoch:   38/50, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 803.0946655273s, input_fired_time: 7.5553460121, output_fired_time: 5.9597353935, propagate_forward_time: 284.2379455566, propagate_backward_time: 495.1672668457, weight_update_time: 0.8926365972, other_time: 10.1743774414
epoch time: 60.8187179565s, input_fired_time: 0.8681523204, output_fired_time: 0.0000000000, propagate_forward_time: 58.1259155273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8246498108
validation[          , genome  5834] predictions:    5933/  10000 (59.33%), best:    5936/10000 (59.36%), error:     12427.67578, best error:     12351.61523 on epoch:    36, epoch:   39/50, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 800.2929077148s, input_fired_time: 8.6620664597, output_fired_time: 5.8856220245, propagate_forward_time: 288.6494445801, propagate_backward_time: 487.0501403809, weight_update_time: 0.8653449416, other_time: 10.0456542969
epoch time: 59.9402389526s, input_fired_time: 0.8747471571, output_fired_time: 0.0000000000, propagate_forward_time: 57.2965202332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7689704895
validation[          , genome  5834] predictions:    5896/  10000 (58.96%), best:    5936/10000 (59.36%), error:     12506.13086, best error:     12351.61523 on epoch:    36, epoch:   40/50, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 797.4785156250s, input_fired_time: 7.6618585587, output_fired_time: 6.9031939507, propagate_forward_time: 286.5856323242, propagate_backward_time: 484.1867675781, weight_update_time: 1.8865731955, other_time: 12.1410522461
epoch time: 59.5940704346s, input_fired_time: 0.8814413548, output_fired_time: 0.0000000000, propagate_forward_time: 56.9744224548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7382049561
validation[          , genome  5834] predictions:    5986/  10000 (59.86%), best:    5986/10000 (59.86%), error:     12318.22168, best error:     12318.22168 on epoch:    41, epoch:   41/50, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 801.7470703125s, input_fired_time: 7.6010322571, output_fired_time: 5.9937496185, propagate_forward_time: 282.3087158203, propagate_backward_time: 495.6265563965, weight_update_time: 0.8817433119, other_time: 10.2170104980
epoch time: 59.9218521118s, input_fired_time: 0.8538808823, output_fired_time: 0.0000000000, propagate_forward_time: 57.3178405762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7501296997
validation[          , genome  5834] predictions:    5980/  10000 (59.80%), best:    5980/10000 (59.80%), error:     12304.88867, best error:     12304.88867 on epoch:    42, epoch:   42/50, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 799.8159790039s, input_fired_time: 8.5368356705, output_fired_time: 5.8842949867, propagate_forward_time: 290.1056518555, propagate_backward_time: 485.1849060059, weight_update_time: 0.8773389459, other_time: 10.1042785645
epoch time: 60.6846580505s, input_fired_time: 1.8974599838, output_fired_time: 0.0000000000, propagate_forward_time: 57.0174560547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7697410583
validation[          , genome  5834] predictions:    5963/  10000 (59.63%), best:    5963/10000 (59.63%), error:     12301.36719, best error:     12301.36719 on epoch:    43, epoch:   43/50, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 798.8990478516s, input_fired_time: 7.5877661705, output_fired_time: 5.8515110016, propagate_forward_time: 293.8003540039, propagate_backward_time: 480.5252075195, weight_update_time: 0.8630237579, other_time: 11.1342163086
epoch time: 60.1814193726s, input_fired_time: 0.9261692762, output_fired_time: 0.0000000000, propagate_forward_time: 57.4232864380, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8319625854
validation[          , genome  5834] predictions:    6035/  10000 (60.35%), best:    5963/10000 (59.63%), error:     12461.18262, best error:     12301.36719 on epoch:    43, epoch:   44/50, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 798.9060058594s, input_fired_time: 7.6203150749, output_fired_time: 5.9171986580, propagate_forward_time: 291.4495239258, propagate_backward_time: 482.7180480957, weight_update_time: 0.9075996280, other_time: 11.2009582520
epoch time: 64.2877960205s, input_fired_time: 0.9478634000, output_fired_time: 0.0000000000, propagate_forward_time: 61.4601287842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8798027039
validation[          , genome  5834] predictions:    5832/  10000 (58.32%), best:    5963/10000 (59.63%), error:     13012.70313, best error:     12301.36719 on epoch:    43, epoch:   45/50, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 798.0348510742s, input_fired_time: 7.6995387077, output_fired_time: 5.8776283264, propagate_forward_time: 286.0002441406, propagate_backward_time: 486.3509216309, weight_update_time: 1.8925044537, other_time: 12.1065368652
epoch time: 60.7818984985s, input_fired_time: 0.9104811549, output_fired_time: 0.0000000000, propagate_forward_time: 58.0855445862, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7858734131
validation[          , genome  5834] predictions:    5990/  10000 (59.90%), best:    5963/10000 (59.63%), error:     12391.63086, best error:     12301.36719 on epoch:    43, epoch:   46/50, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_177_5834_1_r1576638797_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.946025
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00112181
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 4.48724e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 47
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 43
read number_validation_images: 10000
read best_validation_predictions: 5963
read best_validation_error: 12301.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5834
read normal distribution: '0 -0.293411165475845 0.638238370418549'
generator_str: '838827185'
read generator: 838827185
read generated_by_map:
1 change_size 1
reading 87 nodes.
reading 339 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1012.07263183594s, input_fired_time: 9.26899719238281, output_fired_time: 8.05365371704102, propagate_forward_time: 366.195922851563, propagate_backward_time: 615.397033691406, weight_update_time: 1.32019114494324, other_time: 13.1570434570313
epoch time: 68.7500457763672s, input_fired_time: 1.02132296562195, output_fired_time: 0, propagate_forward_time: 65.6610412597656, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.06768035888672
validation[          , genome  5834] predictions:    5975/  10000 (59.75%), best:    5963/10000 (59.63%), error:     12330.61621, best error:     12301.36719 on epoch:    43, epoch:   47/50, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 901.9260253906s, input_fired_time: 9.4776754379, output_fired_time: 6.7683405876, propagate_forward_time: 322.2731933594, propagate_backward_time: 550.9235229492, weight_update_time: 1.0492814779, other_time: 12.4833374023
epoch time: 67.9491271973s, input_fired_time: 1.0394371748, output_fired_time: 0.0000000000, propagate_forward_time: 64.8214340210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0882568359
validation[          , genome  5834] predictions:    5859/  10000 (58.59%), best:    5963/10000 (59.63%), error:     12695.80664, best error:     12301.36719 on epoch:    43, epoch:   48/50, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 904.7606201172s, input_fired_time: 9.6885576248, output_fired_time: 6.7602825165, propagate_forward_time: 325.4840698242, propagate_backward_time: 551.1192626953, weight_update_time: 0.9856829047, other_time: 11.7084960938
epoch time: 69.3219375610s, input_fired_time: 1.0177061558, output_fired_time: 0.0000000000, propagate_forward_time: 66.1637802124, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1404495239
validation[          , genome  5834] predictions:    6003/  10000 (60.03%), best:    5963/10000 (59.63%), error:     12462.04590, best error:     12301.36719 on epoch:    43, epoch:   49/50, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 901.8485107422s, input_fired_time: 8.6775188446, output_fired_time: 7.8370656967, propagate_forward_time: 325.5782470703, propagate_backward_time: 546.9358520508, weight_update_time: 1.0474517345, other_time: 12.8198852539
epoch time: 68.3944320679s, input_fired_time: 0.9760081768, output_fired_time: 0.0000000000, propagate_forward_time: 64.4288024902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9896240234
validation[          , genome  5834] predictions:    6035/  10000 (60.35%), best:    5963/10000 (59.63%), error:     12589.91113, best error:     12301.36719 on epoch:    43, epoch:   50/50, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 347.1041259766s, input_fired_time: 5.1411924362, output_fired_time: 0.0000000000, propagate_forward_time: 331.3858337402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.5770874023
best training[          , genome  5834] predictions:   35932/  50000 (71.86%), best:    5963/10000 (59.63%), error:     40850.50391, best error:     12301.36719 on epoch:    43, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 69.6006546021s, input_fired_time: 1.0132893324, output_fired_time: 0.0000000000, propagate_forward_time: 66.4490127563, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1383514404
   testing[          , genome  5834] predictions:    5963/  10000 (59.63%), best:    5963/10000 (59.63%), error:     12301.36719, best error:     12301.36719 on epoch:    43, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
backpropagation finished successfully!
21:23:15 (12536): called boinc_finish(0)

</stderr_txt>
]]>