Nameexact_genome_1542031203_187_5514_1
Workunit4079594
Created12 Nov 2018, 15:43:34 UTC
Sent12 Nov 2018, 15:43:36 UTC
Report deadline19 Feb 2019, 7:19:39 UTC
Received22 Nov 2018, 18:48:45 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87822
Run time1 days 5 hours 25 min 37 sec
CPU time1 days 0 hours 57 min 49 sec
Validate stateValid
Credit24,662.51
Device peak FLOPS3.37 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34

Stderr output

<core_client_version>7.0.80</core_client_version>
<![CDATA[
<stderr_txt>
    10343.91113, best error:     10258.20020 on epoch:    44, epoch:   48/100, mu: 0.9271528721, learning_rate: 0.0001252612, weight_decay: 0.0000578395

epoch time: 2281.8669433594s, input_fired_time: 48.4290122986, output_fired_time: 24.5751991272, propagate_forward_time: 803.0262451172, propagate_backward_time: 1375.0705566406, weight_update_time: 0.6630025506, other_time: 30.7659912109
epoch time: 191.7640228271s, input_fired_time: 1.2761474848, output_fired_time: 0.0000000000, propagate_forward_time: 185.1369781494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3508911133
validation[          , genome  5514] predictions:    6368/  10000 (63.68%), best:    6404/10000 (64.04%), error:     10417.12500, best error:     10258.20020 on epoch:    44, epoch:   49/100, mu: 0.9297983646, learning_rate: 0.0001154219, weight_decay: 0.0000550575

epoch time: 2943.2131347656s, input_fired_time: 54.6907043457, output_fired_time: 22.7358245850, propagate_forward_time: 1073.5321044922, propagate_backward_time: 1748.9855957031, weight_update_time: 2.0750768185, other_time: 43.2689208984
epoch time: 234.0505676270s, input_fired_time: 3.0326640606, output_fired_time: 0.0000000000, propagate_forward_time: 220.4634094238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.5544891357
validation[          , genome  5514] predictions:    6343/  10000 (63.43%), best:    6404/10000 (64.04%), error:     10463.45703, best error:     10258.20020 on epoch:    44, epoch:   50/100, mu: 0.9323325157, learning_rate: 0.0001063554, weight_decay: 0.0000524092

epoch time: 2975.3527832031s, input_fired_time: 60.0193634033, output_fired_time: 27.2074146271, propagate_forward_time: 1033.2011718750, propagate_backward_time: 1798.8543701172, weight_update_time: 0.9710536599, other_time: 56.0704345703
epoch time: 210.2221527100s, input_fired_time: 9.8216428757, output_fired_time: 0.0000000000, propagate_forward_time: 192.9606933594, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4398193359
validation[          , genome  5514] predictions:    6457/  10000 (64.57%), best:    6457/10000 (64.57%), error:     10174.59863, best error:     10174.59863 on epoch:    51, epoch:   51/100, mu: 0.9347599745, learning_rate: 0.0000980011, weight_decay: 0.0000498884

epoch time: 2842.9335937500s, input_fired_time: 68.1006240845, output_fired_time: 18.9436492920, propagate_forward_time: 1052.1428222656, propagate_backward_time: 1640.1984863281, weight_update_time: 0.9557471871, other_time: 63.5480957031
epoch time: 155.0336761475s, input_fired_time: 4.9974493980, output_fired_time: 0.0000000000, propagate_forward_time: 147.1334381104, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9027862549
validation[          , genome  5514] predictions:    6424/  10000 (64.24%), best:    6457/10000 (64.57%), error:     10272.84180, best error:     10174.59863 on epoch:    51, epoch:   52/100, mu: 0.9370852709, learning_rate: 0.0000903031, weight_decay: 0.0000474888

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514_1_r1067578815_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494793
read mu: 0.939313
read mu_delta: 0.957906
read initial_learning_rate: 0.00635601
read learning_rate: 8.32097e-05
read learning_rate_delta: 0.921449
read initial_weight_decay: 0.000616354
read weight_decay: 4.52046e-05
read weight_decay_delta: 0.9519
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.102514
read input_dropout_probability: 0.00122636
read hidden_dropout_probability: 0.0773452
read velocity_reset: 1116
read epoch: 53
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 51
read number_validation_images: 10000
read best_validation_predictions: 6457
read best_validation_error: 10174.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5514
read normal distribution: '1 -0.902009963989258 -1.10635852813721'
generator_str: '1717140156'
read generator: 1717140156
read generated_by_map:
1 crossover 1
reading 93 nodes.
reading 383 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3356.00805664063s, input_fired_time: 71.2116622924805, output_fired_time: 24.4104919433594, propagate_forward_time: 1242.2802734375, propagate_backward_time: 1963.82580566406, weight_update_time: 1.17418110370636, other_time: 54.2799072265625
epoch time: 240.762390136719s, input_fired_time: 6.17267036437988, output_fired_time: 0, propagate_forward_time: 221.678268432617, propagate_backward_time: 0, weight_update_time: 0, other_time: 12.9114532470703
validation[          , genome  5514] predictions:    6500/  10000 (65.00%), best:    6500/10000 (65.00%), error:     10093.52246, best error:     10093.52246 on epoch:    53, epoch:   53/100, mu: 0.9393126369, learning_rate: 0.0000832097, weight_decay: 0.0000452046

epoch time: 3574.7365722656s, input_fired_time: 71.6640777588, output_fired_time: 24.9067459106, propagate_forward_time: 1292.4134521484, propagate_backward_time: 2121.6010742188, weight_update_time: 1.2778894901, other_time: 64.1513671875
epoch time: 275.0613708496s, input_fired_time: 9.8037080765, output_fired_time: 0.0000000000, propagate_forward_time: 259.4257507324, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8319091797
validation[          , genome  5514] predictions:    6411/  10000 (64.11%), best:    6500/10000 (65.00%), error:     10225.46387, best error:     10093.52246 on epoch:    53, epoch:   54/100, mu: 0.9414462447, learning_rate: 0.0000766735, weight_decay: 0.0000430303

epoch time: 2981.3068847656s, input_fired_time: 45.8186950684, output_fired_time: 28.6235923767, propagate_forward_time: 1093.9035644531, propagate_backward_time: 1754.1118164063, weight_update_time: 2.8193669319, other_time: 58.8493652344
epoch time: 158.3586730957s, input_fired_time: 2.0161695480, output_fired_time: 0.0000000000, propagate_forward_time: 152.4006042480, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9418945313
validation[          , genome  5514] predictions:    6516/  10000 (65.16%), best:    6516/10000 (65.16%), error:     10036.08984, best error:     10036.08984 on epoch:    55, epoch:   55/100, mu: 0.9434900880, learning_rate: 0.0000706508, weight_decay: 0.0000409605

epoch time: 2397.8305664063s, input_fired_time: 73.0239028931, output_fired_time: 32.6154708862, propagate_forward_time: 867.7133789063, propagate_backward_time: 1396.7001953125, weight_update_time: 0.6408531666, other_time: 27.7775878906
epoch time: 181.4920501709s, input_fired_time: 1.0749356747, output_fired_time: 0.0000000000, propagate_forward_time: 178.4214324951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9956817627
validation[          , genome  5514] predictions:    6570/  10000 (65.70%), best:    6570/10000 (65.70%), error:      9924.88770, best error:      9924.88770 on epoch:    56, epoch:   56/100, mu: 0.9454478621, learning_rate: 0.0000651011, weight_decay: 0.0000389903

epoch time: 2270.9196777344s, input_fired_time: 61.0025062561, output_fired_time: 15.2960557938, propagate_forward_time: 825.4915161133, propagate_backward_time: 1346.8365478516, weight_update_time: 2.6403431892, other_time: 22.2930908203
epoch time: 154.9920043945s, input_fired_time: 3.0021929741, output_fired_time: 0.0000000000, propagate_forward_time: 149.1084136963, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8813934326
validation[          , genome  5514] predictions:    6521/  10000 (65.21%), best:    6570/10000 (65.70%), error:     10101.41406, best error:      9924.88770 on epoch:    56, epoch:   57/100, mu: 0.9473232627, learning_rate: 0.0000599874, weight_decay: 0.0000371149

epoch time: 2129.7529296875s, input_fired_time: 35.7229385376, output_fired_time: 15.1729316711, propagate_forward_time: 802.9913330078, propagate_backward_time: 1249.8479003906, weight_update_time: 2.6297147274, other_time: 26.0179443359
epoch time: 181.9689636230s, input_fired_time: 5.2099142075, output_fired_time: 0.0000000000, propagate_forward_time: 167.4255981445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3334503174
validation[          , genome  5514] predictions:    6497/  10000 (64.97%), best:    6570/10000 (65.70%), error:     10077.15234, best error:      9924.88770 on epoch:    56, epoch:   58/100, mu: 0.9491196871, learning_rate: 0.0000552754, weight_decay: 0.0000353297

epoch time: 2379.9602050781s, input_fired_time: 71.4832229614, output_fired_time: 27.9615993500, propagate_forward_time: 847.3781738281, propagate_backward_time: 1400.7736816406, weight_update_time: 0.7508289218, other_time: 32.3635253906
epoch time: 166.8524475098s, input_fired_time: 1.1412913799, output_fired_time: 0.0000000000, propagate_forward_time: 157.6151885986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.0959625244
validation[          , genome  5514] predictions:    6451/  10000 (64.51%), best:    6570/10000 (65.70%), error:     10182.11816, best error:      9924.88770 on epoch:    56, epoch:   59/100, mu: 0.9508405328, learning_rate: 0.0000509334, weight_decay: 0.0000336304

epoch time: 2256.0541992188s, input_fired_time: 37.7121276855, output_fired_time: 11.4715814590, propagate_forward_time: 814.0307617188, propagate_backward_time: 1354.7849121094, weight_update_time: 0.6813801527, other_time: 38.0546875000
epoch time: 175.9122619629s, input_fired_time: 3.1486656666, output_fired_time: 0.0000000000, propagate_forward_time: 168.6263427734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1372528076
validation[          , genome  5514] predictions:    6489/  10000 (64.89%), best:    6570/10000 (65.70%), error:     10086.33984, best error:      9924.88770 on epoch:    56, epoch:   60/100, mu: 0.9524888992, learning_rate: 0.0000469326, weight_decay: 0.0000320128

epoch time: 2164.5888671875s, input_fired_time: 33.0921440125, output_fired_time: 14.1679754257, propagate_forward_time: 795.2266235352, propagate_backward_time: 1288.0339355469, weight_update_time: 2.6938552856, other_time: 34.0683593750
epoch time: 164.0234680176s, input_fired_time: 4.1623373032, output_fired_time: 0.0000000000, propagate_forward_time: 153.8245544434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0365753174
validation[          , genome  5514] predictions:    6545/  10000 (65.45%), best:    6545/10000 (65.45%), error:      9920.92383, best error:      9920.92383 on epoch:    61, epoch:   61/100, mu: 0.9540678859, learning_rate: 0.0000432460, weight_decay: 0.0000304730

epoch time: 2300.2963867188s, input_fired_time: 42.2666778564, output_fired_time: 16.5958023071, propagate_forward_time: 819.4863891602, propagate_backward_time: 1356.1409912109, weight_update_time: 0.6670024395, other_time: 65.8067626953
epoch time: 155.8641815186s, input_fired_time: 2.0049164295, output_fired_time: 0.0000000000, propagate_forward_time: 147.0000305176, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8592376709
validation[          , genome  5514] predictions:    6476/  10000 (64.76%), best:    6545/10000 (65.45%), error:     10065.36523, best error:      9920.92383 on epoch:    61, epoch:   62/100, mu: 0.9555804133, learning_rate: 0.0000398490, weight_decay: 0.0000290072

epoch time: 2165.8791503906s, input_fired_time: 40.9171905518, output_fired_time: 18.1885814667, propagate_forward_time: 802.9346313477, propagate_backward_time: 1272.9326171875, weight_update_time: 0.6183611155, other_time: 30.9062500000
epoch time: 157.2213897705s, input_fired_time: 1.0276218653, output_fired_time: 0.0000000000, propagate_forward_time: 150.2792663574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9145050049
validation[          , genome  5514] predictions:    6496/  10000 (64.96%), best:    6545/10000 (65.45%), error:      9973.42773, best error:      9920.92383 on epoch:    61, epoch:   63/100, mu: 0.9570292830, learning_rate: 0.0000367188, weight_decay: 0.0000276120

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514_1_r1067578815_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494793
read mu: 0.958417
read mu_delta: 0.957906
read initial_learning_rate: 0.00635601
read learning_rate: 3.38346e-05
read learning_rate_delta: 0.921449
read initial_weight_decay: 0.000616354
read weight_decay: 2.62839e-05
read weight_decay_delta: 0.9519
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.102514
read input_dropout_probability: 0.00122636
read hidden_dropout_probability: 0.0773452
read velocity_reset: 1116
read epoch: 64
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 61
read number_validation_images: 10000
read best_validation_predictions: 6545
read best_validation_error: 9920.92
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5514
read normal distribution: '1 -0.902009963989258 -1.10635852813721'
generator_str: '1159140497'
read generator: 1159140497
read generated_by_map:
1 crossover 1
reading 93 nodes.
reading 383 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 143227.734375s, input_fired_time: 51.0018577575684, output_fired_time: 17.7490367889404, propagate_forward_time: 952.251342773438, propagate_backward_time: 142120.625, weight_update_time: 2.70915389060974, other_time: 86.109375
epoch time: 300.391754150391s, input_fired_time: 8.19408798217773, output_fired_time: 0, propagate_forward_time: 265.486694335938, propagate_backward_time: 0, weight_update_time: 0, other_time: 26.7109680175781
validation[          , genome  5514] predictions:    6463/  10000 (64.63%), best:    6545/10000 (65.45%), error:     10088.40820, best error:      9920.92383 on epoch:    61, epoch:   64/100, mu: 0.9584171772, learning_rate: 0.0000338346, weight_decay: 0.0000262839

epoch time: 3440.6191406250s, input_fired_time: 59.5988235474, output_fired_time: 32.0716362000, propagate_forward_time: 1232.5256347656, propagate_backward_time: 1905.8002929688, weight_update_time: 1.1005468369, other_time: 210.6228027344
epoch time: 227.4960021973s, input_fired_time: 5.3677439690, output_fired_time: 0.0000000000, propagate_forward_time: 216.8710174561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2572479248
validation[          , genome  5514] predictions:    6531/  10000 (65.31%), best:    6545/10000 (65.45%), error:      9942.69727, best error:      9920.92383 on epoch:    61, epoch:   65/100, mu: 0.9597465992, learning_rate: 0.0000311768, weight_decay: 0.0000250196

epoch time: 2346.8303222656s, input_fired_time: 50.7899703979, output_fired_time: 31.2423152924, propagate_forward_time: 849.8345336914, propagate_backward_time: 1378.9945068359, weight_update_time: 1.7124652863, other_time: 35.9688720703
epoch time: 152.4959869385s, input_fired_time: 1.9929838181, output_fired_time: 0.0000000000, propagate_forward_time: 147.6489257813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8540802002
validation[          , genome  5514] predictions:    6491/  10000 (64.91%), best:    6545/10000 (65.45%), error:     10095.68164, best error:      9920.92383 on epoch:    61, epoch:   66/100, mu: 0.9610201120, learning_rate: 0.0000287279, weight_decay: 0.0000238162

epoch time: 2037.5566406250s, input_fired_time: 34.3373413086, output_fired_time: 13.8889627457, propagate_forward_time: 770.5584106445, propagate_backward_time: 1186.5114746094, weight_update_time: 0.6052569151, other_time: 32.2604980469
epoch time: 166.4326629639s, input_fired_time: 3.0416054726, output_fired_time: 0.0000000000, propagate_forward_time: 156.4860992432, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9049530029
validation[          , genome  5514] predictions:    6525/  10000 (65.25%), best:    6545/10000 (65.45%), error:      9978.12598, best error:      9920.92383 on epoch:    61, epoch:   67/100, mu: 0.9622399807, learning_rate: 0.0000264713, weight_decay: 0.0000226707

epoch time: 2339.0346679688s, input_fired_time: 39.2807083130, output_fired_time: 15.3268280029, propagate_forward_time: 871.4946289063, propagate_backward_time: 1376.3538818359, weight_update_time: 2.6443400383, other_time: 36.5784912109
epoch time: 170.4021911621s, input_fired_time: 2.1158132553, output_fired_time: 0.0000000000, propagate_forward_time: 161.9852447510, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3011322021
validation[          , genome  5514] predictions:    6566/  10000 (65.66%), best:    6566/10000 (65.66%), error:      9894.13574, best error:      9894.13574 on epoch:    68, epoch:   68/100, mu: 0.9634085298, learning_rate: 0.0000243919, weight_decay: 0.0000215802

epoch time: 2687.1274414063s, input_fired_time: 46.4330177307, output_fired_time: 15.6479206085, propagate_forward_time: 990.6438598633, propagate_backward_time: 1596.2453613281, weight_update_time: 4.7661480904, other_time: 38.1572265625
epoch time: 252.5695953369s, input_fired_time: 7.9392676353, output_fired_time: 0.0000000000, propagate_forward_time: 237.8108520508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8194732666
validation[          , genome  5514] predictions:    6596/  10000 (65.96%), best:    6596/10000 (65.96%), error:      9793.38184, best error:      9793.38184 on epoch:    69, epoch:   69/100, mu: 0.9645278454, learning_rate: 0.0000224759, weight_decay: 0.0000205422

epoch time: 2112.4230957031s, input_fired_time: 38.7648735046, output_fired_time: 11.0087604523, propagate_forward_time: 779.5203857422, propagate_backward_time: 1264.1878662109, weight_update_time: 0.6423199773, other_time: 18.9411621094
epoch time: 165.0190582275s, input_fired_time: 2.0918421745, output_fired_time: 0.0000000000, propagate_forward_time: 155.8997650146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0274505615
validation[          , genome  5514] predictions:    6558/  10000 (65.58%), best:    6596/10000 (65.96%), error:      9875.19238, best error:      9793.38184 on epoch:    69, epoch:   70/100, mu: 0.9656000733, learning_rate: 0.0000207104, weight_decay: 0.0000195541

epoch time: 2204.3813476563s, input_fired_time: 53.5598411560, output_fired_time: 25.4558582306, propagate_forward_time: 769.3720703125, propagate_backward_time: 1316.2779541016, weight_update_time: 0.6663112044, other_time: 39.7156982422
epoch time: 183.3518524170s, input_fired_time: 2.1533858776, output_fired_time: 0.0000000000, propagate_forward_time: 173.7711334229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4273376465
validation[          , genome  5514] predictions:    6598/  10000 (65.98%), best:    6596/10000 (65.96%), error:      9804.18750, best error:      9793.38184 on epoch:    69, epoch:   71/100, mu: 0.9666271806, learning_rate: 0.0000190836, weight_decay: 0.0000186136

epoch time: 2355.5283203125s, input_fired_time: 35.3946647644, output_fired_time: 14.3149232864, propagate_forward_time: 871.2403564453, propagate_backward_time: 1387.8917236328, weight_update_time: 10.6854267120, other_time: 46.6865234375
epoch time: 180.6551513672s, input_fired_time: 5.1427607536, output_fired_time: 0.0000000000, propagate_forward_time: 158.3453063965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.1670837402
validation[          , genome  5514] predictions:    6603/  10000 (66.03%), best:    6603/10000 (66.03%), error:      9789.36133, best error:      9789.36133 on epoch:    72, epoch:   72/100, mu: 0.9676110148, learning_rate: 0.0000175846, weight_decay: 0.0000177183

epoch time: 2400.1296386719s, input_fired_time: 41.3922920227, output_fired_time: 16.7051830292, propagate_forward_time: 902.3679809570, propagate_backward_time: 1408.6019287109, weight_update_time: 4.6768155098, other_time: 31.0623779297
epoch time: 179.2921142578s, input_fired_time: 1.1887528896, output_fired_time: 0.0000000000, propagate_forward_time: 174.0063018799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0970611572
validation[          , genome  5514] predictions:    6510/  10000 (65.10%), best:    6603/10000 (66.03%), error:      9994.24316, best error:      9789.36133 on epoch:    72, epoch:   73/100, mu: 0.9685534835, learning_rate: 0.0000162033, weight_decay: 0.0000168660

epoch time: 2588.8159179688s, input_fired_time: 34.1058197021, output_fired_time: 13.6643095016, propagate_forward_time: 887.6727905273, propagate_backward_time: 1474.8776855469, weight_update_time: 0.6749867797, other_time: 178.4953613281
epoch time: 180.0069122314s, input_fired_time: 2.3875374794, output_fired_time: 0.0000000000, propagate_forward_time: 172.3441619873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2752075195
validation[          , genome  5514] predictions:    6580/  10000 (65.80%), best:    6603/10000 (66.03%), error:      9796.85645, best error:      9789.36133 on epoch:    72, epoch:   74/100, mu: 0.9694562554, learning_rate: 0.0000149305, weight_decay: 0.0000160548

epoch time: 2555.3557128906s, input_fired_time: 60.3569145203, output_fired_time: 27.5926990509, propagate_forward_time: 864.1201782227, propagate_backward_time: 1573.2152099609, weight_update_time: 1.6947407722, other_time: 30.0706787109
epoch time: 162.4470214844s, input_fired_time: 1.0466043949, output_fired_time: 0.0000000000, propagate_forward_time: 158.3894805908, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0109405518
validation[          , genome  5514] predictions:    6594/  10000 (65.94%), best:    6603/10000 (66.03%), error:      9792.60156, best error:      9789.36133 on epoch:    72, epoch:   75/100, mu: 0.9703209996, learning_rate: 0.0000137577, weight_decay: 0.0000152826

epoch time: 57871.9531250000s, input_fired_time: 53.0618438721, output_fired_time: 23.3173904419, propagate_forward_time: 940.3221435547, propagate_backward_time: 56809.4179687500, weight_update_time: 2.8992459774, other_time: 45.8359375000
epoch time: 261.6389465332s, input_fired_time: 5.1935791969, output_fired_time: 0.0000000000, propagate_forward_time: 249.5614471436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8839263916
validation[          , genome  5514] predictions:    6634/  10000 (66.34%), best:    6634/10000 (66.34%), error:      9759.42676, best error:      9759.42676 on epoch:    76, epoch:   76/100, mu: 0.9711493850, learning_rate: 0.0000126771, weight_decay: 0.0000145475

epoch time: 3071.3649902344s, input_fired_time: 61.9419288635, output_fired_time: 25.6980628967, propagate_forward_time: 1135.0711669922, propagate_backward_time: 1788.8620605469, weight_update_time: 3.0087873936, other_time: 59.7918701172
epoch time: 236.8712005615s, input_fired_time: 1.8513456583, output_fired_time: 0.0000000000, propagate_forward_time: 228.4308013916, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5890502930
validation[          , genome  5514] predictions:    6610/  10000 (66.10%), best:    6610/10000 (66.10%), error:      9755.23242, best error:      9755.23242 on epoch:    77, epoch:   77/100, mu: 0.9719429016, learning_rate: 0.0000116813, weight_decay: 0.0000138478

epoch time: 2965.2661132813s, input_fired_time: 52.7604827881, output_fired_time: 26.6838722229, propagate_forward_time: 1051.0557861328, propagate_backward_time: 1778.6870117188, weight_update_time: 0.9410706162, other_time: 56.0789794922
epoch time: 215.1845092773s, input_fired_time: 4.0088458061, output_fired_time: 0.0000000000, propagate_forward_time: 203.5957031250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5799560547
validation[          , genome  5514] predictions:    6583/  10000 (65.83%), best:    6610/10000 (66.10%), error:      9794.17285, best error:      9755.23242 on epoch:    77, epoch:   78/100, mu: 0.9727029800, learning_rate: 0.0000107637, weight_decay: 0.0000131817

epoch time: 2922.6062011719s, input_fired_time: 63.5222778320, output_fired_time: 19.1487541199, propagate_forward_time: 1075.5621337891, propagate_backward_time: 1701.8197021484, weight_update_time: 1.1856442690, other_time: 62.5534667969
epoch time: 188.3120117188s, input_fired_time: 4.2212586403, output_fired_time: 0.0000000000, propagate_forward_time: 181.9235382080, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1672210693
validation[          , genome  5514] predictions:    6633/  10000 (66.33%), best:    6633/10000 (66.33%), error:      9708.65918, best error:      9708.65918 on epoch:    79, epoch:   79/100, mu: 0.9734311104, learning_rate: 0.0000099182, weight_decay: 0.0000125477

epoch time: 2272.6154785156s, input_fired_time: 39.3109130859, output_fired_time: 17.2901401520, propagate_forward_time: 864.6096801758, propagate_backward_time: 1310.9669189453, weight_update_time: 0.6403965950, other_time: 40.4381103516
epoch time: 156.9624633789s, input_fired_time: 1.0116422176, output_fired_time: 0.0000000000, propagate_forward_time: 151.0482482910, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9025726318
validation[          , genome  5514] predictions:    6592/  10000 (65.92%), best:    6633/10000 (66.33%), error:      9783.06934, best error:      9708.65918 on epoch:    79, epoch:   80/100, mu: 0.9741285443, learning_rate: 0.0000091391, weight_decay: 0.0000119441

epoch time: 2423.4230957031s, input_fired_time: 41.5755081177, output_fired_time: 12.5536708832, propagate_forward_time: 889.4677734375, propagate_backward_time: 1446.5085449219, weight_update_time: 2.6364834309, other_time: 33.3176269531
epoch time: 176.7342376709s, input_fired_time: 3.1623938084, output_fired_time: 0.0000000000, propagate_forward_time: 171.3899841309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1818542480
validation[          , genome  5514] predictions:    6635/  10000 (66.35%), best:    6633/10000 (66.33%), error:      9732.50488, best error:      9708.65918 on epoch:    79, epoch:   81/100, mu: 0.9747966528, learning_rate: 0.0000084212, weight_decay: 0.0000113696

epoch time: 2401.9089355469s, input_fired_time: 39.2830162048, output_fired_time: 9.4332332611, propagate_forward_time: 892.0124511719, propagate_backward_time: 1433.0499267578, weight_update_time: 0.6574583650, other_time: 28.1302490234
epoch time: 213.9384613037s, input_fired_time: 2.2669146061, output_fired_time: 0.0000000000, propagate_forward_time: 207.2715301514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4000091553
validation[          , genome  5514] predictions:    6658/  10000 (66.58%), best:    6658/10000 (66.58%), error:      9701.19336, best error:      9701.19336 on epoch:    82, epoch:   82/100, mu: 0.9754366279, learning_rate: 0.0000077597, weight_decay: 0.0000108227

epoch time: 2370.5522460938s, input_fired_time: 57.7793655396, output_fired_time: 14.5863313675, propagate_forward_time: 845.8695678711, propagate_backward_time: 1423.6169433594, weight_update_time: 0.6803554296, other_time: 28.6999511719
epoch time: 232.1743316650s, input_fired_time: 2.1754848957, output_fired_time: 0.0000000000, propagate_forward_time: 189.3353881836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 40.6634521484
validation[          , genome  5514] predictions:    6602/  10000 (66.02%), best:    6658/10000 (66.58%), error:      9800.93457, best error:      9701.19336 on epoch:    82, epoch:   83/100, mu: 0.9760496616, learning_rate: 0.0000071502, weight_decay: 0.0000103022

epoch time: 2578.6193847656s, input_fired_time: 50.1668701172, output_fired_time: 32.4286079407, propagate_forward_time: 830.8339233398, propagate_backward_time: 1372.2360839844, weight_update_time: 1.6384414434, other_time: 292.9538574219
epoch time: 171.6472167969s, input_fired_time: 3.1370882988, output_fired_time: 0.0000000000, propagate_forward_time: 165.0059356689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5041961670
validation[          , genome  5514] predictions:    6630/  10000 (66.30%), best:    6658/10000 (66.58%), error:      9796.49609, best error:      9701.19336 on epoch:    82, epoch:   84/100, mu: 0.9766368866, learning_rate: 0.0000065886, weight_decay: 0.0000098066

epoch time: 2639.0336914063s, input_fired_time: 81.8752517700, output_fired_time: 7.7837848663, propagate_forward_time: 918.6528320313, propagate_backward_time: 1568.1071777344, weight_update_time: 2.4981815815, other_time: 62.6147460938
epoch time: 214.7791900635s, input_fired_time: 3.1474633217, output_fired_time: 0.0000000000, propagate_forward_time: 207.4729461670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1587829590
validation[          , genome  5514] predictions:    6630/  10000 (66.30%), best:    6630/10000 (66.30%), error:      9693.81543, best error:      9693.81543 on epoch:    85, epoch:   85/100, mu: 0.9771993756, learning_rate: 0.0000060710, weight_decay: 0.0000093349

epoch time: 2412.3620605469s, input_fired_time: 55.5640602112, output_fired_time: 13.3704509735, propagate_forward_time: 877.8568725586, propagate_backward_time: 1438.7412109375, weight_update_time: 2.7226207256, other_time: 26.8295898438
epoch time: 167.7767028809s, input_fired_time: 7.0465269089, output_fired_time: 0.0000000000, propagate_forward_time: 158.7316436768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9985351563
validation[          , genome  5514] predictions:    6608/  10000 (66.08%), best:    6630/10000 (66.30%), error:      9785.86328, best error:      9693.81543 on epoch:    85, epoch:   86/100, mu: 0.9777382016, learning_rate: 0.0000055941, weight_decay: 0.0000088859

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514_1_r1067578815_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494793
read mu: 0.978254
read mu_delta: 0.957906
read initial_learning_rate: 0.00635601
read learning_rate: 5.15472e-06
read learning_rate_delta: 0.921449
read initial_weight_decay: 0.000616354
read weight_decay: 8.45853e-06
read weight_decay_delta: 0.9519
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.102514
read input_dropout_probability: 0.00122636
read hidden_dropout_probability: 0.0773452
read velocity_reset: 1116
read epoch: 87
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 85
read number_validation_images: 10000
read best_validation_predictions: 6630
read best_validation_error: 9693.82
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5514
read normal distribution: '1 -0.902009963989258 -1.10635852813721'
generator_str: '1728433568'
read generator: 1728433568
read generated_by_map:
1 crossover 1
reading 93 nodes.
reading 383 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514_1_r1067578815_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494793
read mu: 0.978254
read mu_delta: 0.957906
read initial_learning_rate: 0.00635601
read learning_rate: 5.15472e-06
read learning_rate_delta: 0.921449
read initial_weight_decay: 0.000616354
read weight_decay: 8.45853e-06
read weight_decay_delta: 0.9519
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.102514
read input_dropout_probability: 0.00122636
read hidden_dropout_probability: 0.0773452
read velocity_reset: 1116
read epoch: 87
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 85
read number_validation_images: 10000
read best_validation_predictions: 6630
read best_validation_error: 9693.82
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5514
read normal distribution: '1 -0.902009963989258 -1.10635852813721'
generator_str: '1728433568'
read generator: 1728433568
read generated_by_map:
1 crossover 1
reading 93 nodes.
reading 383 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2316.5966796875s, input_fired_time: 48.2737731933594, output_fired_time: 12.5650310516357, propagate_forward_time: 802.523864746094, propagate_backward_time: 1424.07055664063, weight_update_time: 1.66555082798004, other_time: 29.16357421875
epoch time: 222.621917724609s, input_fired_time: 1.43695950508118, output_fired_time: 0, propagate_forward_time: 218.37255859375, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.81239318847656
validation[          , genome  5514] predictions:    6563/  10000 (65.63%), best:    6630/10000 (66.30%), error:      9830.16113, best error:      9693.81543 on epoch:    85, epoch:   87/100, mu: 0.9782543778, learning_rate: 0.0000051547, weight_decay: 0.0000084585

epoch time: 3157.3393554688s, input_fired_time: 78.9923858643, output_fired_time: 27.6154346466, propagate_forward_time: 1161.3433837891, propagate_backward_time: 1710.2927246094, weight_update_time: 1.1300022602, other_time: 179.0953369141
epoch time: 157.7231445313s, input_fired_time: 5.0384612083, output_fired_time: 0.0000000000, propagate_forward_time: 145.7574462891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9272308350
validation[          , genome  5514] predictions:    6589/  10000 (65.89%), best:    6630/10000 (66.30%), error:      9757.83105, best error:      9693.81543 on epoch:    85, epoch:   88/100, mu: 0.9787487984, learning_rate: 0.0000047498, weight_decay: 0.0000080517

epoch time: 2222.5046386719s, input_fired_time: 39.1878013611, output_fired_time: 17.1854248047, propagate_forward_time: 809.2320556641, propagate_backward_time: 1323.7835693359, weight_update_time: 0.6361554861, other_time: 33.1157226563
epoch time: 148.3601531982s, input_fired_time: 1.1022541523, output_fired_time: 0.0000000000, propagate_forward_time: 143.4242095947, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8336944580
validation[          , genome  5514] predictions:    6573/  10000 (65.73%), best:    6630/10000 (66.30%), error:      9798.46094, best error:      9693.81543 on epoch:    85, epoch:   89/100, mu: 0.9792224169, learning_rate: 0.0000043767, weight_decay: 0.0000076644

epoch time: 2241.4162597656s, input_fired_time: 46.8342437744, output_fired_time: 18.3710479736, propagate_forward_time: 813.9910888672, propagate_backward_time: 1333.0651855469, weight_update_time: 2.6362926960, other_time: 29.1546630859
epoch time: 152.8700714111s, input_fired_time: 3.0032722950, output_fired_time: 0.0000000000, propagate_forward_time: 147.0443878174, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8224182129
validation[          , genome  5514] predictions:    6654/  10000 (66.54%), best:    6654/10000 (66.54%), error:      9672.61426, best error:      9672.61426 on epoch:    90, epoch:   90/100, mu: 0.9796760678, learning_rate: 0.0000040329, weight_decay: 0.0000072957

epoch time: 2254.8186035156s, input_fired_time: 66.6967391968, output_fired_time: 16.1968441010, propagate_forward_time: 769.8933715820, propagate_backward_time: 1374.1190185547, weight_update_time: 0.6317800283, other_time: 27.9127197266
epoch time: 161.3639068604s, input_fired_time: 2.0086488724, output_fired_time: 0.0000000000, propagate_forward_time: 147.4700469971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.8852081299
validation[          , genome  5514] predictions:    6629/  10000 (66.29%), best:    6654/10000 (66.54%), error:      9689.18652, best error:      9672.61426 on epoch:    90, epoch:   91/100, mu: 0.9801106453, learning_rate: 0.0000037161, weight_decay: 0.0000069448

epoch time: 2291.2858886719s, input_fired_time: 38.2034645081, output_fired_time: 16.2760887146, propagate_forward_time: 900.0129394531, propagate_backward_time: 1300.2749023438, weight_update_time: 0.6360577345, other_time: 36.5185546875
epoch time: 191.7035675049s, input_fired_time: 1.3823091984, output_fired_time: 0.0000000000, propagate_forward_time: 184.8659667969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4552917480
validation[          , genome  5514] predictions:    6633/  10000 (66.33%), best:    6654/10000 (66.54%), error:      9712.45215, best error:      9672.61426 on epoch:    90, epoch:   92/100, mu: 0.9805269241, learning_rate: 0.0000034242, weight_decay: 0.0000066108

epoch time: 2378.2385253906s, input_fired_time: 51.1809539795, output_fired_time: 34.5844345093, propagate_forward_time: 828.9939575195, propagate_backward_time: 1431.0895996094, weight_update_time: 1.6848882437, other_time: 32.3896484375
epoch time: 177.1304168701s, input_fired_time: 3.0775928497, output_fired_time: 0.0000000000, propagate_forward_time: 168.0343627930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0184631348
validation[          , genome  5514] predictions:    6634/  10000 (66.34%), best:    6654/10000 (66.54%), error:      9748.79297, best error:      9672.61426 on epoch:    90, epoch:   93/100, mu: 0.9809256792, learning_rate: 0.0000031553, weight_decay: 0.0000062928

epoch time: 2227.1867675781s, input_fired_time: 46.9807205200, output_fired_time: 14.1592483521, propagate_forward_time: 853.2994995117, propagate_backward_time: 1279.6108398438, weight_update_time: 3.7025284767, other_time: 33.1364746094
epoch time: 159.9356079102s, input_fired_time: 4.1121635437, output_fired_time: 0.0000000000, propagate_forward_time: 145.6754913330, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.1479492188
validation[          , genome  5514] predictions:    6674/  10000 (66.74%), best:    6674/10000 (66.74%), error:      9643.15527, best error:      9643.15527 on epoch:    94, epoch:   94/100, mu: 0.9813076258, learning_rate: 0.0000029074, weight_decay: 0.0000059901

epoch time: 2221.7612304688s, input_fired_time: 39.4796447754, output_fired_time: 15.0722045898, propagate_forward_time: 843.6455688477, propagate_backward_time: 1296.8242187500, weight_update_time: 0.6548365951, other_time: 26.7395019531
epoch time: 172.7708587646s, input_fired_time: 1.1048659086, output_fired_time: 0.0000000000, propagate_forward_time: 169.7131500244, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9528503418
validation[          , genome  5514] predictions:    6645/  10000 (66.45%), best:    6674/10000 (66.74%), error:      9671.33496, best error:      9643.15527 on epoch:    94, epoch:   95/100, mu: 0.9816735387, learning_rate: 0.0000026790, weight_decay: 0.0000057020

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5514_1_r1067578815_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494793
read mu: 0.982024
read mu_delta: 0.957906
read initial_learning_rate: 0.00635601
read learning_rate: 2.46859e-06
read learning_rate_delta: 0.921449
read initial_weight_decay: 0.000616354
read weight_decay: 5.42774e-06
read weight_decay_delta: 0.9519
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.102514
read input_dropout_probability: 0.00122636
read hidden_dropout_probability: 0.0773452
read velocity_reset: 1116
read epoch: 96
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 94
read number_validation_images: 10000
read best_validation_predictions: 6674
read best_validation_error: 9643.16
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5514
read normal distribution: '1 -0.902009963989258 -1.10635852813721'
generator_str: '1145475199'
read generator: 1145475199
read generated_by_map:
1 crossover 1
reading 93 nodes.
reading 383 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2202.63305664063s, input_fired_time: 38.9225158691406, output_fired_time: 25.2733364105225, propagate_forward_time: 768.313720703125, propagate_backward_time: 1337.10339355469, weight_update_time: 4.64813804626465, other_time: 33.0198974609375
epoch time: 166.537719726563s, input_fired_time: 2.11865019798279, output_fired_time: 0, propagate_forward_time: 161.450866699219, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.96820068359375
validation[          , genome  5514] predictions:    6587/  10000 (65.87%), best:    6674/10000 (66.74%), error:      9758.83398, best error:      9643.15527 on epoch:    94, epoch:   96/100, mu: 0.9820240140, learning_rate: 0.0000024686, weight_decay: 0.0000054277

epoch time: 2447.9604492188s, input_fired_time: 49.0559310913, output_fired_time: 20.3488006592, propagate_forward_time: 860.1785888672, propagate_backward_time: 1470.3646240234, weight_update_time: 0.8608754277, other_time: 48.0124511719
epoch time: 203.4972229004s, input_fired_time: 3.6245999336, output_fired_time: 0.0000000000, propagate_forward_time: 195.6181793213, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2544403076
validation[          , genome  5514] predictions:    6661/  10000 (66.61%), best:    6674/10000 (66.74%), error:      9660.04883, best error:      9643.15527 on epoch:    94, epoch:   97/100, mu: 0.9823597670, learning_rate: 0.0000022747, weight_decay: 0.0000051667

epoch time: 2500.8996582031s, input_fired_time: 59.1778984070, output_fired_time: 20.3214950562, propagate_forward_time: 921.1935424805, propagate_backward_time: 1451.7320556641, weight_update_time: 3.9067897797, other_time: 48.4744873047
epoch time: 182.9469757080s, input_fired_time: 2.1113765240, output_fired_time: 0.0000000000, propagate_forward_time: 178.8047027588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0308990479
validation[          , genome  5514] predictions:    6646/  10000 (66.46%), best:    6674/10000 (66.74%), error:      9716.64355, best error:      9643.15527 on epoch:    94, epoch:   98/100, mu: 0.9826813936, learning_rate: 0.0000020960, weight_decay: 0.0000049182

epoch time: 2176.0419921875s, input_fired_time: 70.5303955078, output_fired_time: 16.9485473633, propagate_forward_time: 762.2755126953, propagate_backward_time: 1301.7275390625, weight_update_time: 0.6078091264, other_time: 24.5601806641
epoch time: 193.2036285400s, input_fired_time: 4.2490510941, output_fired_time: 0.0000000000, propagate_forward_time: 173.3339538574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 15.6206207275
validation[          , genome  5514] predictions:    6668/  10000 (66.68%), best:    6674/10000 (66.74%), error:      9643.94629, best error:      9643.15527 on epoch:    94, epoch:   99/100, mu: 0.9829894900, learning_rate: 0.0000019314, weight_decay: 0.0000046816

epoch time: 2486.7856445313s, input_fired_time: 62.0034675598, output_fired_time: 15.4247655869, propagate_forward_time: 870.9179687500, propagate_backward_time: 1497.3691406250, weight_update_time: 2.7855231762, other_time: 41.0703125000
epoch time: 151.8229980469s, input_fired_time: 3.0024268627, output_fired_time: 0.0000000000, propagate_forward_time: 144.0065460205, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8140258789
validation[          , genome  5514] predictions:    6672/  10000 (66.72%), best:    6674/10000 (66.74%), error:      9648.43262, best error:      9643.15527 on epoch:    94, epoch:  100/100, mu: 0.9832845926, learning_rate: 0.0000017796, weight_decay: 0.0000044564

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 856.2401123047s, input_fired_time: 14.3068695068, output_fired_time: 0.0000000000, propagate_forward_time: 822.2487182617, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.6845092773
best training[          , genome  5514] predictions:   36987/  50000 (73.97%), best:    6674/10000 (66.74%), error:     39972.26172, best error:      9643.15527 on epoch:    94, epoch:  101/100, mu: 0.9835672975, learning_rate: 0.0000016399, weight_decay: 0.0000042421
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 204.1861572266s, input_fired_time: 2.1242215633, output_fired_time: 0.0000000000, propagate_forward_time: 198.0213623047, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0405731201
   testing[          , genome  5514] predictions:    6674/  10000 (66.74%), best:    6674/10000 (66.74%), error:      9643.15527, best error:      9643.15527 on epoch:    94, epoch:  101/100, mu: 0.9835672975, learning_rate: 0.0000016399, weight_decay: 0.0000042421
backpropagation finished successfully!
13:47:42 (3116): called boinc_finish(0)

</stderr_txt>
]]>