Nameexact_genome_1542303603_178_6005_2
Workunit4091406
Created10 Jan 2019, 16:54:44 UTC
Sent10 Jan 2019, 16:54:48 UTC
Report deadline7 Mar 2019, 13:04:36 UTC
Received15 Jan 2019, 23:44:10 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87944
Run time1 days 11 hours 55 min 13 sec
CPU time1 days 8 hours 27 min 53 sec
Validate stateValid
Credit13,960.04
Device peak FLOPS2.56 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size550.10 MB
Peak swap size570.29 MB
Peak disk usage13.21 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
ome  6005] predictions:    6034/  10000 (60.34%), best:    6034/10000 (60.34%), error:     11621.83301, best error:     11621.83301 on epoch:    11, epoch:   11/50, mu: 0.7189452648, learning_rate: 0.0005201704, weight_decay: 0.0002846888

epoch time: 3980.8391113281s, input_fired_time: 39.5627441406, output_fired_time: 29.2387886047, propagate_forward_time: 1430.7343750000, propagate_backward_time: 2438.0634765625, weight_update_time: 3.0377542973, other_time: 43.2397460938
epoch time: 321.6183471680s, input_fired_time: 3.5151481628, output_fired_time: 0.0000000000, propagate_forward_time: 309.1954345703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.9077758789
validation[          , genome  6005] predictions:    6011/  10000 (60.11%), best:    6011/10000 (60.11%), error:     11563.87207, best error:     11563.87207 on epoch:    12, epoch:   12/50, mu: 0.7332156301, learning_rate: 0.0004770865, weight_decay: 0.0002644951

epoch time: 3991.4155273438s, input_fired_time: 48.1517715454, output_fired_time: 21.9293537140, propagate_forward_time: 1432.2404785156, propagate_backward_time: 2453.2609863281, weight_update_time: 2.9573900700, other_time: 35.8327636719
epoch time: 313.0601501465s, input_fired_time: 3.3140380383, output_fired_time: 0.0000000000, propagate_forward_time: 303.7014160156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0447082520
validation[          , genome  6005] predictions:    6129/  10000 (61.29%), best:    6129/10000 (61.29%), error:     11317.38770, best error:     11317.38770 on epoch:    13, epoch:   13/50, mu: 0.7467347383, learning_rate: 0.0004375711, weight_decay: 0.0002457338

epoch time: 4022.0695800781s, input_fired_time: 42.5402030945, output_fired_time: 21.1446857452, propagate_forward_time: 1454.9606933594, propagate_backward_time: 2468.0756835938, weight_update_time: 0.8678348064, other_time: 35.3481445313
epoch time: 303.1087036133s, input_fired_time: 4.2791395187, output_fired_time: 0.0000000000, propagate_forward_time: 293.8474121094, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9821472168
validation[          , genome  6005] predictions:    6086/  10000 (60.86%), best:    6129/10000 (61.29%), error:     11387.63574, best error:     11317.38770 on epoch:    13, epoch:   14/50, mu: 0.7595421076, learning_rate: 0.0004013286, weight_decay: 0.0002283033

epoch time: 3837.1577148438s, input_fired_time: 43.4293975830, output_fired_time: 16.9142112732, propagate_forward_time: 1433.4880371094, propagate_backward_time: 2319.1274414063, weight_update_time: 0.6975861192, other_time: 24.1984863281
epoch time: 284.0923461914s, input_fired_time: 2.7649745941, output_fired_time: 0.0000000000, propagate_forward_time: 275.9722290039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3551330566
validation[          , genome  6005] predictions:    6110/  10000 (61.10%), best:    6110/10000 (61.10%), error:     11186.97949, best error:     11186.97949 on epoch:    15, epoch:   15/50, mu: 0.7716751695, learning_rate: 0.0003680879, weight_decay: 0.0002121092

epoch time: 4645.1665039063s, input_fired_time: 55.2890319824, output_fired_time: 23.0699729919, propagate_forward_time: 1641.7133789063, propagate_backward_time: 2832.5214843750, weight_update_time: 1.8612978458, other_time: 92.5727539063
epoch time: 320.9880065918s, input_fired_time: 5.4929175377, output_fired_time: 0.0000000000, propagate_forward_time: 308.0764160156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4186706543
validation[          , genome  6005] predictions:    6082/  10000 (60.82%), best:    6110/10000 (61.10%), error:     11303.50098, best error:     11186.97949 on epoch:    15, epoch:   16/50, mu: 0.7831694484, learning_rate: 0.0003376004, weight_decay: 0.0001970637

epoch time: 3948.4201660156s, input_fired_time: 47.3517570496, output_fired_time: 27.9779663086, propagate_forward_time: 1441.8336181641, propagate_backward_time: 2388.5175781250, weight_update_time: 0.8072620034, other_time: 42.7392578125
epoch time: 356.9864807129s, input_fired_time: 2.3455986977, output_fired_time: 0.0000000000, propagate_forward_time: 347.5910644531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0498046875
validation[          , genome  6005] predictions:    6121/  10000 (61.21%), best:    6110/10000 (61.10%), error:     11327.93066, best error:     11186.97949 on epoch:    15, epoch:   17/50, mu: 0.7940585613, learning_rate: 0.0003096382, weight_decay: 0.0001830855

epoch time: 3544.5920410156s, input_fired_time: 41.9270057678, output_fired_time: 19.4250602722, propagate_forward_time: 1281.2789306641, propagate_backward_time: 2170.2880859375, weight_update_time: 0.6013823152, other_time: 31.6728515625
epoch time: 337.9366149902s, input_fired_time: 5.9139842987, output_fired_time: 0.0000000000, propagate_forward_time: 322.3840637207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.6385803223
validation[          , genome  6005] predictions:    6163/  10000 (61.63%), best:    6163/10000 (61.63%), error:     11068.23535, best error:     11068.23535 on epoch:    18, epoch:   18/50, mu: 0.8043743968, learning_rate: 0.0002839919, weight_decay: 0.0001700988

epoch time: 3631.9526367188s, input_fired_time: 44.3847160339, output_fired_time: 13.7238092422, propagate_forward_time: 1341.1516113281, propagate_backward_time: 2191.7246093750, weight_update_time: 0.6584302187, other_time: 40.9677734375
epoch time: 285.1141357422s, input_fired_time: 3.6013379097, output_fired_time: 0.0000000000, propagate_forward_time: 277.5451660156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9676208496
validation[          , genome  6005] predictions:    6105/  10000 (61.05%), best:    6163/10000 (61.63%), error:     11446.59570, best error:     11068.23535 on epoch:    18, epoch:   19/50, mu: 0.8141471744, learning_rate: 0.0002604698, weight_decay: 0.0001580333

epoch time: 4078.3425292969s, input_fired_time: 40.7290077209, output_fired_time: 19.8237400055, propagate_forward_time: 1513.3642578125, propagate_backward_time: 2463.9067382813, weight_update_time: 0.7347254753, other_time: 40.5187988281
epoch time: 300.9840087891s, input_fired_time: 5.9266633987, output_fired_time: 0.0000000000, propagate_forward_time: 286.0818786621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.9754638672
validation[          , genome  6005] predictions:    6324/  10000 (63.24%), best:    6324/10000 (63.24%), error:     10857.24512, best error:     10857.24512 on epoch:    20, epoch:   20/50, mu: 0.8234053850, learning_rate: 0.0002388960, weight_decay: 0.0001468236

epoch time: 4095.1159667969s, input_fired_time: 41.7338600159, output_fired_time: 25.4680461884, propagate_forward_time: 1521.3084716797, propagate_backward_time: 2455.7934570313, weight_update_time: 0.7326601148, other_time: 50.8120117188
epoch time: 324.9218139648s, input_fired_time: 2.4215385914, output_fired_time: 0.0000000000, propagate_forward_time: 314.9959106445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5043640137
validation[          , genome  6005] predictions:    6278/  10000 (62.78%), best:    6324/10000 (63.24%), error:     10903.24023, best error:     10857.24512 on epoch:    20, epoch:   21/50, mu: 0.8321762085, learning_rate: 0.0002191091, weight_decay: 0.0001364090

epoch time: 4604.7158203125s, input_fired_time: 52.4596290588, output_fired_time: 33.3621330261, propagate_forward_time: 1661.4969482422, propagate_backward_time: 2783.3974609375, weight_update_time: 0.9184046984, other_time: 73.9995117188
epoch time: 311.8219604492s, input_fired_time: 3.8791074753, output_fired_time: 0.0000000000, propagate_forward_time: 300.9908752441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9519653320
validation[          , genome  6005] predictions:    6156/  10000 (61.56%), best:    6324/10000 (63.24%), error:     11174.65918, best error:     10857.24512 on epoch:    20, epoch:   22/50, mu: 0.8404852748, learning_rate: 0.0002009610, weight_decay: 0.0001267332

epoch time: 4140.2377929688s, input_fired_time: 43.9392776489, output_fired_time: 22.3325424194, propagate_forward_time: 1523.4114990234, propagate_backward_time: 2507.4521484375, weight_update_time: 0.7582432628, other_time: 43.1020507813
epoch time: 324.4646301270s, input_fired_time: 3.9113790989, output_fired_time: 0.0000000000, propagate_forward_time: 316.5814819336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9717712402
validation[          , genome  6005] predictions:    6213/  10000 (62.13%), best:    6324/10000 (63.24%), error:     10900.02832, best error:     10857.24512 on epoch:    20, epoch:   23/50, mu: 0.8483569026, learning_rate: 0.0001843161, weight_decay: 0.0001177437

epoch time: 3975.6904296875s, input_fired_time: 45.2310295105, output_fired_time: 15.8858299255, propagate_forward_time: 1472.8548583984, propagate_backward_time: 2404.4714355469, weight_update_time: 0.7328268290, other_time: 37.2473144531
epoch time: 252.4591522217s, input_fired_time: 3.6560187340, output_fired_time: 0.0000000000, propagate_forward_time: 240.2729034424, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5302276611
validation[          , genome  6005] predictions:    6234/  10000 (62.34%), best:    6324/10000 (63.24%), error:     10901.46484, best error:     10857.24512 on epoch:    20, epoch:   24/50, mu: 0.8558140993, learning_rate: 0.0001690498, weight_decay: 0.0001093918

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.862879
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 0.000155048
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 0.000101632
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 25
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 20
read number_validation_images: 10000
read best_validation_predictions: 6324
read best_validation_error: 10857.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '1328042915'
read generator: 1328042915
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3898.29833984375s, input_fired_time: 45.2337226867676, output_fired_time: 21.1553611755371, propagate_forward_time: 1428.998046875, propagate_backward_time: 2359.05249023438, weight_update_time: 2.70972871780396, other_time: 43.85888671875
epoch time: 277.011322021484s, input_fired_time: 1.6961658000946, output_fired_time: 0, propagate_forward_time: 265.649841308594, propagate_backward_time: 0, weight_update_time: 0, other_time: 9.66531372070313
validation[          , genome  6005] predictions:    6161/  10000 (61.61%), best:    6324/10000 (63.24%), error:     11037.71094, best error:     10857.24512 on epoch:    20, epoch:   25/50, mu: 0.8628786802, learning_rate: 0.0001550480, weight_decay: 0.0001016324

epoch time: 3572.0646972656s, input_fired_time: 40.5467185974, output_fired_time: 16.9278202057, propagate_forward_time: 1335.4658203125, propagate_backward_time: 2153.7712402344, weight_update_time: 1.6577572823, other_time: 25.3532714844
epoch time: 269.2849426270s, input_fired_time: 2.6616859436, output_fired_time: 0.0000000000, propagate_forward_time: 260.9544677734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6687927246
validation[          , genome  6005] predictions:    6387/  10000 (63.87%), best:    6387/10000 (63.87%), error:     10578.03125, best error:     10578.03125 on epoch:    26, epoch:   26/50, mu: 0.8695713282, learning_rate: 0.0001422059, weight_decay: 0.0000944233

epoch time: 4093.4206542969s, input_fired_time: 36.7672767639, output_fired_time: 22.9702320099, propagate_forward_time: 1557.5946044922, propagate_backward_time: 2435.2773437500, weight_update_time: 0.8541495800, other_time: 40.8110351563
epoch time: 246.2678527832s, input_fired_time: 3.5181043148, output_fired_time: 0.0000000000, propagate_forward_time: 237.4806671143, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2690887451
validation[          , genome  6005] predictions:    6357/  10000 (63.57%), best:    6387/10000 (63.87%), error:     10622.50977, best error:     10578.03125 on epoch:    26, epoch:   27/50, mu: 0.8759115934, learning_rate: 0.0001304275, weight_decay: 0.0000877257

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.881918
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 0.000119625
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 8.15031e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 28
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 26
read number_validation_images: 10000
read best_validation_predictions: 6387
read best_validation_error: 10578
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '757493502'
read generator: 757493502
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4208.75146484375s, input_fired_time: 47.871826171875, output_fired_time: 21.9646396636963, propagate_forward_time: 1566.55944824219, propagate_backward_time: 2518.25268554688, weight_update_time: 1.68351626396179, other_time: 54.102783203125
epoch time: 317.255615234375s, input_fired_time: 1.8927948474884, output_fired_time: 0, propagate_forward_time: 307.438110351563, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.92471313476563
validation[          , genome  6005] predictions:    6257/  10000 (62.57%), best:    6387/10000 (63.87%), error:     10842.77930, best error:     10578.03125 on epoch:    26, epoch:   28/50, mu: 0.8819180727, learning_rate: 0.0001196247, weight_decay: 0.0000815031

epoch time: 3687.1699218750s, input_fired_time: 35.6259193420, output_fired_time: 15.2940568924, propagate_forward_time: 1348.3989257813, propagate_backward_time: 2251.6042480469, weight_update_time: 1.6197896004, other_time: 36.2468261719
epoch time: 303.1300659180s, input_fired_time: 1.8421672583, output_fired_time: 0.0000000000, propagate_forward_time: 296.3345336914, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9533691406
validation[          , genome  6005] predictions:    6282/  10000 (62.82%), best:    6387/10000 (63.87%), error:     10786.18652, best error:     10578.03125 on epoch:    26, epoch:   29/50, mu: 0.8876083493, learning_rate: 0.0001097166, weight_decay: 0.0000757218

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.892999
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 0.000100629
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 7.03507e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 30
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 26
read number_validation_images: 10000
read best_validation_predictions: 6387
read best_validation_error: 10578
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '712445032'
read generator: 712445032
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3985.06616210938s, input_fired_time: 36.3893051147461, output_fired_time: 19.7687549591064, propagate_forward_time: 1477.447265625, propagate_backward_time: 2397.3173828125, weight_update_time: 0.756423830986023, other_time: 54.143310546875
epoch time: 294.1630859375s, input_fired_time: 3.31945514678955, output_fired_time: 0, propagate_forward_time: 283.755249023438, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.08837890625
validation[          , genome  6005] predictions:    6295/  10000 (62.95%), best:    6387/10000 (63.87%), error:     10717.80859, best error:     10578.03125 on epoch:    26, epoch:   30/50, mu: 0.8929990530, learning_rate: 0.0001006291, weight_decay: 0.0000703507

epoch time: 3939.7062988281s, input_fired_time: 43.0072021484, output_fired_time: 23.1008415222, propagate_forward_time: 1433.5762939453, propagate_backward_time: 2396.1577148438, weight_update_time: 0.8061634302, other_time: 43.8642578125
epoch time: 285.7254943848s, input_fired_time: 1.6923232079, output_fired_time: 0.0000000000, propagate_forward_time: 281.2142639160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8189086914
validation[          , genome  6005] predictions:    6404/  10000 (64.04%), best:    6404/10000 (64.04%), error:     10539.71777, best error:     10539.71777 on epoch:    31, epoch:   31/50, mu: 0.8981059194, learning_rate: 0.0000922944, weight_decay: 0.0000653606

epoch time: 3658.4980468750s, input_fired_time: 34.2456626892, output_fired_time: 17.3233337402, propagate_forward_time: 1363.9144287109, propagate_backward_time: 2206.1545410156, weight_update_time: 1.7760759592, other_time: 36.8601074219
epoch time: 279.3272705078s, input_fired_time: 1.7861523628, output_fired_time: 0.0000000000, propagate_forward_time: 274.6152648926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9258422852
validation[          , genome  6005] predictions:    6383/  10000 (63.83%), best:    6383/10000 (63.83%), error:     10357.99414, best error:     10357.99414 on epoch:    32, epoch:   32/50, mu: 0.9029439092, learning_rate: 0.0000846499, weight_decay: 0.0000607244

epoch time: 3831.0883789063s, input_fired_time: 39.5412750244, output_fired_time: 23.1773185730, propagate_forward_time: 1412.2476806641, propagate_backward_time: 2318.9199218750, weight_update_time: 0.7043061852, other_time: 37.2021484375
epoch time: 328.6730346680s, input_fired_time: 4.3519649506, output_fired_time: 0.0000000000, propagate_forward_time: 319.6899719238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6311035156
validation[          , genome  6005] predictions:    6377/  10000 (63.77%), best:    6383/10000 (63.83%), error:     10552.43164, best error:     10357.99414 on epoch:    32, epoch:   33/50, mu: 0.9075272083, learning_rate: 0.0000776387, weight_decay: 0.0000564170

epoch time: 3729.9770507813s, input_fired_time: 39.2978973389, output_fired_time: 15.5468978882, propagate_forward_time: 1384.0891113281, propagate_backward_time: 2259.2778320313, weight_update_time: 0.7028935552, other_time: 31.7653808594
epoch time: 287.2588806152s, input_fired_time: 2.7738809586, output_fired_time: 0.0000000000, propagate_forward_time: 278.0819396973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4030456543
validation[          , genome  6005] predictions:    6383/  10000 (63.83%), best:    6383/10000 (63.83%), error:     10457.79199, best error:     10357.99414 on epoch:    32, epoch:   34/50, mu: 0.9118692279, learning_rate: 0.0000712081, weight_decay: 0.0000524152

epoch time: 3762.5827636719s, input_fired_time: 40.0785598755, output_fired_time: 15.1537532806, propagate_forward_time: 1385.5697021484, propagate_backward_time: 2288.0405273438, weight_update_time: 0.6925685406, other_time: 33.7402343750
epoch time: 283.6690063477s, input_fired_time: 2.3423225880, output_fired_time: 0.0000000000, propagate_forward_time: 271.5473937988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.7792968750
validation[          , genome  6005] predictions:    6425/  10000 (64.25%), best:    6383/10000 (63.83%), error:     10484.34863, best error:     10357.99414 on epoch:    32, epoch:   35/50, mu: 0.9159826040, learning_rate: 0.0000653102, weight_decay: 0.0000486973

epoch time: 3858.9741210938s, input_fired_time: 40.5045204163, output_fired_time: 22.4537200928, propagate_forward_time: 1404.1706542969, propagate_backward_time: 2357.4675292969, weight_update_time: 0.7934470177, other_time: 34.3776855469
epoch time: 328.3532409668s, input_fired_time: 2.2810618877, output_fired_time: 0.0000000000, propagate_forward_time: 322.5606689453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5115051270
validation[          , genome  6005] predictions:    6448/  10000 (64.48%), best:    6383/10000 (63.83%), error:     10398.28223, best error:     10357.99414 on epoch:    32, epoch:   36/50, mu: 0.9198794365, learning_rate: 0.0000599008, weight_decay: 0.0000452431

epoch time: 3624.5183105469s, input_fired_time: 37.9933662415, output_fired_time: 19.7768287659, propagate_forward_time: 1355.5037841797, propagate_backward_time: 2185.8994140625, weight_update_time: 0.6200870872, other_time: 25.3447265625
epoch time: 278.0349426270s, input_fired_time: 3.6161630154, output_fired_time: 0.0000000000, propagate_forward_time: 270.5250244141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8937683105
validation[          , genome  6005] predictions:    6397/  10000 (63.97%), best:    6383/10000 (63.83%), error:     10539.86719, best error:     10357.99414 on epoch:    32, epoch:   37/50, mu: 0.9235711098, learning_rate: 0.0000549394, weight_decay: 0.0000420339

epoch time: 3793.1958007813s, input_fired_time: 46.7571830750, output_fired_time: 19.5301647186, propagate_forward_time: 1394.6910400391, propagate_backward_time: 2306.1296386719, weight_update_time: 0.6392145157, other_time: 26.0876464844
epoch time: 299.2912597656s, input_fired_time: 3.8728070259, output_fired_time: 0.0000000000, propagate_forward_time: 290.3334045410, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0850524902
validation[          , genome  6005] predictions:    6414/  10000 (64.14%), best:    6383/10000 (63.83%), error:     10408.09961, best error:     10357.99414 on epoch:    32, epoch:   38/50, mu: 0.9270684123, learning_rate: 0.0000503890, weight_decay: 0.0000390523

epoch time: 3754.9626464844s, input_fired_time: 34.5549278259, output_fired_time: 20.3031234741, propagate_forward_time: 1411.2462158203, propagate_backward_time: 2257.8305664063, weight_update_time: 1.7064515352, other_time: 31.0278320313
epoch time: 293.7002868652s, input_fired_time: 3.9725260735, output_fired_time: 0.0000000000, propagate_forward_time: 282.2644653320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4632873535
validation[          , genome  6005] predictions:    6442/  10000 (64.42%), best:    6442/10000 (64.42%), error:     10303.04004, best error:     10303.04004 on epoch:    39, epoch:   39/50, mu: 0.9303815961, learning_rate: 0.0000462154, weight_decay: 0.0000362822

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.93352
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 4.23876e-05
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 3.37086e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 40
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 39
read number_validation_images: 10000
read best_validation_predictions: 6442
read best_validation_error: 10303
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '1198340958'
read generator: 1198340958
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3777.74145507813s, input_fired_time: 36.6858444213867, output_fired_time: 17.4787845611572, propagate_forward_time: 1431.58264160156, propagate_backward_time: 2245.4453125, weight_update_time: 1.67916798591614, other_time: 46.548828125
epoch time: 309.362182617188s, input_fired_time: 2.96666526794434, output_fired_time: 0, propagate_forward_time: 300.523040771484, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.87246704101563
validation[          , genome  6005] predictions:    6495/  10000 (64.95%), best:    6495/10000 (64.95%), error:     10235.52246, best error:     10235.52246 on epoch:    40, epoch:   40/50, mu: 0.9335203767, learning_rate: 0.0000423876, weight_decay: 0.0000337086

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.936494
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 3.88767e-05
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 3.13176e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 41
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 40
read number_validation_images: 10000
read best_validation_predictions: 6495
read best_validation_error: 10235.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '1850185648'
read generator: 1850185648
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 3875.99291992188s, input_fired_time: 35.0101737976074, output_fired_time: 19.9665641784668, propagate_forward_time: 1453.47021484375, propagate_backward_time: 2317.14697265625, weight_update_time: 1.70885920524597, other_time: 50.39892578125
epoch time: 280.291900634766s, input_fired_time: 1.72549617290497, output_fired_time: 0, propagate_forward_time: 271.706817626953, propagate_backward_time: 0, weight_update_time: 0, other_time: 6.85958862304688
validation[          , genome  6005] predictions:    6529/  10000 (65.29%), best:    6529/10000 (65.29%), error:     10199.36426, best error:     10199.36426 on epoch:    41, epoch:   41/50, mu: 0.9364938736, learning_rate: 0.0000388767, weight_decay: 0.0000313176

epoch time: 3768.6474609375s, input_fired_time: 43.7764625549, output_fired_time: 21.8923931122, propagate_forward_time: 1412.8504638672, propagate_backward_time: 2257.1799316406, weight_update_time: 1.6627532244, other_time: 32.9484863281
epoch time: 298.7977294922s, input_fired_time: 1.9268474579, output_fired_time: 0.0000000000, propagate_forward_time: 291.5642089844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3066711426
validation[          , genome  6005] predictions:    6509/  10000 (65.09%), best:    6509/10000 (65.09%), error:     10165.23926, best error:     10165.23926 on epoch:    42, epoch:   42/50, mu: 0.9393108487, learning_rate: 0.0000356567, weight_decay: 0.0000290962

epoch time: 3889.3549804688s, input_fired_time: 46.9626274109, output_fired_time: 21.5645046234, propagate_forward_time: 1444.8856201172, propagate_backward_time: 2343.5541992188, weight_update_time: 0.7411549687, other_time: 32.3881835938
epoch time: 288.8554382324s, input_fired_time: 3.8186905384, output_fired_time: 0.0000000000, propagate_forward_time: 282.0929260254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9438171387
validation[          , genome  6005] predictions:    6494/  10000 (64.94%), best:    6509/10000 (65.09%), error:     10234.15039, best error:     10165.23926 on epoch:    42, epoch:   43/50, mu: 0.9419795275, learning_rate: 0.0000327034, weight_decay: 0.0000270323

epoch time: 3743.5625000000s, input_fired_time: 36.6376800537, output_fired_time: 15.2184295654, propagate_forward_time: 1416.7062988281, propagate_backward_time: 2245.8125000000, weight_update_time: 0.6705226302, other_time: 29.1875000000
epoch time: 316.2462463379s, input_fired_time: 4.0248880386, output_fired_time: 0.0000000000, propagate_forward_time: 307.1077575684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1135864258
validation[          , genome  6005] predictions:    6472/  10000 (64.72%), best:    6509/10000 (65.09%), error:     10234.53613, best error:     10165.23926 on epoch:    42, epoch:   44/50, mu: 0.9445077181, learning_rate: 0.0000299947, weight_decay: 0.0000251148

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.946903
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 2.75103e-05
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 2.33334e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 45
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 42
read number_validation_images: 10000
read best_validation_predictions: 6509
read best_validation_error: 10165.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '1957160200'
read generator: 1957160200
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_178_6005_2_r1254987904_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498606
read mu: 0.946903
read mu_delta: 0.947352
read initial_learning_rate: 0.00134642
read learning_rate: 2.75103e-05
read learning_rate_delta: 0.917173
read initial_weight_decay: 0.000639515
read weight_decay: 2.33334e-05
read weight_decay_delta: 0.929067
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0779941
read input_dropout_probability: 0.00118842
read hidden_dropout_probability: 0.110803
read velocity_reset: 2149
read epoch: 45
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 42
read number_validation_images: 10000
read best_validation_predictions: 6509
read best_validation_error: 10165.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6005
read normal distribution: '1 0 0'
generator_str: '1957160200'
read generator: 1957160200
read generated_by_map:
1 disable_node 1
reading 89 nodes.
reading 454 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4082.65502929688s, input_fired_time: 45.2085914611816, output_fired_time: 22.6806926727295, propagate_forward_time: 1481.35485839844, propagate_backward_time: 2484.64331054688, weight_update_time: 1.78096520900726, other_time: 48.767822265625
epoch time: 316.062622070313s, input_fired_time: 3.35712432861328, output_fired_time: 0, propagate_forward_time: 306.751129150391, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.95437622070313
validation[          , genome  6005] predictions:    6471/  10000 (64.71%), best:    6509/10000 (65.09%), error:     10331.63574, best error:     10165.23926 on epoch:    42, epoch:   45/50, mu: 0.9469027519, learning_rate: 0.0000275103, weight_decay: 0.0000233334

epoch time: 3725.8020019531s, input_fired_time: 36.4775428772, output_fired_time: 16.0776615143, propagate_forward_time: 1394.2359619141, propagate_backward_time: 2244.2106933594, weight_update_time: 1.6725430489, other_time: 34.8000488281
epoch time: 297.0134582520s, input_fired_time: 3.8431382179, output_fired_time: 0.0000000000, propagate_forward_time: 287.3567199707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8135986328
validation[          , genome  6005] predictions:    6545/  10000 (65.45%), best:    6545/10000 (65.45%), error:     10131.50977, best error:     10131.50977 on epoch:    46, epoch:   46/50, mu: 0.9491717219, learning_rate: 0.0000252317, weight_decay: 0.0000216783

epoch time: 3939.0927734375s, input_fired_time: 41.1157569885, output_fired_time: 20.9644832611, propagate_forward_time: 1450.9143066406, propagate_backward_time: 2391.2297363281, weight_update_time: 1.8382971287, other_time: 34.8684082031
epoch time: 362.6879577637s, input_fired_time: 4.8662414551, output_fired_time: 0.0000000000, propagate_forward_time: 347.3546142578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.4671020508
validation[          , genome  6005] predictions:    6526/  10000 (65.26%), best:    6526/10000 (65.26%), error:     10116.05078, best error:     10116.05078 on epoch:    47, epoch:   47/50, mu: 0.9513212442, learning_rate: 0.0000231419, weight_decay: 0.0000201406

epoch time: 4087.9206542969s, input_fired_time: 50.0798301697, output_fired_time: 22.9243526459, propagate_forward_time: 1490.0041503906, propagate_backward_time: 2482.4936523438, weight_update_time: 0.8565530777, other_time: 42.4187011719
epoch time: 275.2013244629s, input_fired_time: 5.8313465118, output_fired_time: 0.0000000000, propagate_forward_time: 266.1245422363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2454223633
validation[          , genome  6005] predictions:    6472/  10000 (64.72%), best:    6526/10000 (65.26%), error:     10170.96777, best error:     10116.05078 on epoch:    47, epoch:   48/50, mu: 0.9533575773, learning_rate: 0.0000212251, weight_decay: 0.0000187120

epoch time: 3897.3173828125s, input_fired_time: 42.3962173462, output_fired_time: 17.0010776520, propagate_forward_time: 1425.1783447266, propagate_backward_time: 2377.9135742188, weight_update_time: 1.7528892756, other_time: 34.8281250000
epoch time: 293.1766967773s, input_fired_time: 3.3824665546, output_fired_time: 0.0000000000, propagate_forward_time: 283.4430541992, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3511657715
validation[          , genome  6005] predictions:    6512/  10000 (65.12%), best:    6526/10000 (65.26%), error:     10188.26465, best error:     10116.05078 on epoch:    47, epoch:   49/50, mu: 0.9552867413, learning_rate: 0.0000194671, weight_decay: 0.0000173847

epoch time: 3828.3686523438s, input_fired_time: 49.4529838562, output_fired_time: 23.9551544189, propagate_forward_time: 1371.5343017578, propagate_backward_time: 2344.6000976563, weight_update_time: 1.8559912443, other_time: 38.8261718750
epoch time: 284.0843505859s, input_fired_time: 2.2609138489, output_fired_time: 0.0000000000, propagate_forward_time: 276.1101989746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7132263184
validation[          , genome  6005] predictions:    6565/  10000 (65.65%), best:    6565/10000 (65.65%), error:     10087.29883, best error:     10087.29883 on epoch:    50, epoch:   50/50, mu: 0.9571143389, learning_rate: 0.0000178547, weight_decay: 0.0000161515

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1371.6696777344s, input_fired_time: 26.4046916962, output_fired_time: 0.0000000000, propagate_forward_time: 1313.5274658203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 31.7375488281
best training[          , genome  6005] predictions:   35831/  50000 (71.66%), best:    6565/10000 (65.65%), error:     43308.74609, best error:     10087.29883 on epoch:    50, epoch:   51/50, mu: 0.9588456750, learning_rate: 0.0000163759, weight_decay: 0.0000150059
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 318.6787414551s, input_fired_time: 4.2671394348, output_fired_time: 0.0000000000, propagate_forward_time: 303.5784606934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.8331298828
   testing[          , genome  6005] predictions:    6565/  10000 (65.65%), best:    6565/10000 (65.65%), error:     10087.29883, best error:     10087.29883 on epoch:    50, epoch:   51/50, mu: 0.9588456750, learning_rate: 0.0000163759, weight_decay: 0.0000150059
backpropagation finished successfully!
00:03:01 (9152): called boinc_finish(0)

</stderr_txt>
]]>