Nameexact_genome_1542127803_186_5817_1
Workunit4086605
Created13 Nov 2018, 17:48:41 UTC
Sent13 Nov 2018, 18:01:54 UTC
Report deadline23 Feb 2019, 20:36:04 UTC
Received16 Nov 2018, 1:51:57 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID10687
Run time1 days 16 hours 23 min 43 sec
CPU time1 days 15 hours 37 min 21 sec
Validate stateValid
Credit25,526.77
Device peak FLOPS4.48 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size538.80 MB
Peak swap size537.03 MB
Peak disk usage11.47 MB

Stderr output

<core_client_version>7.4.42</core_client_version>
<![CDATA[
<stderr_txt>
me: 0, weight_update_time: 0, other_time: 2.04408264160156
validation[          , genome  5817] predictions:    6326/  10000 (63.26%), best:    6375/10000 (63.75%), error:     10617.23535, best error:     10606.53809 on epoch:    52, epoch:   55/100, mu: 0.9616910219, learning_rate: 0.0000285793, weight_decay: 0.0000308985

epoch time: 1353.1047363281s, input_fired_time: 22.3991355896, output_fired_time: 7.8558568954, propagate_forward_time: 501.1977844238, propagate_backward_time: 808.5334472656, weight_update_time: 0.4511747062, other_time: 13.1185302734
epoch time: 107.0447006226s, input_fired_time: 1.3496801853, output_fired_time: 0.0000000000, propagate_forward_time: 103.4437637329, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2512588501
validation[          , genome  5817] predictions:    6356/  10000 (63.56%), best:    6375/10000 (63.75%), error:     10609.50586, best error:     10606.53809 on epoch:    52, epoch:   56/100, mu: 0.9631041884, learning_rate: 0.0000267366, weight_decay: 0.0000292896

epoch time: 1360.0211181641s, input_fired_time: 22.5141239166, output_fired_time: 7.9050221443, propagate_forward_time: 504.3107910156, propagate_backward_time: 811.7915039063, weight_update_time: 0.4947359860, other_time: 13.4996337891
epoch time: 109.3403778076s, input_fired_time: 1.4021680355, output_fired_time: 0.0000000000, propagate_forward_time: 105.7658386230, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1723709106
validation[          , genome  5817] predictions:    6347/  10000 (63.47%), best:    6347/10000 (63.47%), error:     10532.35449, best error:     10532.35449 on epoch:    57, epoch:   57/100, mu: 0.9644468427, learning_rate: 0.0000250127, weight_decay: 0.0000277644

epoch time: 1399.7736816406s, input_fired_time: 23.1912593842, output_fired_time: 9.1053247452, propagate_forward_time: 517.2739868164, propagate_backward_time: 836.4312133789, weight_update_time: 0.4583610892, other_time: 13.7718505859
epoch time: 105.3668594360s, input_fired_time: 1.3579704762, output_fired_time: 0.0000000000, propagate_forward_time: 101.8007659912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2081222534
validation[          , genome  5817] predictions:    6325/  10000 (63.25%), best:    6347/10000 (63.47%), error:     10593.99902, best error:     10532.35449 on epoch:    57, epoch:   58/100, mu: 0.9657224417, learning_rate: 0.0000233999, weight_decay: 0.0000263187

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.966934
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 2.18912e-05
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 2.49483e-05
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 59
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 57
read number_validation_images: 10000
read best_validation_predictions: 6347
read best_validation_error: 10532.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '1653972712'
read generator: 1653972712
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1342.80688476563s, input_fired_time: 22.1910953521729, output_fired_time: 7.71398591995239, propagate_forward_time: 500.800201416016, propagate_backward_time: 798.788146972656, weight_update_time: 0.444644182920456, other_time: 13.3135375976563
epoch time: 102.4326171875s, input_fired_time: 1.31915581226349, output_fired_time: 0, propagate_forward_time: 98.9806671142578, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.13279724121094
validation[          , genome  5817] predictions:    6338/  10000 (63.38%), best:    6347/10000 (63.47%), error:     10591.46777, best error:     10532.35449 on epoch:    57, epoch:   59/100, mu: 0.9669343829, learning_rate: 0.0000218912, weight_decay: 0.0000249483

epoch time: 1353.7365722656s, input_fired_time: 22.3705730438, output_fired_time: 7.8310871124, propagate_forward_time: 503.7717895508, propagate_backward_time: 806.4424438477, weight_update_time: 0.4479567707, other_time: 13.3206787109
epoch time: 103.1399612427s, input_fired_time: 1.3398998976, output_fired_time: 0.0000000000, propagate_forward_time: 99.6308441162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1692199707
validation[          , genome  5817] predictions:    6356/  10000 (63.56%), best:    6356/10000 (63.56%), error:     10523.55566, best error:     10523.55566 on epoch:    60, epoch:   60/100, mu: 0.9680858254, learning_rate: 0.0000204797, weight_decay: 0.0000236492

epoch time: 1330.4227294922s, input_fired_time: 22.1365814209, output_fired_time: 7.7314348221, propagate_forward_time: 493.3096008301, propagate_backward_time: 794.1112670898, weight_update_time: 0.4423854351, other_time: 13.1338500977
epoch time: 102.5200500488s, input_fired_time: 1.3215360641, output_fired_time: 0.0000000000, propagate_forward_time: 99.0665130615, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1320037842
validation[          , genome  5817] predictions:    6329/  10000 (63.29%), best:    6356/10000 (63.56%), error:     10677.45313, best error:     10523.55566 on epoch:    60, epoch:   61/100, mu: 0.9691797495, learning_rate: 0.0000191592, weight_decay: 0.0000224178

epoch time: 1372.4462890625s, input_fired_time: 22.7566871643, output_fired_time: 7.8760738373, propagate_forward_time: 509.4120788574, propagate_backward_time: 818.9957275391, weight_update_time: 0.4519051909, other_time: 13.4056396484
epoch time: 102.7145843506s, input_fired_time: 1.3231781721, output_fired_time: 0.0000000000, propagate_forward_time: 99.2768783569, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1145248413
validation[          , genome  5817] predictions:    6388/  10000 (63.88%), best:    6356/10000 (63.56%), error:     10523.80176, best error:     10523.55566 on epoch:    60, epoch:   62/100, mu: 0.9702190757, learning_rate: 0.0000179238, weight_decay: 0.0000212504

epoch time: 1329.8048095703s, input_fired_time: 22.1712074280, output_fired_time: 7.8147153854, propagate_forward_time: 492.8745727539, propagate_backward_time: 793.7644042969, weight_update_time: 0.4437505901, other_time: 13.1798706055
epoch time: 102.2288970947s, input_fired_time: 1.3203270435, output_fired_time: 0.0000000000, propagate_forward_time: 98.7946548462, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1139144897
validation[          , genome  5817] predictions:    6337/  10000 (63.37%), best:    6356/10000 (63.56%), error:     10588.66602, best error:     10523.55566 on epoch:    60, epoch:   63/100, mu: 0.9712065458, learning_rate: 0.0000167682, weight_decay: 0.0000201439

epoch time: 1330.0837402344s, input_fired_time: 22.0784778595, output_fired_time: 7.7667417526, propagate_forward_time: 493.1388244629, propagate_backward_time: 793.9484863281, weight_update_time: 0.4460706115, other_time: 13.1512451172
epoch time: 102.3001327515s, input_fired_time: 1.3255801201, output_fired_time: 0.0000000000, propagate_forward_time: 98.8550415039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1195144653
validation[          , genome  5817] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10500.22754, best error:     10500.22754 on epoch:    64, epoch:   64/100, mu: 0.9721447229, learning_rate: 0.0000156870, weight_decay: 0.0000190950

epoch time: 1344.9584960938s, input_fired_time: 22.3924140930, output_fired_time: 7.7790360451, propagate_forward_time: 498.4337158203, propagate_backward_time: 803.0388183594, weight_update_time: 0.4456864297, other_time: 13.3144531250
epoch time: 102.2437133789s, input_fired_time: 1.3314287663, output_fired_time: 0.0000000000, propagate_forward_time: 98.8342666626, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0780181885
validation[          , genome  5817] predictions:    6366/  10000 (63.66%), best:    6366/10000 (63.66%), error:     10490.84863, best error:     10490.84863 on epoch:    65, epoch:   65/100, mu: 0.9730360508, learning_rate: 0.0000146755, weight_decay: 0.0000181007

epoch time: 1314.4346923828s, input_fired_time: 21.9291133881, output_fired_time: 7.6369018555, propagate_forward_time: 487.8938293457, propagate_backward_time: 783.9012451172, weight_update_time: 0.4388580322, other_time: 13.0736083984
epoch time: 100.1831512451s, input_fired_time: 1.2970559597, output_fired_time: 0.0000000000, propagate_forward_time: 96.7837829590, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1023101807
validation[          , genome  5817] predictions:    6272/  10000 (62.72%), best:    6366/10000 (63.66%), error:     10647.50879, best error:     10490.84863 on epoch:    65, epoch:   66/100, mu: 0.9738828540, learning_rate: 0.0000137293, weight_decay: 0.0000171582

epoch time: 1302.9775390625s, input_fired_time: 21.5950813293, output_fired_time: 7.6346569061, propagate_forward_time: 480.0447998047, propagate_backward_time: 781.1885375977, weight_update_time: 0.4351091981, other_time: 12.5144653320
epoch time: 100.0509796143s, input_fired_time: 1.2848799229, output_fired_time: 0.0000000000, propagate_forward_time: 96.6957855225, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0703125000
validation[          , genome  5817] predictions:    6381/  10000 (63.81%), best:    6366/10000 (63.66%), error:     10492.78223, best error:     10490.84863 on epoch:    65, epoch:   67/100, mu: 0.9746873975, learning_rate: 0.0000128440, weight_decay: 0.0000162647

epoch time: 1329.0030517578s, input_fired_time: 22.0341243744, output_fired_time: 7.7040200233, propagate_forward_time: 490.9329833984, propagate_backward_time: 795.0874023438, weight_update_time: 0.4462110996, other_time: 13.2445068359
epoch time: 100.3419036865s, input_fired_time: 1.3006695509, output_fired_time: 0.0000000000, propagate_forward_time: 96.9592971802, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0819396973
validation[          , genome  5817] predictions:    6352/  10000 (63.52%), best:    6366/10000 (63.66%), error:     10560.69922, best error:     10490.84863 on epoch:    65, epoch:   68/100, mu: 0.9754517674, learning_rate: 0.0000120159, weight_decay: 0.0000154178

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.976178
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 1.12411e-05
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 1.4615e-05
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 69
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 65
read number_validation_images: 10000
read best_validation_predictions: 6366
read best_validation_error: 10490.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '1310917146'
read generator: 1310917146
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1456.37182617188s, input_fired_time: 24.2339286804199, output_fired_time: 8.26261138916016, propagate_forward_time: 527.705383300781, propagate_backward_time: 882.176147460938, weight_update_time: 0.458997815847397, other_time: 13.9938354492188
epoch time: 152.837844848633s, input_fired_time: 1.48970413208008, output_fired_time: 0, propagate_forward_time: 148.76496887207, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.58317565917969
validation[          , genome  5817] predictions:    6356/  10000 (63.56%), best:    6366/10000 (63.66%), error:     10502.37891, best error:     10490.84863 on epoch:    65, epoch:   69/100, mu: 0.9761779904, learning_rate: 0.0000112411, weight_decay: 0.0000146150

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.976868
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 1.05163e-05
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 1.3854e-05
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 70
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 65
read number_validation_images: 10000
read best_validation_predictions: 6366
read best_validation_error: 10490.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '2033658337'
read generator: 2033658337
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.976868
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 1.05163e-05
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 1.3854e-05
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 70
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 65
read number_validation_images: 10000
read best_validation_predictions: 6366
read best_validation_error: 10490.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '2033658337'
read generator: 2033658337
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.976868
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 1.05163e-05
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 1.3854e-05
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 70
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 65
read number_validation_images: 10000
read best_validation_predictions: 6366
read best_validation_error: 10490.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '2033658337'
read generator: 2033658337
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1298.1044921875s, input_fired_time: 21.5380897521973, output_fired_time: 7.58211851119995, propagate_forward_time: 475.010955810547, propagate_backward_time: 781.388488769531, weight_update_time: 0.427078515291214, other_time: 12.5847778320313
epoch time: 101.28491973877s, input_fired_time: 1.33163011074066, output_fired_time: 0, propagate_forward_time: 97.8703918457031, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.08290100097656
validation[          , genome  5817] predictions:    6357/  10000 (63.57%), best:    6357/10000 (63.57%), error:     10474.29297, best error:     10474.29297 on epoch:    70, epoch:   70/100, mu: 0.9768679738, learning_rate: 0.0000105163, weight_decay: 0.0000138540

epoch time: 1296.2440185547s, input_fired_time: 21.7804756165, output_fired_time: 7.5452632904, propagate_forward_time: 479.9972839355, propagate_backward_time: 774.0817260742, weight_update_time: 0.4363666177, other_time: 12.8391723633
epoch time: 99.0021209717s, input_fired_time: 1.2959973812, output_fired_time: 0.0000000000, propagate_forward_time: 95.6456146240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0605087280
validation[          , genome  5817] predictions:    6391/  10000 (63.91%), best:    6391/10000 (63.91%), error:     10377.44238, best error:     10377.44238 on epoch:    71, epoch:   71/100, mu: 0.9775235057, learning_rate: 0.0000098383, weight_decay: 0.0000131326

epoch time: 1300.9444580078s, input_fired_time: 21.6195125580, output_fired_time: 7.5898447037, propagate_forward_time: 481.0023803711, propagate_backward_time: 778.0720825195, weight_update_time: 0.4344241619, other_time: 12.6606445313
epoch time: 100.1392669678s, input_fired_time: 1.3205274343, output_fired_time: 0.0000000000, propagate_forward_time: 96.7593536377, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0593872070
validation[          , genome  5817] predictions:    6366/  10000 (63.66%), best:    6391/10000 (63.91%), error:     10455.47070, best error:     10377.44238 on epoch:    71, epoch:   72/100, mu: 0.9781463146, learning_rate: 0.0000092039, weight_decay: 0.0000124488

epoch time: 1306.0214843750s, input_fired_time: 21.6771411896, output_fired_time: 7.7091507912, propagate_forward_time: 480.3732604980, propagate_backward_time: 783.7655029297, weight_update_time: 0.4363395274, other_time: 12.4964599609
epoch time: 100.1399383545s, input_fired_time: 1.3282204866, output_fired_time: 0.0000000000, propagate_forward_time: 96.7778167725, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0338973999
validation[          , genome  5817] predictions:    6398/  10000 (63.98%), best:    6391/10000 (63.91%), error:     10467.96191, best error:     10377.44238 on epoch:    71, epoch:   73/100, mu: 0.9787380695, learning_rate: 0.0000086105, weight_decay: 0.0000118006

epoch time: 1324.4897460938s, input_fired_time: 22.0847892761, output_fired_time: 7.7706322670, propagate_forward_time: 492.6670227051, propagate_backward_time: 789.1865234375, weight_update_time: 0.4351202846, other_time: 12.7807617188
epoch time: 100.5522079468s, input_fired_time: 1.3437345028, output_fired_time: 0.0000000000, propagate_forward_time: 97.1902542114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0182189941
validation[          , genome  5817] predictions:    6397/  10000 (63.97%), best:    6391/10000 (63.91%), error:     10390.36328, best error:     10377.44238 on epoch:    71, epoch:   74/100, mu: 0.9793002605, learning_rate: 0.0000080553, weight_decay: 0.0000111861

epoch time: 1330.1141357422s, input_fired_time: 21.6900424957, output_fired_time: 7.6451931000, propagate_forward_time: 483.1543884277, propagate_backward_time: 804.7592163086, weight_update_time: 0.4875579476, other_time: 12.8652954102
epoch time: 100.4667739868s, input_fired_time: 1.3147126436, output_fired_time: 0.0000000000, propagate_forward_time: 97.1372375488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0148239136
validation[          , genome  5817] predictions:    6402/  10000 (64.02%), best:    6391/10000 (63.91%), error:     10428.96191, best error:     10377.44238 on epoch:    71, epoch:   75/100, mu: 0.9798343778, learning_rate: 0.0000075359, weight_decay: 0.0000106036

epoch time: 1327.8874511719s, input_fired_time: 22.0418148041, output_fired_time: 7.6210799217, propagate_forward_time: 496.9854736328, propagate_backward_time: 788.1335449219, weight_update_time: 0.4364162683, other_time: 13.1054687500
epoch time: 100.7977752686s, input_fired_time: 1.3221387863, output_fired_time: 0.0000000000, propagate_forward_time: 97.4639663696, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0116729736
validation[          , genome  5817] predictions:    6416/  10000 (64.16%), best:    6391/10000 (63.91%), error:     10404.10742, best error:     10377.44238 on epoch:    71, epoch:   76/100, mu: 0.9803418517, learning_rate: 0.0000070500, weight_decay: 0.0000100515

epoch time: 1312.5988769531s, input_fired_time: 21.8584194183, output_fired_time: 7.7906126976, propagate_forward_time: 489.1221008301, propagate_backward_time: 780.9624633789, weight_update_time: 0.4369626343, other_time: 12.8652954102
epoch time: 100.8541717529s, input_fired_time: 1.3342609406, output_fired_time: 0.0000000000, propagate_forward_time: 97.3720016479, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1479110718
validation[          , genome  5817] predictions:    6439/  10000 (64.39%), best:    6439/10000 (64.39%), error:     10347.94238, best error:     10347.94238 on epoch:    77, epoch:   77/100, mu: 0.9808239937, learning_rate: 0.0000065954, weight_decay: 0.0000095281

epoch time: 1348.2077636719s, input_fired_time: 22.4563560486, output_fired_time: 7.9414644241, propagate_forward_time: 492.9476013184, propagate_backward_time: 811.8666381836, weight_update_time: 0.4609810710, other_time: 12.9957885742
epoch time: 102.3997344971s, input_fired_time: 1.3627123833, output_fired_time: 0.0000000000, propagate_forward_time: 98.9935379028, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0434875488
validation[          , genome  5817] predictions:    6401/  10000 (64.01%), best:    6439/10000 (64.39%), error:     10418.13086, best error:     10347.94238 on epoch:    77, epoch:   78/100, mu: 0.9812820554, learning_rate: 0.0000061702, weight_decay: 0.0000090320

epoch time: 1318.5225830078s, input_fired_time: 21.8862762451, output_fired_time: 7.7849221230, propagate_forward_time: 487.9527587891, propagate_backward_time: 788.3223876953, weight_update_time: 0.4382173419, other_time: 12.5762939453
epoch time: 102.0774002075s, input_fired_time: 1.3622095585, output_fired_time: 0.0000000000, propagate_forward_time: 98.6074676514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1077194214
validation[          , genome  5817] predictions:    6388/  10000 (63.88%), best:    6439/10000 (64.39%), error:     10395.65137, best error:     10347.94238 on epoch:    77, epoch:   79/100, mu: 0.9817172289, learning_rate: 0.0000057723, weight_decay: 0.0000085617

epoch time: 1343.5004882813s, input_fired_time: 21.9335727692, output_fired_time: 7.7051496506, propagate_forward_time: 485.9321899414, propagate_backward_time: 815.1199340820, weight_update_time: 0.4408260882, other_time: 12.8095703125
epoch time: 101.9795837402s, input_fired_time: 1.3454256058, output_fired_time: 0.0000000000, propagate_forward_time: 98.6084594727, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0256958008
validation[          , genome  5817] predictions:    6389/  10000 (63.89%), best:    6439/10000 (64.39%), error:     10421.44922, best error:     10347.94238 on epoch:    77, epoch:   80/100, mu: 0.9821307063, learning_rate: 0.0000054001, weight_decay: 0.0000081158

epoch time: 1339.0766601563s, input_fired_time: 22.3225078583, output_fired_time: 7.9340152740, propagate_forward_time: 492.9182434082, propagate_backward_time: 802.8395385742, weight_update_time: 0.4434237182, other_time: 13.0624389648
epoch time: 101.5320129395s, input_fired_time: 1.3341565132, output_fired_time: 0.0000000000, propagate_forward_time: 98.0870819092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1107711792
validation[          , genome  5817] predictions:    6413/  10000 (64.13%), best:    6439/10000 (64.39%), error:     10381.85742, best error:     10347.94238 on epoch:    77, epoch:   81/100, mu: 0.9825235605, learning_rate: 0.0000050520, weight_decay: 0.0000076932

epoch time: 1350.9764404297s, input_fired_time: 22.5996189117, output_fired_time: 7.7059102058, propagate_forward_time: 500.9784851074, propagate_backward_time: 806.6198730469, weight_update_time: 0.4437778890, other_time: 13.0725097656
epoch time: 104.6059494019s, input_fired_time: 1.3840215206, output_fired_time: 0.0000000000, propagate_forward_time: 101.1796875000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0422439575
validation[          , genome  5817] predictions:    6458/  10000 (64.58%), best:    6458/10000 (64.58%), error:     10327.17676, best error:     10327.17676 on epoch:    82, epoch:   82/100, mu: 0.9828968048, learning_rate: 0.0000047262, weight_decay: 0.0000072926

epoch time: 1334.7784423828s, input_fired_time: 22.1824722290, output_fired_time: 7.8150482178, propagate_forward_time: 491.6964416504, propagate_backward_time: 800.0789794922, weight_update_time: 0.4448463917, other_time: 13.0054931641
epoch time: 104.9032440186s, input_fired_time: 1.3954209089, output_fired_time: 0.0000000000, propagate_forward_time: 101.3117218018, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1960983276
validation[          , genome  5817] predictions:    6427/  10000 (64.27%), best:    6427/10000 (64.27%), error:     10326.24414, best error:     10326.24414 on epoch:    83, epoch:   83/100, mu: 0.9832513928, learning_rate: 0.0000044215, weight_decay: 0.0000069129

epoch time: 1348.7204589844s, input_fired_time: 22.3884391785, output_fired_time: 8.5800857544, propagate_forward_time: 498.6455688477, propagate_backward_time: 805.6743164063, weight_update_time: 0.4449706376, other_time: 13.4320678711
epoch time: 104.1806259155s, input_fired_time: 1.3731279373, output_fired_time: 0.0000000000, propagate_forward_time: 100.6747360229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1327590942
validation[          , genome  5817] predictions:    6464/  10000 (64.64%), best:    6464/10000 (64.64%), error:     10323.52930, best error:     10323.52930 on epoch:    84, epoch:   84/100, mu: 0.9835882783, learning_rate: 0.0000041364, weight_decay: 0.0000065529

epoch time: 1352.0097656250s, input_fired_time: 22.4659290314, output_fired_time: 7.9583959579, propagate_forward_time: 502.4939575195, propagate_backward_time: 805.8884277344, weight_update_time: 0.4492308795, other_time: 13.2030639648
epoch time: 104.7580642700s, input_fired_time: 1.3734171391, output_fired_time: 0.0000000000, propagate_forward_time: 101.2970428467, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0876007080
validation[          , genome  5817] predictions:    6445/  10000 (64.45%), best:    6464/10000 (64.64%), error:     10343.52832, best error:     10323.52930 on epoch:    84, epoch:   85/100, mu: 0.9839083552, learning_rate: 0.0000038697, weight_decay: 0.0000062117

epoch time: 1332.8863525391s, input_fired_time: 22.2063713074, output_fired_time: 7.7529139519, propagate_forward_time: 493.9814453125, propagate_backward_time: 795.7663574219, weight_update_time: 0.4479273558, other_time: 13.1791992188
epoch time: 103.0060806274s, input_fired_time: 1.3569221497, output_fired_time: 0.0000000000, propagate_forward_time: 99.5398788452, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1092758179
validation[          , genome  5817] predictions:    6380/  10000 (63.80%), best:    6464/10000 (64.64%), error:     10425.60156, best error:     10323.52930 on epoch:    84, epoch:   86/100, mu: 0.9842124581, learning_rate: 0.0000036202, weight_decay: 0.0000058883

epoch time: 1354.3668212891s, input_fired_time: 22.6036987305, output_fired_time: 7.9040951729, propagate_forward_time: 500.0042419434, propagate_backward_time: 810.7592773438, weight_update_time: 0.4507903755, other_time: 13.0955810547
epoch time: 104.1748123169s, input_fired_time: 1.3626642227, output_fired_time: 0.0000000000, propagate_forward_time: 100.6936874390, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1184616089
validation[          , genome  5817] predictions:    6394/  10000 (63.94%), best:    6464/10000 (64.64%), error:     10387.97363, best error:     10323.52930 on epoch:    84, epoch:   87/100, mu: 0.9845013618, learning_rate: 0.0000033868, weight_decay: 0.0000055817

epoch time: 1356.0168457031s, input_fired_time: 22.5198993683, output_fired_time: 7.9615316391, propagate_forward_time: 499.1611022949, propagate_backward_time: 813.5583496094, weight_update_time: 0.4549083710, other_time: 12.8159179688
epoch time: 104.0080184937s, input_fired_time: 1.3747471571, output_fired_time: 0.0000000000, propagate_forward_time: 100.5490188599, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0842514038
validation[          , genome  5817] predictions:    6445/  10000 (64.45%), best:    6464/10000 (64.64%), error:     10338.33887, best error:     10323.52930 on epoch:    84, epoch:   88/100, mu: 0.9847758412, learning_rate: 0.0000031684, weight_decay: 0.0000052910

epoch time: 1356.0745849609s, input_fired_time: 22.7017078400, output_fired_time: 7.9327912331, propagate_forward_time: 499.7847900391, propagate_backward_time: 812.6166381836, weight_update_time: 0.4546762109, other_time: 13.0387573242
epoch time: 104.5995330811s, input_fired_time: 1.3573479652, output_fired_time: 0.0000000000, propagate_forward_time: 101.1058502197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1363372803
validation[          , genome  5817] predictions:    6411/  10000 (64.11%), best:    6464/10000 (64.64%), error:     10346.89941, best error:     10323.52930 on epoch:    84, epoch:   89/100, mu: 0.9850366116, learning_rate: 0.0000029641, weight_decay: 0.0000050155

epoch time: 1350.6848144531s, input_fired_time: 22.6337833405, output_fired_time: 7.8516540527, propagate_forward_time: 502.1212768555, propagate_backward_time: 804.8395996094, weight_update_time: 0.4584438205, other_time: 13.2384643555
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542127803_186_5817_1_r1998742746_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516735
read mu: 0.985284
read mu_delta: 0.95008
read initial_learning_rate: 0.00111706
read learning_rate: 2.77298e-06
read learning_rate_delta: 0.935522
read initial_weight_decay: 0.000585148
read weight_decay: 4.75436e-06
read weight_decay_delta: 0.947929
read batch_size: 95
read epsilon: 1e-07
read alpha: 0.0840616
read input_dropout_probability: 0.00120912
read hidden_dropout_probability: 0.109168
read velocity_reset: 1842
read epoch: 90
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 84
read number_validation_images: 10000
read best_validation_predictions: 6464
read best_validation_error: 10323.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5817
read normal distribution: '1 -0.558461308479309 1.34882891178131'
generator_str: '380052286'
read generator: 380052286
read generated_by_map:
1 split_node 1
reading 85 nodes.
reading 320 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1332.48937988281s, input_fired_time: 22.1803760528564, output_fired_time: 7.7448673248291, propagate_forward_time: 489.081237792969, propagate_backward_time: 800.608276367188, weight_update_time: 0.449935466051102, other_time: 12.8745727539063
epoch time: 101.892936706543s, input_fired_time: 1.36099588871002, output_fired_time: 0, propagate_forward_time: 98.478645324707, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.05329895019531
validation[          , genome  5817] predictions:    6449/  10000 (64.49%), best:    6449/10000 (64.49%), error:     10323.01855, best error:     10323.01855 on epoch:    90, epoch:   90/100, mu: 0.9852843881, learning_rate: 0.0000027730, weight_decay: 0.0000047544

epoch time: 1314.7147216797s, input_fired_time: 21.9674587250, output_fired_time: 7.6211419106, propagate_forward_time: 486.5263671875, propagate_backward_time: 785.7009887695, weight_update_time: 0.4480617642, other_time: 12.8988647461
epoch time: 100.5054702759s, input_fired_time: 1.3195120096, output_fired_time: 0.0000000000, propagate_forward_time: 97.0932235718, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0927352905
validation[          , genome  5817] predictions:    6424/  10000 (64.24%), best:    6449/10000 (64.49%), error:     10335.61035, best error:     10323.01855 on epoch:    90, epoch:   91/100, mu: 0.9855197668, learning_rate: 0.0000025942, weight_decay: 0.0000045068

epoch time: 1321.3559570313s, input_fired_time: 22.0182228088, output_fired_time: 7.6152591705, propagate_forward_time: 489.6011047363, propagate_backward_time: 789.2020874023, weight_update_time: 0.4470050633, other_time: 12.9193725586
epoch time: 125.1447525024s, input_fired_time: 1.5338792801, output_fired_time: 0.0000000000, propagate_forward_time: 121.4383850098, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1724853516
validation[          , genome  5817] predictions:    6418/  10000 (64.18%), best:    6449/10000 (64.49%), error:     10351.27051, best error:     10323.01855 on epoch:    90, epoch:   92/100, mu: 0.9857434034, learning_rate: 0.0000024269, weight_decay: 0.0000042721

epoch time: 1308.1511230469s, input_fired_time: 21.7633590698, output_fired_time: 7.6224932671, propagate_forward_time: 484.7819213867, propagate_backward_time: 781.1849975586, weight_update_time: 0.4442458451, other_time: 12.7984619141
epoch time: 100.1087417603s, input_fired_time: 1.3167332411, output_fired_time: 0.0000000000, propagate_forward_time: 96.7140350342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0779724121
validation[          , genome  5817] predictions:    6465/  10000 (64.65%), best:    6465/10000 (64.65%), error:     10296.82910, best error:     10296.82910 on epoch:    93, epoch:   93/100, mu: 0.9859558940, learning_rate: 0.0000022704, weight_decay: 0.0000040497

epoch time: 1302.7863769531s, input_fired_time: 21.8169422150, output_fired_time: 7.5461063385, propagate_forward_time: 483.7369689941, propagate_backward_time: 776.8780517578, weight_update_time: 0.4424097538, other_time: 12.8083496094
epoch time: 100.3971633911s, input_fired_time: 1.3198770285, output_fired_time: 0.0000000000, propagate_forward_time: 96.9863204956, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0909652710
validation[          , genome  5817] predictions:    6433/  10000 (64.33%), best:    6465/10000 (64.65%), error:     10343.54883, best error:     10296.82910 on epoch:    93, epoch:   94/100, mu: 0.9861577749, learning_rate: 0.0000021240, weight_decay: 0.0000038388

epoch time: 1317.1776123047s, input_fired_time: 21.8656215668, output_fired_time: 7.7049446106, propagate_forward_time: 488.3349304199, propagate_backward_time: 786.6780395508, weight_update_time: 0.4472219646, other_time: 12.5940551758
epoch time: 101.0270767212s, input_fired_time: 1.3241962194, output_fired_time: 0.0000000000, propagate_forward_time: 97.7009658813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0019149780
validation[          , genome  5817] predictions:    6458/  10000 (64.58%), best:    6458/10000 (64.58%), error:     10280.09473, best error:     10280.09473 on epoch:    95, epoch:   95/100, mu: 0.9863495827, learning_rate: 0.0000019871, weight_decay: 0.0000036389

epoch time: 1309.5837402344s, input_fired_time: 21.9197235107, output_fired_time: 7.6820211411, propagate_forward_time: 483.3396911621, propagate_backward_time: 784.0531616211, weight_update_time: 0.4442174435, other_time: 12.5891723633
epoch time: 101.6007690430s, input_fired_time: 1.3218548298, output_fired_time: 0.0000000000, propagate_forward_time: 98.2730026245, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0059127808
validation[          , genome  5817] predictions:    6450/  10000 (64.50%), best:    6458/10000 (64.58%), error:     10312.47559, best error:     10280.09473 on epoch:    95, epoch:   96/100, mu: 0.9865317941, learning_rate: 0.0000018590, weight_decay: 0.0000034494

epoch time: 1310.5277099609s, input_fired_time: 21.8562774658, output_fired_time: 7.5526599884, propagate_forward_time: 484.9520568848, propagate_backward_time: 783.2730102539, weight_update_time: 0.4427166581, other_time: 12.8937377930
epoch time: 101.1303405762s, input_fired_time: 1.3495802879, output_fired_time: 0.0000000000, propagate_forward_time: 97.6400756836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1406860352
validation[          , genome  5817] predictions:    6439/  10000 (64.39%), best:    6458/10000 (64.58%), error:     10316.66113, best error:     10280.09473 on epoch:    95, epoch:   97/100, mu: 0.9867049456, learning_rate: 0.0000017391, weight_decay: 0.0000032698

epoch time: 1300.0198974609s, input_fired_time: 21.7087020874, output_fired_time: 7.5218982697, propagate_forward_time: 485.9681091309, propagate_backward_time: 771.9885864258, weight_update_time: 0.4430365562, other_time: 12.8325805664
epoch time: 100.1296005249s, input_fired_time: 1.3144042492, output_fired_time: 0.0000000000, propagate_forward_time: 96.7420120239, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0731811523
validation[          , genome  5817] predictions:    6435/  10000 (64.35%), best:    6458/10000 (64.58%), error:     10305.74512, best error:     10280.09473 on epoch:    95, epoch:   98/100, mu: 0.9868694544, learning_rate: 0.0000016270, weight_decay: 0.0000030995

epoch time: 1308.4727783203s, input_fired_time: 21.8036136627, output_fired_time: 7.5912184715, propagate_forward_time: 481.5829467773, propagate_backward_time: 784.7993774414, weight_update_time: 0.4441406727, other_time: 12.6956787109
epoch time: 100.8305358887s, input_fired_time: 1.3211094141, output_fired_time: 0.0000000000, propagate_forward_time: 97.4804763794, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0289535522
validation[          , genome  5817] predictions:    6468/  10000 (64.68%), best:    6458/10000 (64.58%), error:     10294.08887, best error:     10280.09473 on epoch:    95, epoch:   99/100, mu: 0.9870257378, learning_rate: 0.0000015221, weight_decay: 0.0000029382

epoch time: 1299.2609863281s, input_fired_time: 21.7978057861, output_fired_time: 7.5026292801, propagate_forward_time: 481.0496520996, propagate_backward_time: 776.0379638672, weight_update_time: 0.4437528253, other_time: 12.8728027344
epoch time: 99.7569656372s, input_fired_time: 1.3169499636, output_fired_time: 0.0000000000, propagate_forward_time: 96.3735275269, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0664901733
validation[          , genome  5817] predictions:    6451/  10000 (64.51%), best:    6458/10000 (64.58%), error:     10324.63477, best error:     10280.09473 on epoch:    95, epoch:  100/100, mu: 0.9871742129, learning_rate: 0.0000014239, weight_decay: 0.0000027852

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 505.7526550293s, input_fired_time: 6.6333098412, output_fired_time: 0.0000000000, propagate_forward_time: 488.5621948242, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.5571594238
best training[          , genome  5817] predictions:   35281/  50000 (70.56%), best:    6458/10000 (64.58%), error:     44698.99609, best error:     10280.09473 on epoch:    95, epoch:  101/100, mu: 0.9873152971, learning_rate: 0.0000013321, weight_decay: 0.0000026401
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 99.1677246094s, input_fired_time: 1.3188227415, output_fired_time: 0.0000000000, propagate_forward_time: 95.7705383301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0783615112
   testing[          , genome  5817] predictions:    6458/  10000 (64.58%), best:    6458/10000 (64.58%), error:     10280.09473, best error:     10280.09473 on epoch:    95, epoch:  101/100, mu: 0.9873152971, learning_rate: 0.0000013321, weight_decay: 0.0000026401
backpropagation finished successfully!
12:05:30 (1688): called boinc_finish(0)

</stderr_txt>
]]>