Nameexact_genome_1542628805_181_6315_0
Workunit4098781
Created19 Nov 2018, 14:42:53 UTC
Sent19 Nov 2018, 14:43:31 UTC
Report deadline14 Jan 2019, 15:54:45 UTC
Received21 Nov 2018, 11:04:51 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time1 days 0 hours 0 min 42 sec
CPU time23 hours 25 min 9 sec
Validate stateValid
Credit14,012.37
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size560.13 MB
Peak swap size579.83 MB
Peak disk usage11.74 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
th 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 693592997
post shuffle 1: 1692478127
epoch time: 1745.13537597656s, input_fired_time: 18.5765018463135, output_fired_time: 8.29922389984131, propagate_forward_time: 676.81201171875, propagate_backward_time: 1007.77532958984, weight_update_time: 0.445154696702957, other_time: 33.6723022460938
epoch time: 144.537612915039s, input_fired_time: 1.35766911506653, output_fired_time: 0, propagate_forward_time: 137.888595581055, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.29135131835938
validation[          , genome  6315] predictions:    5363/  10000 (53.63%), best:    5363/10000 (53.63%), error:     13122.49707, best error:     13122.49707 on epoch:     0, epoch:    0/50, mu: 0.4870123565, learning_rate: 0.0021714908, weight_decay: 0.0003894347

epoch time: 1782.6884765625s, input_fired_time: 18.7705879211, output_fired_time: 8.3349351883, propagate_forward_time: 694.6889648438, propagate_backward_time: 1018.4128417969, weight_update_time: 0.4178677499, other_time: 42.4810791016
epoch time: 143.5085449219s, input_fired_time: 1.3777242899, output_fired_time: 0.0000000000, propagate_forward_time: 137.7886047363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3422088623
validation[          , genome  6315] predictions:    5578/  10000 (55.78%), best:    5578/10000 (55.78%), error:     12566.99219, best error:     12566.99219 on epoch:     1, epoch:    1/50, mu: 0.5198482275, learning_rate: 0.0019936687, weight_decay: 0.0003624565

epoch time: 1756.1225585938s, input_fired_time: 18.6573123932, output_fired_time: 8.2293024063, propagate_forward_time: 687.3436889648, propagate_backward_time: 1008.5014648438, weight_update_time: 0.4069676101, other_time: 33.3908691406
epoch time: 145.8372497559s, input_fired_time: 1.3949323893, output_fired_time: 0.0000000000, propagate_forward_time: 140.1544952393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2878265381
validation[          , genome  6315] predictions:    5724/  10000 (57.24%), best:    5724/10000 (57.24%), error:     12358.59668, best error:     12358.59668 on epoch:     2, epoch:    2/50, mu: 0.5505405068, learning_rate: 0.0018304085, weight_decay: 0.0003373472

epoch time: 1785.3699951172s, input_fired_time: 19.0413017273, output_fired_time: 8.4091405869, propagate_forward_time: 698.9362182617, propagate_backward_time: 1022.1682128906, weight_update_time: 0.4127063155, other_time: 36.8151855469
epoch time: 142.8061981201s, input_fired_time: 1.3782100677, output_fired_time: 0.0000000000, propagate_forward_time: 137.6802978516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7476959229
validation[          , genome  6315] predictions:    5806/  10000 (58.06%), best:    5806/10000 (58.06%), error:     12095.60254, best error:     12095.60254 on epoch:     3, epoch:    3/50, mu: 0.5792291164, learning_rate: 0.0016805175, weight_decay: 0.0003139774

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315_0_r2137330144_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487012
read mu: 0.606045
read mu_delta: 0.934718
read initial_learning_rate: 0.00217149
read learning_rate: 0.0015429
read learning_rate_delta: 0.918111
read initial_weight_decay: 0.000389435
read weight_decay: 0.000292227
read weight_decay_delta: 0.930725
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0486348
read input_dropout_probability: 0.00134763
read hidden_dropout_probability: 0.111079
read velocity_reset: 1021
read epoch: 4
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 3
read number_validation_images: 10000
read best_validation_predictions: 5806
read best_validation_error: 12095.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6315
read normal distribution: '1 0 0'
generator_str: '850858780'
read generator: 850858780
read generated_by_map:
1 crossover 1
reading 98 nodes.
reading 356 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1861.07666015625s, input_fired_time: 19.7582912445068, output_fired_time: 8.65031433105469, propagate_forward_time: 731.953674316406, propagate_backward_time: 1062.8818359375, weight_update_time: 0.417751014232635, other_time: 37.83251953125
epoch time: 149.197952270508s, input_fired_time: 1.44150054454803, output_fired_time: 0, propagate_forward_time: 145.076034545898, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.680419921875
validation[          , genome  6315] predictions:    5667/  10000 (56.67%), best:    5806/10000 (58.06%), error:     12398.72949, best error:     12095.60254 on epoch:     3, epoch:    4/50, mu: 0.6060448885, learning_rate: 0.0015429009, weight_decay: 0.0002922265

epoch time: 1701.0589599609s, input_fired_time: 18.1299724579, output_fired_time: 8.2023649216, propagate_forward_time: 670.4105224609, propagate_backward_time: 989.4019165039, weight_update_time: 0.3614172339, other_time: 14.9141235352
epoch time: 143.4212951660s, input_fired_time: 1.3382903337, output_fired_time: 0.0000000000, propagate_forward_time: 139.2634735107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8195343018
validation[          , genome  6315] predictions:    5559/  10000 (55.59%), best:    5806/10000 (58.06%), error:     12783.73145, best error:     12095.60254 on epoch:     3, epoch:    5/50, mu: 0.6311101317, learning_rate: 0.0014165537, weight_decay: 0.0002719825

epoch time: 1716.9121093750s, input_fired_time: 18.2577362061, output_fired_time: 8.0519227982, propagate_forward_time: 677.2520751953, propagate_backward_time: 998.3693847656, weight_update_time: 0.3823762834, other_time: 14.9810791016
epoch time: 140.8759002686s, input_fired_time: 1.3305507898, output_fired_time: 0.0000000000, propagate_forward_time: 136.9436950684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6016540527
validation[          , genome  6315] predictions:    5681/  10000 (56.81%), best:    5806/10000 (58.06%), error:     12351.48340, best error:     12095.60254 on epoch:     3, epoch:    6/50, mu: 0.6545390487, learning_rate: 0.0013005530, weight_decay: 0.0002531408

epoch time: 1693.7950439453s, input_fired_time: 18.1043376923, output_fired_time: 8.0767107010, propagate_forward_time: 667.1838989258, propagate_backward_time: 985.8804931641, weight_update_time: 0.3666425943, other_time: 14.5496215820
epoch time: 136.8457946777s, input_fired_time: 1.3382354975, output_fired_time: 0.0000000000, propagate_forward_time: 132.9153900146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5921630859
validation[          , genome  6315] predictions:    5642/  10000 (56.42%), best:    5806/10000 (58.06%), error:     12530.89941, best error:     12095.60254 on epoch:     3, epoch:    7/50, mu: 0.6764385104, learning_rate: 0.0011940515, weight_decay: 0.0002356044

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315_0_r2137330144_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487012
read mu: 0.696908
read mu_delta: 0.934718
read initial_learning_rate: 0.00217149
read learning_rate: 0.00109627
read learning_rate_delta: 0.918111
read initial_weight_decay: 0.000389435
read weight_decay: 0.000219283
read weight_decay_delta: 0.930725
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0486348
read input_dropout_probability: 0.00134763
read hidden_dropout_probability: 0.111079
read velocity_reset: 1021
read epoch: 8
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 3
read number_validation_images: 10000
read best_validation_predictions: 5806
read best_validation_error: 12095.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6315
read normal distribution: '1 0 0'
generator_str: '156587630'
read generator: 156587630
read generated_by_map:
1 crossover 1
reading 98 nodes.
reading 356 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1747.65747070313s, input_fired_time: 18.3251895904541, output_fired_time: 7.98961544036865, propagate_forward_time: 711.616577148438, propagate_backward_time: 974.878295898438, weight_update_time: 0.362673848867416, other_time: 34.8477783203125
epoch time: 149.218978881836s, input_fired_time: 1.35102641582489, output_fired_time: 0, propagate_forward_time: 144.064193725586, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.80375671386719
validation[          , genome  6315] predictions:    5734/  10000 (57.34%), best:    5806/10000 (58.06%), error:     12239.66211, best error:     12095.60254 on epoch:     3, epoch:    8/50, mu: 0.6969082952, learning_rate: 0.0010962713, weight_decay: 0.0002192829

epoch time: 1721.9481201172s, input_fired_time: 18.2185688019, output_fired_time: 8.0678377151, propagate_forward_time: 705.2512817383, propagate_backward_time: 974.8505249023, weight_update_time: 0.3536446989, other_time: 15.5598144531
epoch time: 146.0150909424s, input_fired_time: 1.3390501738, output_fired_time: 0.0000000000, propagate_forward_time: 141.9198303223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7562103271
validation[          , genome  6315] predictions:    5923/  10000 (59.23%), best:    5923/10000 (59.23%), error:     11837.18066, best error:     11837.18066 on epoch:     9, epoch:    9/50, mu: 0.7160418034, learning_rate: 0.0010064984, weight_decay: 0.0002040920

epoch time: 1774.6671142578s, input_fired_time: 18.4573917389, output_fired_time: 8.1845388412, propagate_forward_time: 734.7613525391, propagate_backward_time: 982.7841796875, weight_update_time: 0.3710370660, other_time: 30.4796142578
epoch time: 150.8645629883s, input_fired_time: 1.3715844154, output_fired_time: 0.0000000000, propagate_forward_time: 145.7630462646, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7299346924
validation[          , genome  6315] predictions:    5683/  10000 (56.83%), best:    5923/10000 (59.23%), error:     12408.25977, best error:     11837.18066 on epoch:     9, epoch:   10/50, mu: 0.7339262366, learning_rate: 0.0009240768, weight_decay: 0.0001899535

epoch time: 1650.6671142578s, input_fired_time: 17.8703880310, output_fired_time: 7.6669378281, propagate_forward_time: 665.4610595703, propagate_backward_time: 924.5749511719, weight_update_time: 0.3515926301, other_time: 35.0937500000
epoch time: 127.2926559448s, input_fired_time: 1.2062759399, output_fired_time: 0.0000000000, propagate_forward_time: 123.6140365601, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4723434448
validation[          , genome  6315] predictions:    6021/  10000 (60.21%), best:    6021/10000 (60.21%), error:     11697.71777, best error:     11697.71777 on epoch:    11, epoch:   11/50, mu: 0.7506431341, learning_rate: 0.0008484047, weight_decay: 0.0001767944

epoch time: 1554.3382568359s, input_fired_time: 17.5363349915, output_fired_time: 7.3022670746, propagate_forward_time: 639.0316162109, propagate_backward_time: 876.1300048828, weight_update_time: 0.3425961137, other_time: 14.3380126953
epoch time: 128.4735870361s, input_fired_time: 1.2262517214, output_fired_time: 0.0000000000, propagate_forward_time: 124.7880020142, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4593353271
validation[          , genome  6315] predictions:    5913/  10000 (59.13%), best:    6021/10000 (60.21%), error:     11822.97461, best error:     11697.71777 on epoch:    11, epoch:   12/50, mu: 0.7662687302, learning_rate: 0.0007789294, weight_decay: 0.0001645469

epoch time: 1283.2297363281s, input_fired_time: 16.3904743195, output_fired_time: 5.9507756233, propagate_forward_time: 508.4409179688, propagate_backward_time: 736.4615478516, weight_update_time: 0.2808906734, other_time: 15.9859619141
epoch time: 93.8916854858s, input_fired_time: 0.9691566825, output_fired_time: 0.0000000000, propagate_forward_time: 90.1634368896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7590942383
validation[          , genome  6315] predictions:    5889/  10000 (58.89%), best:    6021/10000 (60.21%), error:     11781.52246, best error:     11697.71777 on epoch:    11, epoch:   13/50, mu: 0.7808742523, learning_rate: 0.0007151433, weight_decay: 0.0001531479

epoch time: 1144.9666748047s, input_fired_time: 15.8174934387, output_fired_time: 5.3324980736, propagate_forward_time: 446.7459106445, propagate_backward_time: 661.7609863281, weight_update_time: 0.2534525692, other_time: 15.3097534180
epoch time: 94.2059783936s, input_fired_time: 0.9760742784, output_fired_time: 0.0000000000, propagate_forward_time: 90.5482711792, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6816329956
validation[          , genome  6315] predictions:    5981/  10000 (59.81%), best:    5981/10000 (59.81%), error:     11562.58105, best error:     11562.58105 on epoch:    14, epoch:   14/50, mu: 0.7945263386, learning_rate: 0.0006565807, weight_decay: 0.0001425385

epoch time: 1145.4273681641s, input_fired_time: 15.8069734573, output_fired_time: 5.3646073341, propagate_forward_time: 448.1130981445, propagate_backward_time: 660.8477172852, weight_update_time: 0.2517614961, other_time: 15.2949218750
epoch time: 93.3329238892s, input_fired_time: 0.9761472344, output_fired_time: 0.0000000000, propagate_forward_time: 89.6464385986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7103347778
validation[          , genome  6315] predictions:    5961/  10000 (59.61%), best:    5981/10000 (59.81%), error:     11672.80859, best error:     11562.58105 on epoch:    14, epoch:   15/50, mu: 0.8072871566, learning_rate: 0.0006028137, weight_decay: 0.0001326641

epoch time: 1146.0372314453s, input_fired_time: 15.8561286926, output_fired_time: 5.3896894455, propagate_forward_time: 449.0258178711, propagate_backward_time: 660.4096069336, weight_update_time: 0.2502031922, other_time: 15.3560791016
epoch time: 82.5112838745s, input_fired_time: 0.8954190016, output_fired_time: 0.0000000000, propagate_forward_time: 78.8031692505, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8126983643
validation[          , genome  6315] predictions:    5962/  10000 (59.62%), best:    5981/10000 (59.81%), error:     11632.54395, best error:     11562.58105 on epoch:    14, epoch:   16/50, mu: 0.8192149401, learning_rate: 0.0005534496, weight_decay: 0.0001234738

epoch time: 993.1114501953s, input_fired_time: 15.3906393051, output_fired_time: 4.5589909554, propagate_forward_time: 385.0721130371, propagate_backward_time: 572.6795654297, weight_update_time: 0.2289067954, other_time: 15.4101562500
epoch time: 71.5199966431s, input_fired_time: 0.8349615335, output_fired_time: 0.0000000000, propagate_forward_time: 67.5804214478, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1046142578
validation[          , genome  6315] predictions:    5912/  10000 (59.12%), best:    5981/10000 (59.81%), error:     11751.51563, best error:     11562.58105 on epoch:    14, epoch:   17/50, mu: 0.8303640485, learning_rate: 0.0005081280, weight_decay: 0.0001149201

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315_0_r2137330144_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487012
read mu: 0.840785
read mu_delta: 0.934718
read initial_learning_rate: 0.00217149
read learning_rate: 0.000466518
read learning_rate_delta: 0.918111
read initial_weight_decay: 0.000389435
read weight_decay: 0.000106959
read weight_decay_delta: 0.930725
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0486348
read input_dropout_probability: 0.00134763
read hidden_dropout_probability: 0.111079
read velocity_reset: 1021
read epoch: 18
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 14
read number_validation_images: 10000
read best_validation_predictions: 5981
read best_validation_error: 11562.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6315
read normal distribution: '1 0 0'
generator_str: '1309691520'
read generator: 1309691520
read generated_by_map:
1 crossover 1
reading 98 nodes.
reading 356 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1603.02551269531s, input_fired_time: 18.2608013153076, output_fired_time: 7.76315307617188, propagate_forward_time: 631.074768066406, propagate_backward_time: 932.550842285156, weight_update_time: 0.359965026378632, other_time: 13.3759765625
epoch time: 127.554344177246s, input_fired_time: 1.32374620437622, output_fired_time: 0, propagate_forward_time: 123.895530700684, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.33506774902344
validation[          , genome  6315] predictions:    6068/  10000 (60.68%), best:    6068/10000 (60.68%), error:     11293.54102, best error:     11293.54102 on epoch:    18, epoch:   18/50, mu: 0.8407853246, learning_rate: 0.0004665177, weight_decay: 0.0001069590

epoch time: 1587.6854248047s, input_fired_time: 17.7641906738, output_fired_time: 7.6619782448, propagate_forward_time: 625.4448852539, propagate_backward_time: 923.2671508789, weight_update_time: 0.3501651287, other_time: 13.5472412109
epoch time: 126.5861511230s, input_fired_time: 1.3184479475, output_fired_time: 0.0000000000, propagate_forward_time: 122.9724273682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2952728271
validation[          , genome  6315] predictions:    5990/  10000 (59.90%), best:    6068/10000 (60.68%), error:     11727.21777, best error:     11293.54102 on epoch:    18, epoch:   19/50, mu: 0.8505262733, learning_rate: 0.0004283148, weight_decay: 0.0000995494

epoch time: 1590.0532226563s, input_fired_time: 17.7325534821, output_fired_time: 7.6578221321, propagate_forward_time: 626.2078857422, propagate_backward_time: 924.9003906250, weight_update_time: 0.3542857766, other_time: 13.5545654297
epoch time: 128.9852447510s, input_fired_time: 1.3260235786, output_fired_time: 0.0000000000, propagate_forward_time: 125.2656860352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3935317993
validation[          , genome  6315] predictions:    6170/  10000 (61.70%), best:    6170/10000 (61.70%), error:     11152.93457, best error:     11152.93457 on epoch:    20, epoch:   20/50, mu: 0.8596313596, learning_rate: 0.0003932404, weight_decay: 0.0000926531

epoch time: 1590.5032958984s, input_fired_time: 17.7553806305, output_fired_time: 7.6682348251, propagate_forward_time: 626.6604614258, propagate_backward_time: 924.7657470703, weight_update_time: 0.3498381376, other_time: 13.6535034180
epoch time: 130.7589569092s, input_fired_time: 1.3158154488, output_fired_time: 0.0000000000, propagate_forward_time: 127.0345535278, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4085922241
validation[          , genome  6315] predictions:    6057/  10000 (60.57%), best:    6170/10000 (61.70%), error:     11338.53125, best error:     11152.93457 on epoch:    20, epoch:   21/50, mu: 0.8681420088, learning_rate: 0.0003610381, weight_decay: 0.0000862345

epoch time: 1608.1894531250s, input_fired_time: 17.7673282623, output_fired_time: 7.7135567665, propagate_forward_time: 633.8732910156, propagate_backward_time: 935.0026855469, weight_update_time: 0.3545848131, other_time: 13.8326416016
epoch time: 133.2516937256s, input_fired_time: 1.3091239929, output_fired_time: 0.0000000000, propagate_forward_time: 129.4048156738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5377502441
validation[          , genome  6315] predictions:    6028/  10000 (60.28%), best:    6170/10000 (61.70%), error:     11543.07324, best error:     11152.93457 on epoch:    20, epoch:   22/50, mu: 0.8760970831, learning_rate: 0.0003314730, weight_decay: 0.0000802606

epoch time: 1619.2611083984s, input_fired_time: 17.7016563416, output_fired_time: 7.7251396179, propagate_forward_time: 638.3657836914, propagate_backward_time: 941.5689697266, weight_update_time: 0.3553105295, other_time: 13.8995971680
epoch time: 127.8293075562s, input_fired_time: 1.3369725943, output_fired_time: 0.0000000000, propagate_forward_time: 124.1472015381, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3451309204
validation[          , genome  6315] predictions:    6058/  10000 (60.58%), best:    6170/10000 (61.70%), error:     11324.74512, best error:     11152.93457 on epoch:    20, epoch:   23/50, mu: 0.8835328221, learning_rate: 0.0003043288, weight_decay: 0.0000747005

epoch time: 1666.2803955078s, input_fired_time: 17.7583789825, output_fired_time: 7.8604183197, propagate_forward_time: 657.3193359375, propagate_backward_time: 968.3533325195, weight_update_time: 0.3543676138, other_time: 14.9888305664
epoch time: 136.6817016602s, input_fired_time: 1.3606294394, output_fired_time: 0.0000000000, propagate_forward_time: 132.7407684326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5803070068
validation[          , genome  6315] predictions:    6116/  10000 (61.16%), best:    6170/10000 (61.70%), error:     11188.81152, best error:     11152.93457 on epoch:    20, epoch:   24/50, mu: 0.8904832006, learning_rate: 0.0002794075, weight_decay: 0.0000695256

epoch time: 1682.7221679688s, input_fired_time: 17.8521652222, output_fired_time: 7.9758009911, propagate_forward_time: 663.0968627930, propagate_backward_time: 978.8572998047, weight_update_time: 0.3502677381, other_time: 14.9400024414
epoch time: 136.1736145020s, input_fired_time: 1.3319828510, output_fired_time: 0.0000000000, propagate_forward_time: 132.1574707031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6841583252
validation[          , genome  6315] predictions:    6060/  10000 (60.60%), best:    6170/10000 (61.70%), error:     11357.19043, best error:     11152.93457 on epoch:    20, epoch:   25/50, mu: 0.8969798088, learning_rate: 0.0002565270, weight_decay: 0.0000647092

epoch time: 1687.9178466797s, input_fired_time: 17.7892246246, output_fired_time: 7.9463052750, propagate_forward_time: 665.8002929688, propagate_backward_time: 981.4613647461, weight_update_time: 0.3510075808, other_time: 14.9207153320
epoch time: 138.1910095215s, input_fired_time: 1.3412370682, output_fired_time: 0.0000000000, propagate_forward_time: 134.1539306641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6958465576
validation[          , genome  6315] predictions:    6214/  10000 (62.14%), best:    6214/10000 (62.14%), error:     11070.31738, best error:     11070.31738 on epoch:    26, epoch:   26/50, mu: 0.9030523300, learning_rate: 0.0002355202, weight_decay: 0.0000602265

epoch time: 1696.5541992188s, input_fired_time: 17.8591175079, output_fired_time: 8.0169954300, propagate_forward_time: 669.0942382813, propagate_backward_time: 985.2800903320, weight_update_time: 0.3470441699, other_time: 16.3037719727
epoch time: 138.2265625000s, input_fired_time: 1.3706365824, output_fired_time: 0.0000000000, propagate_forward_time: 134.0400390625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8158874512
validation[          , genome  6315] predictions:    6145/  10000 (61.45%), best:    6214/10000 (62.14%), error:     11247.26172, best error:     11070.31738 on epoch:    26, epoch:   27/50, mu: 0.9087284207, learning_rate: 0.0002162336, weight_decay: 0.0000560543

epoch time: 1694.9130859375s, input_fired_time: 17.8862819672, output_fired_time: 7.9713940620, propagate_forward_time: 668.6925048828, propagate_backward_time: 984.9151000977, weight_update_time: 0.3550350070, other_time: 15.4478149414
epoch time: 138.1837463379s, input_fired_time: 1.3521506786, output_fired_time: 0.0000000000, propagate_forward_time: 134.0919342041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7396545410
validation[          , genome  6315] predictions:    6212/  10000 (62.12%), best:    6212/10000 (62.12%), error:     11017.83496, best error:     11017.83496 on epoch:    28, epoch:   28/50, mu: 0.9140339494, learning_rate: 0.0001985264, weight_decay: 0.0000521711

epoch time: 1695.0272216797s, input_fired_time: 17.8738307953, output_fired_time: 7.9747009277, propagate_forward_time: 668.9466552734, propagate_backward_time: 984.5916137695, weight_update_time: 0.3466193378, other_time: 15.6404418945
epoch time: 138.5573730469s, input_fired_time: 1.3517801762, output_fired_time: 0.0000000000, propagate_forward_time: 134.3451080322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8604888916
validation[          , genome  6315] predictions:    6163/  10000 (61.63%), best:    6212/10000 (62.12%), error:     11206.11621, best error:     11017.83496 on epoch:    28, epoch:   29/50, mu: 0.9189931154, learning_rate: 0.0001822692, weight_decay: 0.0000485569

epoch time: 1694.5260009766s, input_fired_time: 17.9047050476, output_fired_time: 7.9806923866, propagate_forward_time: 668.4741210938, propagate_backward_time: 984.4752197266, weight_update_time: 0.3553205431, other_time: 15.6912841797
epoch time: 138.1684417725s, input_fired_time: 1.3603651524, output_fired_time: 0.0000000000, propagate_forward_time: 134.0975646973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7105102539
validation[          , genome  6315] predictions:    6158/  10000 (61.58%), best:    6212/10000 (62.12%), error:     11152.55469, best error:     11017.83496 on epoch:    28, epoch:   30/50, mu: 0.9236285686, learning_rate: 0.0001673433, weight_decay: 0.0000451931

epoch time: 1695.4022216797s, input_fired_time: 17.8783473969, output_fired_time: 7.9416141510, propagate_forward_time: 668.4484863281, propagate_backward_time: 985.7855834961, weight_update_time: 0.3476679921, other_time: 15.3482055664
epoch time: 138.3295898438s, input_fired_time: 1.3510180712, output_fired_time: 0.0000000000, propagate_forward_time: 134.2263641357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7522125244
validation[          , genome  6315] predictions:    6161/  10000 (61.61%), best:    6212/10000 (62.12%), error:     11152.16895, best error:     11017.83496 on epoch:    28, epoch:   31/50, mu: 0.9279614091, learning_rate: 0.0001536396, weight_decay: 0.0000420624

epoch time: 1694.5983886719s, input_fired_time: 17.8283538818, output_fired_time: 8.0308570862, propagate_forward_time: 668.7256469727, propagate_backward_time: 984.5872802734, weight_update_time: 0.3457471728, other_time: 15.4262084961
epoch time: 138.1057434082s, input_fired_time: 1.3546649218, output_fired_time: 0.0000000000, propagate_forward_time: 134.0675048828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6835784912
validation[          , genome  6315] predictions:    6071/  10000 (60.71%), best:    6212/10000 (62.12%), error:     11417.43359, best error:     11017.83496 on epoch:    28, epoch:   32/50, mu: 0.9320113659, learning_rate: 0.0001410582, weight_decay: 0.0000391485

epoch time: 1695.1442871094s, input_fired_time: 17.8361473083, output_fired_time: 7.9719591141, propagate_forward_time: 668.3605346680, propagate_backward_time: 985.6737060547, weight_update_time: 0.3471722305, other_time: 15.3019409180
epoch time: 134.7445068359s, input_fired_time: 1.3276668787, output_fired_time: 0.0000000000, propagate_forward_time: 130.7117767334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7050628662
validation[          , genome  6315] predictions:    6173/  10000 (61.73%), best:    6212/10000 (62.12%), error:     11078.62207, best error:     11017.83496 on epoch:    28, epoch:   33/50, mu: 0.9357969761, learning_rate: 0.0001295070, weight_decay: 0.0000364365

epoch time: 1553.7153320313s, input_fired_time: 17.3017539978, output_fired_time: 7.2614498138, propagate_forward_time: 604.4891357422, propagate_backward_time: 909.1427001953, weight_update_time: 0.3262524605, other_time: 15.5202636719
epoch time: 126.0721511841s, input_fired_time: 1.2687602043, output_fired_time: 0.0000000000, propagate_forward_time: 122.0317077637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7716827393
validation[          , genome  6315] predictions:    6252/  10000 (62.52%), best:    6252/10000 (62.52%), error:     10813.86426, best error:     10813.86426 on epoch:    34, epoch:   34/50, mu: 0.9393354654, learning_rate: 0.0001189017, weight_decay: 0.0000339123

epoch time: 1553.4024658203s, input_fired_time: 17.2817001343, output_fired_time: 7.3015737534, propagate_forward_time: 603.7283935547, propagate_backward_time: 909.7803955078, weight_update_time: 0.3260616362, other_time: 15.3104248047
epoch time: 124.4773941040s, input_fired_time: 1.2560812235, output_fired_time: 0.0000000000, propagate_forward_time: 120.6679077148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5534057617
validation[          , genome  6315] predictions:    6270/  10000 (62.70%), best:    6252/10000 (62.52%), error:     10832.78613, best error:     10813.86426 on epoch:    34, epoch:   35/50, mu: 0.9426429272, learning_rate: 0.0001091650, weight_decay: 0.0000315630

epoch time: 1553.3996582031s, input_fired_time: 17.2811985016, output_fired_time: 7.2948660851, propagate_forward_time: 603.9385375977, propagate_backward_time: 909.5405883789, weight_update_time: 0.3230919838, other_time: 15.3443603516
epoch time: 125.0687713623s, input_fired_time: 1.2775907516, output_fired_time: 0.0000000000, propagate_forward_time: 120.9612197876, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8299636841
validation[          , genome  6315] predictions:    6301/  10000 (63.01%), best:    6301/10000 (63.01%), error:     10723.19629, best error:     10723.19629 on epoch:    36, epoch:   36/50, mu: 0.9457345009, learning_rate: 0.0001002255, weight_decay: 0.0000293765

epoch time: 1556.2941894531s, input_fired_time: 17.2833900452, output_fired_time: 7.2469601631, propagate_forward_time: 605.4740600586, propagate_backward_time: 910.7293090820, weight_update_time: 0.3237153292, other_time: 15.5604248047
epoch time: 125.6411590576s, input_fired_time: 1.2738585472, output_fired_time: 0.0000000000, propagate_forward_time: 121.6610260010, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7062759399
validation[          , genome  6315] predictions:    6181/  10000 (61.81%), best:    6301/10000 (63.01%), error:     10996.24121, best error:     10723.19629 on epoch:    36, epoch:   37/50, mu: 0.9486242533, learning_rate: 0.0000920181, weight_decay: 0.0000273414

epoch time: 1448.9899902344s, input_fired_time: 16.9457378387, output_fired_time: 6.8158164024, propagate_forward_time: 558.0802612305, propagate_backward_time: 851.5638427734, weight_update_time: 0.3061359823, other_time: 15.5844116211
epoch time: 115.5462265015s, input_fired_time: 1.2085658312, output_fired_time: 0.0000000000, propagate_forward_time: 111.6310195923, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7066421509
validation[          , genome  6315] predictions:    6290/  10000 (62.90%), best:    6301/10000 (63.01%), error:     10748.38770, best error:     10723.19629 on epoch:    36, epoch:   38/50, mu: 0.9513253570, learning_rate: 0.0000844828, weight_decay: 0.0000254473

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315_0_r2137330144_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487012
read mu: 0.95385
read mu_delta: 0.934718
read initial_learning_rate: 0.00217149
read learning_rate: 7.75646e-05
read learning_rate_delta: 0.918111
read initial_weight_decay: 0.000389435
read weight_decay: 2.36845e-05
read weight_decay_delta: 0.930725
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0486348
read input_dropout_probability: 0.00134763
read hidden_dropout_probability: 0.111079
read velocity_reset: 1021
read epoch: 39
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 36
read number_validation_images: 10000
read best_validation_predictions: 6301
read best_validation_error: 10723.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6315
read normal distribution: '1 0 0'
generator_str: '2110608912'
read generator: 2110608912
read generated_by_map:
1 crossover 1
reading 98 nodes.
reading 356 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1410.77404785156s, input_fired_time: 17.1376495361328, output_fired_time: 6.68922233581543, propagate_forward_time: 533.389526367188, propagate_backward_time: 819.529663085938, weight_update_time: 0.318373382091522, other_time: 34.0279541015625
epoch time: 105.990119934082s, input_fired_time: 1.12983596324921, output_fired_time: 0, propagate_forward_time: 101.122581481934, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.73770141601563
validation[          , genome  6315] predictions:    6232/  10000 (62.32%), best:    6301/10000 (63.01%), error:     10928.41504, best error:     10723.19629 on epoch:    36, epoch:   39/50, mu: 0.9538500905, learning_rate: 0.0000775646, weight_decay: 0.0000236845

epoch time: 1336.4185791016s, input_fired_time: 16.6288928986, output_fired_time: 6.4353117943, propagate_forward_time: 508.4968566895, propagate_backward_time: 789.2038574219, weight_update_time: 0.2975328565, other_time: 15.6536865234
epoch time: 112.7028579712s, input_fired_time: 1.1852149963, output_fired_time: 0.0000000000, propagate_forward_time: 108.7707519531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7468872070
validation[          , genome  6315] predictions:    6299/  10000 (62.99%), best:    6301/10000 (63.01%), error:     10777.22852, best error:     10723.19629 on epoch:    36, epoch:   40/50, mu: 0.9562100172, learning_rate: 0.0000712128, weight_decay: 0.0000220437

epoch time: 1378.6236572266s, input_fired_time: 16.7691440582, output_fired_time: 6.4362840652, propagate_forward_time: 537.6341552734, propagate_backward_time: 802.3323974609, weight_update_time: 0.3127894700, other_time: 15.4516601563
epoch time: 110.8186264038s, input_fired_time: 1.1608786583, output_fired_time: 0.0000000000, propagate_forward_time: 106.8332977295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8244476318
validation[          , genome  6315] predictions:    6373/  10000 (63.73%), best:    6373/10000 (63.73%), error:     10658.09668, best error:     10658.09668 on epoch:    41, epoch:   41/50, mu: 0.9584158659, learning_rate: 0.0000653813, weight_decay: 0.0000205166

epoch time: 1392.3577880859s, input_fired_time: 16.8928203583, output_fired_time: 6.5419549942, propagate_forward_time: 544.7613525391, propagate_backward_time: 808.7947998047, weight_update_time: 0.3078014553, other_time: 15.3668212891
epoch time: 111.8048934937s, input_fired_time: 1.1861261129, output_fired_time: 0.0000000000, propagate_forward_time: 107.9556808472, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6630859375
validation[          , genome  6315] predictions:    6309/  10000 (63.09%), best:    6373/10000 (63.73%), error:     10771.83008, best error:     10658.09668 on epoch:    41, epoch:   42/50, mu: 0.9604777098, learning_rate: 0.0000600272, weight_decay: 0.0000190953

epoch time: 1479.3037109375s, input_fired_time: 17.3331394196, output_fired_time: 6.9996533394, propagate_forward_time: 575.6585693359, propagate_backward_time: 864.5212402344, weight_update_time: 0.3236378133, other_time: 14.7911376953
epoch time: 127.3747787476s, input_fired_time: 1.3110495806, output_fired_time: 0.0000000000, propagate_forward_time: 123.6380310059, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4256973267
validation[          , genome  6315] predictions:    6336/  10000 (63.36%), best:    6373/10000 (63.73%), error:     10668.98242, best error:     10658.09668 on epoch:    41, epoch:   43/50, mu: 0.9624049664, learning_rate: 0.0000551116, weight_decay: 0.0000177725

epoch time: 1424.0577392578s, input_fired_time: 16.9916591644, output_fired_time: 6.6977882385, propagate_forward_time: 548.5675659180, propagate_backward_time: 836.9434204102, weight_update_time: 0.3150032163, other_time: 14.8572998047
epoch time: 106.2776412964s, input_fired_time: 1.1495434046, output_fired_time: 0.0000000000, propagate_forward_time: 102.2909164429, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8371810913
validation[          , genome  6315] predictions:    6313/  10000 (63.13%), best:    6373/10000 (63.73%), error:     10660.51172, best error:     10658.09668 on epoch:    41, epoch:   44/50, mu: 0.9642063975, learning_rate: 0.0000505986, weight_decay: 0.0000165413

epoch time: 1335.5346679688s, input_fired_time: 16.6679592133, output_fired_time: 6.4280757904, propagate_forward_time: 507.7888488770, propagate_backward_time: 789.1891479492, weight_update_time: 0.2924090624, other_time: 15.4606323242
epoch time: 106.2386474609s, input_fired_time: 1.1439286470, output_fired_time: 0.0000000000, propagate_forward_time: 102.4412841797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6534347534
validation[          , genome  6315] predictions:    6354/  10000 (63.54%), best:    6354/10000 (63.54%), error:     10644.30762, best error:     10644.30762 on epoch:    45, epoch:   45/50, mu: 0.9658902287, learning_rate: 0.0000464551, weight_decay: 0.0000153954

epoch time: 1358.6568603516s, input_fired_time: 16.7777137756, output_fired_time: 6.4205155373, propagate_forward_time: 516.7939453125, propagate_backward_time: 803.3870849609, weight_update_time: 0.3014678061, other_time: 15.2775878906
epoch time: 117.3854827881s, input_fired_time: 1.2235720158, output_fired_time: 0.0000000000, propagate_forward_time: 113.3406143188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8212966919
validation[          , genome  6315] predictions:    6334/  10000 (63.34%), best:    6354/10000 (63.54%), error:     10652.32031, best error:     10644.30762 on epoch:    45, epoch:   46/50, mu: 0.9674641490, learning_rate: 0.0000426509, weight_decay: 0.0000143289

epoch time: 1424.5925292969s, input_fired_time: 17.1368751526, output_fired_time: 6.7418618202, propagate_forward_time: 553.5484619141, propagate_backward_time: 831.7775268555, weight_update_time: 0.3117312789, other_time: 15.3878784180
epoch time: 116.9622726440s, input_fired_time: 1.2215228081, output_fired_time: 0.0000000000, propagate_forward_time: 113.0124282837, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7283248901
validation[          , genome  6315] predictions:    6339/  10000 (63.39%), best:    6354/10000 (63.54%), error:     10659.70215, best error:     10644.30762 on epoch:    45, epoch:   47/50, mu: 0.9689353108, learning_rate: 0.0000391583, weight_decay: 0.0000133363

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6315_0_r2137330144_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487012
read mu: 0.97031
read mu_delta: 0.934718
read initial_learning_rate: 0.00217149
read learning_rate: 3.59516e-05
read learning_rate_delta: 0.918111
read initial_weight_decay: 0.000389435
read weight_decay: 1.24124e-05
read weight_decay_delta: 0.930725
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0486348
read input_dropout_probability: 0.00134763
read hidden_dropout_probability: 0.111079
read velocity_reset: 1021
read epoch: 48
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 45
read number_validation_images: 10000
read best_validation_predictions: 6354
read best_validation_error: 10644.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6315
read normal distribution: '1 0 0'
generator_str: '113862539'
read generator: 113862539
read generated_by_map:
1 crossover 1
reading 98 nodes.
reading 356 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1440.90075683594s, input_fired_time: 17.2013626098633, output_fired_time: 6.81276798248291, propagate_forward_time: 551.468688964844, propagate_backward_time: 839.031311035156, weight_update_time: 0.314541727304459, other_time: 26.3865966796875
epoch time: 120.144996643066s, input_fired_time: 1.24971175193787, output_fired_time: 0, propagate_forward_time: 115.072906494141, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.82238006591797
validation[          , genome  6315] predictions:    6360/  10000 (63.60%), best:    6360/10000 (63.60%), error:     10636.49414, best error:     10636.49414 on epoch:    48, epoch:   48/50, mu: 0.9703104496, learning_rate: 0.0000359516, weight_decay: 0.0000124124

epoch time: 1379.9783935547s, input_fired_time: 16.8575668335, output_fired_time: 6.5060553551, propagate_forward_time: 526.5053710938, propagate_backward_time: 815.6099853516, weight_update_time: 0.3030267358, other_time: 14.4993896484
epoch time: 107.2023849487s, input_fired_time: 1.1765013933, output_fired_time: 0.0000000000, propagate_forward_time: 103.4424438477, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5834426880
validation[          , genome  6315] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     10580.46582, best error:     10580.46582 on epoch:    49, epoch:   49/50, mu: 0.9715958238, learning_rate: 0.0000330076, weight_decay: 0.0000115525

epoch time: 1427.4223632813s, input_fired_time: 17.1173858643, output_fired_time: 6.7565627098, propagate_forward_time: 549.8730468750, propagate_backward_time: 839.3145141602, weight_update_time: 0.3255414963, other_time: 14.3607788086
epoch time: 129.9718017578s, input_fired_time: 1.3414030075, output_fired_time: 0.0000000000, propagate_forward_time: 126.2242431641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4061584473
validation[          , genome  6315] predictions:    6376/  10000 (63.76%), best:    6340/10000 (63.40%), error:     10592.05859, best error:     10580.46582 on epoch:    49, epoch:   50/50, mu: 0.9727972746, learning_rate: 0.0000303046, weight_decay: 0.0000107522

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 544.5641479492s, input_fired_time: 5.8816795349, output_fired_time: 0.0000000000, propagate_forward_time: 525.5816040039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.1008911133
best training[          , genome  6315] predictions:   35180/  50000 (70.36%), best:    6340/10000 (63.40%), error:     44811.22656, best error:     10580.46582 on epoch:    49, epoch:   51/50, mu: 0.9739202857, learning_rate: 0.0000278230, weight_decay: 0.0000100073
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 108.4068374634s, input_fired_time: 1.1734502316, output_fired_time: 0.0000000000, propagate_forward_time: 104.0041503906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2292404175
   testing[          , genome  6315] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     10580.46582, best error:     10580.46582 on epoch:    49, epoch:   51/50, mu: 0.9739202857, learning_rate: 0.0000278230, weight_decay: 0.0000100073
backpropagation finished successfully!
15:03:24 (8284): called boinc_finish(0)

</stderr_txt>
]]>