Nameexact_genome_1542649503_173_6354_0
Workunit4099682
Created20 Nov 2018, 3:19:56 UTC
Sent20 Nov 2018, 3:30:00 UTC
Report deadline2 Jan 2019, 18:05:19 UTC
Received22 Nov 2018, 11:17:32 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID58206
Run time19 hours 7 min 15 sec
CPU time18 hours 34 min 39 sec
Validate stateValid
Credit10,901.97
Device peak FLOPS2.94 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33

Stderr output

<core_client_version>6.10.58</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354_0_r350304492_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.559627
read mu: 0.559627
read mu_delta: 0.980221
read initial_learning_rate: 0.00541358
read learning_rate: 0.00541358
read learning_rate_delta: 0.911019
read initial_weight_decay: 0.000273969
read weight_decay: 0.000273969
read weight_decay_delta: 0.907458
read batch_size: 125
read epsilon: 1e-07
read alpha: 0.129418
read input_dropout_probability: 0.00180688
read hidden_dropout_probability: 0.0494066
read velocity_reset: 1758
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6354
read normal distribution: '0 1.56687045097351 -1.60382509231567'
generator_str: '1875064798'
read generator: 1875064798
read generated_by_map:
1 split_edge 1
reading 121 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2039023908
post shuffle 1: 1729723175
epoch time: 2973.2353515625s, input_fired_time: 43.8175315856934, output_fired_time: 13.9158563613892, propagate_forward_time: 1316.82250976562, propagate_backward_time: 1567.05432128906, weight_update_time: 0.497993111610413, other_time: 31.6251220703125
epoch time: 266.308929443359s, input_fired_time: 1.38223469257355, output_fired_time: 0, propagate_forward_time: 258.932678222656, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.9940185546875
validation[          , genome  6354] predictions:    5240/  10000 (52.40%), best:    5240/10000 (52.40%), error:     14065.66016, best error:     14065.66016 on epoch:     0, epoch:    0/25, mu: 0.5596265197, learning_rate: 0.0054135816, weight_decay: 0.0002739685

epoch time: 2973.1115722656s, input_fired_time: 36.7530479431, output_fired_time: 15.9925727844, propagate_forward_time: 1320.0294189453, propagate_backward_time: 1567.3665771484, weight_update_time: 0.4966049194, other_time: 32.9699707031
epoch time: 271.6502685547s, input_fired_time: 1.4407962561, output_fired_time: 0.0000000000, propagate_forward_time: 263.2326660156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9768066406
validation[          , genome  6354] predictions:    5536/  10000 (55.36%), best:    5536/10000 (55.36%), error:     12997.36133, best error:     12997.36133 on epoch:     1, epoch:    1/25, mu: 0.5681390762, learning_rate: 0.0049318764, weight_decay: 0.0002486149

epoch time: 2980.6105957031s, input_fired_time: 38.1594467163, output_fired_time: 10.0392103195, propagate_forward_time: 1321.1076660156, propagate_backward_time: 1580.0789794922, weight_update_time: 0.5000151992, other_time: 31.2252197266
epoch time: 270.2870178223s, input_fired_time: 1.4782474041, output_fired_time: 0.0000000000, propagate_forward_time: 262.8229064941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9858703613
validation[          , genome  6354] predictions:    5707/  10000 (57.07%), best:    5707/10000 (57.07%), error:     12602.90918, best error:     12602.90918 on epoch:     2, epoch:    2/25, mu: 0.5764832497, learning_rate: 0.0044930340, weight_decay: 0.0002256076

epoch time: 2967.7072753906s, input_fired_time: 39.0109367371, output_fired_time: 11.0267210007, propagate_forward_time: 1312.9338378906, propagate_backward_time: 1565.7778320312, weight_update_time: 0.4898160696, other_time: 38.9580078125
epoch time: 270.9412536621s, input_fired_time: 1.5496833324, output_fired_time: 0.0000000000, propagate_forward_time: 264.3818054199, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0097656250
validation[          , genome  6354] predictions:    5914/  10000 (59.14%), best:    5914/10000 (59.14%), error:     11836.40918, best error:     11836.40918 on epoch:     3, epoch:    3/25, mu: 0.5846623778, learning_rate: 0.0040932400, weight_decay: 0.0002047294

epoch time: 2972.0417480469s, input_fired_time: 37.0144500732, output_fired_time: 12.9755802155, propagate_forward_time: 1317.2559814453, propagate_backward_time: 1566.9997558594, weight_update_time: 0.4910604954, other_time: 37.7960205078
epoch time: 272.8802490234s, input_fired_time: 1.5270125866, output_fired_time: 0.0000000000, propagate_forward_time: 263.3901062012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9631347656
validation[          , genome  6354] predictions:    5786/  10000 (57.86%), best:    5914/10000 (59.14%), error:     12271.78320, best error:     11836.40918 on epoch:     3, epoch:    4/25, mu: 0.5926797390, learning_rate: 0.0037290200, weight_decay: 0.0001857833

epoch time: 2979.4240722656s, input_fired_time: 39.3481483459, output_fired_time: 13.9068565369, propagate_forward_time: 1310.3843994141, propagate_backward_time: 1578.9888916016, weight_update_time: 0.4951582551, other_time: 36.7958984375
epoch time: 272.2787170410s, input_fired_time: 1.5023794174, output_fired_time: 0.0000000000, propagate_forward_time: 263.7846984863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9916381836
validation[          , genome  6354] predictions:    5914/  10000 (59.14%), best:    5914/10000 (59.14%), error:     11663.20996, best error:     11663.20996 on epoch:     5, epoch:    5/25, mu: 0.6005384922, learning_rate: 0.0033972086, weight_decay: 0.0001685906

epoch time: 2972.6315917969s, input_fired_time: 39.3706893921, output_fired_time: 16.1133289337, propagate_forward_time: 1315.8933105469, propagate_backward_time: 1562.7313232422, weight_update_time: 0.4965427220, other_time: 38.5230712891
epoch time: 271.1202697754s, input_fired_time: 4.5833106041, output_fired_time: 0.0000000000, propagate_forward_time: 261.4919128418, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0450439453
validation[          , genome  6354] predictions:    6030/  10000 (60.30%), best:    6030/10000 (60.30%), error:     11395.74902, best error:     11395.74902 on epoch:     6, epoch:    6/25, mu: 0.6082417965, learning_rate: 0.0030949221, weight_decay: 0.0001529889

epoch time: 2979.2160644531s, input_fired_time: 37.3270988464, output_fired_time: 9.9414405823, propagate_forward_time: 1325.2102050781, propagate_backward_time: 1576.4805908203, weight_update_time: 0.5027241707, other_time: 30.2567138672
epoch time: 269.1488037109s, input_fired_time: 1.5169659853, output_fired_time: 0.0000000000, propagate_forward_time: 261.2430419922, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3887939453
validation[          , genome  6354] predictions:    6060/  10000 (60.60%), best:    6030/10000 (60.30%), error:     11462.77832, best error:     11395.74902 on epoch:     6, epoch:    7/25, mu: 0.6157927513, learning_rate: 0.0028195332, weight_decay: 0.0001388310

epoch time: 2981.4855957031s, input_fired_time: 38.4988632202, output_fired_time: 12.8723068237, propagate_forward_time: 1317.9271240234, propagate_backward_time: 1575.7365722656, weight_update_time: 0.4867120981, other_time: 36.4508056641
epoch time: 271.8329772949s, input_fired_time: 1.6974725723, output_fired_time: 0.0000000000, propagate_forward_time: 263.9988098145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1366882324
validation[          , genome  6354] predictions:    6149/  10000 (61.49%), best:    6149/10000 (61.49%), error:     11210.73633, best error:     11210.73633 on epoch:     8, epoch:    8/25, mu: 0.6231943369, learning_rate: 0.0025686487, weight_decay: 0.0001259833

epoch time: 2976.8798828125s, input_fired_time: 32.4497604370, output_fired_time: 11.8973121643, propagate_forward_time: 1322.0394287109, propagate_backward_time: 1575.2509765625, weight_update_time: 0.5055592060, other_time: 35.2425537109
epoch time: 275.4168090820s, input_fired_time: 1.5456047058, output_fired_time: 0.0000000000, propagate_forward_time: 267.8399963379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0312194824
validation[          , genome  6354] predictions:    6292/  10000 (62.92%), best:    6292/10000 (62.92%), error:     10785.74414, best error:     10785.74414 on epoch:     9, epoch:    9/25, mu: 0.6304495335, learning_rate: 0.0023400881, weight_decay: 0.0001143245

epoch time: 2976.6076660156s, input_fired_time: 38.4904670715, output_fired_time: 14.9826030731, propagate_forward_time: 1311.6500244141, propagate_backward_time: 1576.2281494141, weight_update_time: 0.4878629446, other_time: 35.2563476562
epoch time: 270.8094482422s, input_fired_time: 1.5694005489, output_fired_time: 0.0000000000, propagate_forward_time: 264.1878662109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0521850586
validation[          , genome  6354] predictions:    6361/  10000 (63.61%), best:    6361/10000 (63.61%), error:     10633.11035, best error:     10633.11035 on epoch:    10, epoch:   10/25, mu: 0.6375612617, learning_rate: 0.0021318651, weight_decay: 0.0001037447

epoch time: 2980.3830566406s, input_fired_time: 39.2521591187, output_fired_time: 12.9127016068, propagate_forward_time: 1311.8284912109, propagate_backward_time: 1582.6219482422, weight_update_time: 0.4965968728, other_time: 33.7678222656
epoch time: 275.7416076660s, input_fired_time: 2.5847506523, output_fired_time: 0.0000000000, propagate_forward_time: 267.0140991211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1427612305
validation[          , genome  6354] predictions:    6270/  10000 (62.70%), best:    6361/10000 (63.61%), error:     10831.63965, best error:     10633.11035 on epoch:    10, epoch:   11/25, mu: 0.6445323229, learning_rate: 0.0019421700, weight_decay: 0.0000941440

epoch time: 2974.0556640625s, input_fired_time: 37.4845962524, output_fired_time: 9.8409128189, propagate_forward_time: 1318.0716552734, propagate_backward_time: 1575.2340087891, weight_update_time: 0.4946720898, other_time: 33.4245605469
epoch time: 270.5161132812s, input_fired_time: 1.5807766914, output_fired_time: 0.0000000000, propagate_forward_time: 260.8135375977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1217956543
validation[          , genome  6354] predictions:    6345/  10000 (63.45%), best:    6345/10000 (63.45%), error:     10627.36230, best error:     10627.36230 on epoch:    12, epoch:   12/25, mu: 0.6513654590, learning_rate: 0.0017693540, weight_decay: 0.0000854317

epoch time: 2984.8833007812s, input_fired_time: 43.0773506165, output_fired_time: 10.1932353973, propagate_forward_time: 1302.7364501953, propagate_backward_time: 1586.2000732422, weight_update_time: 0.5037519336, other_time: 42.6762695312
epoch time: 271.8573303223s, input_fired_time: 1.5758887529, output_fired_time: 0.0000000000, propagate_forward_time: 263.5728759766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7085571289
validation[          , genome  6354] predictions:    6352/  10000 (63.52%), best:    6352/10000 (63.52%), error:     10484.41016, best error:     10484.41016 on epoch:    13, epoch:   13/25, mu: 0.6580634713, learning_rate: 0.0016119153, weight_decay: 0.0000775257

epoch time: 2979.4604492188s, input_fired_time: 41.6263008118, output_fired_time: 12.8965196609, propagate_forward_time: 1313.8470458984, propagate_backward_time: 1574.2568359375, weight_update_time: 0.4906530678, other_time: 36.8338623047
epoch time: 275.1131591797s, input_fired_time: 1.5724682808, output_fired_time: 0.0000000000, propagate_forward_time: 268.5322875977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0083923340
validation[          , genome  6354] predictions:    6405/  10000 (64.05%), best:    6405/10000 (64.05%), error:     10484.15430, best error:     10484.15430 on epoch:    14, epoch:   14/25, mu: 0.6646289825, learning_rate: 0.0014684857, weight_decay: 0.0000703513

arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354_0_r350304492_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.559627
read mu: 0.671065
read mu_delta: 0.980221
read initial_learning_rate: 0.00541358
read learning_rate: 0.00133782
read learning_rate_delta: 0.911019
read initial_weight_decay: 0.000273969
read weight_decay: 6.38409e-05
read weight_decay_delta: 0.907458
read batch_size: 125
read epsilon: 1e-07
read alpha: 0.129418
read input_dropout_probability: 0.00180688
read hidden_dropout_probability: 0.0494066
read velocity_reset: 1758
read epoch: 15
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 14
read number_validation_images: 10000
read best_validation_predictions: 6405
read best_validation_error: 10484.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6354
read normal distribution: '0 1.56687045097351 -1.60382509231567'
generator_str: '995852627'
read generator: 995852627
read generated_by_map:
1 split_edge 1
reading 121 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2981.634765625s, input_fired_time: 38.6637649536133, output_fired_time: 12.9088697433472, propagate_forward_time: 1322.38879394531, propagate_backward_time: 1573.15454101562, weight_update_time: 1.4948765039444, other_time: 34.5186767578125
epoch time: 272.42041015625s, input_fired_time: 1.56636428833008, output_fired_time: 0, propagate_forward_time: 264.806518554688, propagate_backward_time: 0, weight_update_time: 0, other_time: 6.04751586914062
validation[          , genome  6354] predictions:    6457/  10000 (64.57%), best:    6457/10000 (64.57%), error:     10166.46582, best error:     10166.46582 on epoch:    15, epoch:   15/25, mu: 0.6710646152, learning_rate: 0.0013378186, weight_decay: 0.0000638409

epoch time: 2969.1049804688s, input_fired_time: 37.6373138428, output_fired_time: 13.9121694565, propagate_forward_time: 1318.1131591797, propagate_backward_time: 1566.8922119141, weight_update_time: 1.4945152998, other_time: 32.5502929688
epoch time: 270.4335327148s, input_fired_time: 3.6121687889, output_fired_time: 0.0000000000, propagate_forward_time: 261.6133117676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2080383301
validation[          , genome  6354] predictions:    6446/  10000 (64.46%), best:    6457/10000 (64.57%), error:     10286.07031, best error:     10166.46582 on epoch:    15, epoch:   16/25, mu: 0.6773729920, learning_rate: 0.0012187783, weight_decay: 0.0000579329

epoch time: 2981.0615234375s, input_fired_time: 35.8887557983, output_fired_time: 13.8429021835, propagate_forward_time: 1336.8277587891, propagate_backward_time: 1561.7651367188, weight_update_time: 0.4894187152, other_time: 32.7369384766
epoch time: 271.7630310059s, input_fired_time: 1.5949407816, output_fired_time: 0.0000000000, propagate_forward_time: 263.2079467773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9601440430
validation[          , genome  6354] predictions:    6335/  10000 (63.35%), best:    6457/10000 (64.57%), error:     10649.29297, best error:     10166.46582 on epoch:    15, epoch:   17/25, mu: 0.6835565567, learning_rate: 0.0011103303, weight_decay: 0.0000525717

epoch time: 2975.0913085938s, input_fired_time: 37.5960617065, output_fired_time: 13.9107141495, propagate_forward_time: 1320.7265625000, propagate_backward_time: 1565.0615234375, weight_update_time: 1.4909607172, other_time: 37.7966308594
epoch time: 273.7251586914s, input_fired_time: 2.5996322632, output_fired_time: 0.0000000000, propagate_forward_time: 264.0720520020, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0534667969
validation[          , genome  6354] predictions:    6580/  10000 (65.80%), best:    6580/10000 (65.80%), error:     10116.58496, best error:     10116.58496 on epoch:    18, epoch:   18/25, mu: 0.6896178126, learning_rate: 0.0010115321, weight_decay: 0.0000477066

epoch time: 2972.0720214844s, input_fired_time: 32.7871093750, output_fired_time: 13.0799207687, propagate_forward_time: 1327.5302734375, propagate_backward_time: 1564.2424316406, weight_update_time: 0.4900426865, other_time: 34.4323730469
epoch time: 270.8552551270s, input_fired_time: 2.5949039459, output_fired_time: 0.0000000000, propagate_forward_time: 263.1995544434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0607910156
validation[          , genome  6354] predictions:    6539/  10000 (65.39%), best:    6580/10000 (65.80%), error:     10163.96484, best error:     10116.58496 on epoch:    18, epoch:   19/25, mu: 0.6955592036, learning_rate: 0.0009215252, weight_decay: 0.0000432917

epoch time: 2978.8837890625s, input_fired_time: 36.7547264099, output_fired_time: 11.9210243225, propagate_forward_time: 1321.5803222656, propagate_backward_time: 1573.0640869141, weight_update_time: 0.5667707920, other_time: 35.5635986328
epoch time: 276.3971557617s, input_fired_time: 1.5936154127, output_fired_time: 0.0000000000, propagate_forward_time: 267.7825622559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0209655762
validation[          , genome  6354] predictions:    6553/  10000 (65.53%), best:    6553/10000 (65.53%), error:     10067.58008, best error:     10067.58008 on epoch:    20, epoch:   20/25, mu: 0.7013830543, learning_rate: 0.0008395271, weight_decay: 0.0000392854

epoch time: 2976.4750976562s, input_fired_time: 37.7280616760, output_fired_time: 10.9669027328, propagate_forward_time: 1327.4291992188, propagate_backward_time: 1561.6842041016, weight_update_time: 0.4937429130, other_time: 38.6668701172
epoch time: 265.9326782227s, input_fired_time: 2.6338012218, output_fired_time: 0.0000000000, propagate_forward_time: 257.3318481445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9670410156
validation[          , genome  6354] predictions:    6553/  10000 (65.53%), best:    6553/10000 (65.53%), error:     10015.74805, best error:     10015.74805 on epoch:    21, epoch:   21/25, mu: 0.7070917487, learning_rate: 0.0007648252, weight_decay: 0.0000356499

epoch time: 2972.2731933594s, input_fired_time: 40.9722633362, output_fired_time: 10.9576578140, propagate_forward_time: 1314.9726562500, propagate_backward_time: 1569.5737304688, weight_update_time: 0.4960699081, other_time: 35.7968750000
epoch time: 273.0927734375s, input_fired_time: 1.6121332645, output_fired_time: 0.0000000000, propagate_forward_time: 263.2726135254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2080383301
validation[          , genome  6354] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:     10012.75000, best error:     10012.75000 on epoch:    22, epoch:   22/25, mu: 0.7126874924, learning_rate: 0.0006967704, weight_decay: 0.0000323508

epoch time: 2974.5454101562s, input_fired_time: 36.7503585815, output_fired_time: 10.9135112762, propagate_forward_time: 1309.1563720703, propagate_backward_time: 1572.8822021484, weight_update_time: 0.4981398582, other_time: 44.8430175781
epoch time: 273.6952819824s, input_fired_time: 3.6343755722, output_fired_time: 0.0000000000, propagate_forward_time: 264.0907592773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9701538086
validation[          , genome  6354] predictions:    6589/  10000 (65.89%), best:    6546/10000 (65.46%), error:     10035.33496, best error:     10012.75000 on epoch:    22, epoch:   23/25, mu: 0.7181725502, learning_rate: 0.0006347711, weight_decay: 0.0000293570

epoch time: 2977.9914550781s, input_fired_time: 36.1219367981, output_fired_time: 13.9154062271, propagate_forward_time: 1321.8669433594, propagate_backward_time: 1571.2551269531, weight_update_time: 0.4938957095, other_time: 34.8320312500
epoch time: 271.6207885742s, input_fired_time: 2.6116936207, output_fired_time: 0.0000000000, propagate_forward_time: 263.9649353027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0441589355
validation[          , genome  6354] predictions:    6525/  10000 (65.25%), best:    6546/10000 (65.46%), error:     10146.96094, best error:     10012.75000 on epoch:    22, epoch:   24/25, mu: 0.7235491276, learning_rate: 0.0005782886, weight_decay: 0.0000266402

epoch time: 2981.5588378906s, input_fired_time: 41.6305503845, output_fired_time: 14.8956489563, propagate_forward_time: 1323.0898437500, propagate_backward_time: 1570.3327636719, weight_update_time: 0.4960249960, other_time: 31.6098632812
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6354_0_r350304492_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.559627
read mu: 0.728819
read mu_delta: 0.980221
read initial_learning_rate: 0.00541358
read learning_rate: 0.000526832
read learning_rate_delta: 0.911019
read initial_weight_decay: 0.000273969
read weight_decay: 2.41749e-05
read weight_decay_delta: 0.907458
read batch_size: 125
read epsilon: 1e-07
read alpha: 0.129418
read input_dropout_probability: 0.00180688
read hidden_dropout_probability: 0.0494066
read velocity_reset: 1758
read epoch: 25
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 22
read number_validation_images: 10000
read best_validation_predictions: 6546
read best_validation_error: 10012.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6354
read normal distribution: '0 1.56687045097351 -1.60382509231567'
generator_str: '1109098460'
read generator: 1109098460
read generated_by_map:
1 split_edge 1
reading 121 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2933.66772460938s, input_fired_time: 38.673770904541, output_fired_time: 9.64034080505371, propagate_forward_time: 1298.02160644531, propagate_backward_time: 1550.94689941406, weight_update_time: 0.475425183773041, other_time: 36.385009765625
epoch time: 267.368438720703s, input_fired_time: 3.59053945541382, output_fired_time: 0, propagate_forward_time: 257.991851806641, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.78604125976562
validation[          , genome  6354] predictions:    6592/  10000 (65.92%), best:    6546/10000 (65.46%), error:     10020.10449, best error:     10012.75000 on epoch:    22, epoch:   25/25, mu: 0.7288193703, learning_rate: 0.0005268320, weight_decay: 0.0000241749

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1338.6820068359s, input_fired_time: 8.9459943771, output_fired_time: 0.0000000000, propagate_forward_time: 1297.4282226562, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 32.3077392578
best training[          , genome  6354] predictions:   37462/  50000 (74.92%), best:    6546/10000 (65.46%), error:     37230.54297, best error:     10012.75000 on epoch:    22, epoch:   26/25, mu: 0.7339853644, learning_rate: 0.0004799540, weight_decay: 0.0000219377
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 266.8688354492s, input_fired_time: 3.5877940655, output_fired_time: 0.0000000000, propagate_forward_time: 255.5088195801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.7722167969
   testing[          , genome  6354] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:     10012.75000, best error:     10012.75000 on epoch:    22, epoch:   26/25, mu: 0.7339853644, learning_rate: 0.0004799540, weight_decay: 0.0000219377
backpropagation finished successfully!
10:29:18 (28183): called boinc_finish(0)

</stderr_txt>
]]>