Nameexact_genome_1540823703_187_5441_0
Workunit4077941
Created29 Oct 2018, 18:02:41 UTC
Sent29 Oct 2018, 18:56:29 UTC
Report deadline18 Dec 2018, 20:45:10 UTC
Received9 Dec 2018, 18:51:41 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64195
Run time23 hours 48 min 30 sec
CPU time23 hours 47 min 32 sec
Validate stateValid
Credit12,518.87
Device peak FLOPS3.80 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size531.51 MB
Peak swap size550.96 MB
Peak disk usage7.85 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
ard_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2955284119
validation[          , genome  5441] predictions:    6054/  10000 (60.54%), best:    6054/10000 (60.54%), error:     11376.18652, best error:     11376.18652 on epoch:    28, epoch:   28/100, mu: 0.9040497541, learning_rate: 0.0003471685, weight_decay: 0.0001600289

epoch time: 501.1320190430s, input_fired_time: 10.8813066483, output_fired_time: 3.7288596630, propagate_forward_time: 170.7102203369, propagate_backward_time: 307.3156738281, weight_update_time: 0.4382773340, other_time: 8.4959716797
epoch time: 36.0238533020s, input_fired_time: 0.6499150991, output_fired_time: 0.0000000000, propagate_forward_time: 34.0834274292, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2905120850
validation[          , genome  5441] predictions:    6109/  10000 (61.09%), best:    6109/10000 (61.09%), error:     11199.78613, best error:     11199.78613 on epoch:    29, epoch:   29/100, mu: 0.9092680812, learning_rate: 0.0003242994, weight_decay: 0.0001523676

epoch time: 501.0932312012s, input_fired_time: 10.8907976151, output_fired_time: 3.7213861942, propagate_forward_time: 170.7248077393, propagate_backward_time: 307.2643737793, weight_update_time: 0.4381908476, other_time: 8.4918518066
epoch time: 36.0126190186s, input_fired_time: 0.6446717381, output_fired_time: 0.0000000000, propagate_forward_time: 34.0839538574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2839927673
validation[          , genome  5441] predictions:    5968/  10000 (59.68%), best:    6109/10000 (61.09%), error:     11587.36133, best error:     11199.78613 on epoch:    29, epoch:   30/100, mu: 0.9141695499, learning_rate: 0.0003029368, weight_decay: 0.0001450731

epoch time: 500.7743835449s, input_fired_time: 10.8893499374, output_fired_time: 3.7210733891, propagate_forward_time: 170.5820770264, propagate_backward_time: 307.1129760742, weight_update_time: 0.4374179542, other_time: 8.4689331055
epoch time: 36.0171852112s, input_fired_time: 0.6561792493, output_fired_time: 0.0000000000, propagate_forward_time: 34.0790901184, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2819175720
validation[          , genome  5441] predictions:    5922/  10000 (59.22%), best:    6109/10000 (61.09%), error:     11681.15430, best error:     11199.78613 on epoch:    29, epoch:   31/100, mu: 0.9187734723, learning_rate: 0.0002829814, weight_decay: 0.0001381278

epoch time: 501.1626892090s, input_fired_time: 10.8965311050, output_fired_time: 3.7246379852, propagate_forward_time: 170.7198181152, propagate_backward_time: 307.3455200195, weight_update_time: 0.4378816783, other_time: 8.4761657715
epoch time: 36.0714530945s, input_fired_time: 0.6602796912, output_fired_time: 0.0000000000, propagate_forward_time: 34.1250495911, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2861251831
validation[          , genome  5441] predictions:    5877/  10000 (58.77%), best:    6109/10000 (61.09%), error:     11939.80371, best error:     11199.78613 on epoch:    29, epoch:   32/100, mu: 0.9230978489, learning_rate: 0.0002643406, weight_decay: 0.0001315150

epoch time: 501.0024108887s, input_fired_time: 10.8926515579, output_fired_time: 3.7183654308, propagate_forward_time: 170.6445922852, propagate_backward_time: 307.2723388672, weight_update_time: 0.4375387728, other_time: 8.4744873047
epoch time: 36.0477638245s, input_fired_time: 0.6511249542, output_fired_time: 0.0000000000, propagate_forward_time: 34.1073760986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2892608643
validation[          , genome  5441] predictions:    6034/  10000 (60.34%), best:    6109/10000 (61.09%), error:     11537.28418, best error:     11199.78613 on epoch:    29, epoch:   33/100, mu: 0.9271596670, learning_rate: 0.0002469276, weight_decay: 0.0001252188

epoch time: 501.2629089355s, input_fired_time: 10.9032154083, output_fired_time: 3.7220556736, propagate_forward_time: 170.8425598145, propagate_backward_time: 307.3093566895, weight_update_time: 0.4377329350, other_time: 8.4857177734
epoch time: 36.0379028320s, input_fired_time: 0.6556431055, output_fired_time: 0.0000000000, propagate_forward_time: 34.0989151001, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2833442688
validation[          , genome  5441] predictions:    6050/  10000 (60.50%), best:    6109/10000 (61.09%), error:     11351.48828, best error:     11199.78613 on epoch:    29, epoch:   34/100, mu: 0.9309749007, learning_rate: 0.0002306618, weight_decay: 0.0001192241

epoch time: 500.9568481445s, input_fired_time: 10.9063282013, output_fired_time: 3.7188291550, propagate_forward_time: 170.6689758301, propagate_backward_time: 307.1789245605, weight_update_time: 0.4372669458, other_time: 8.4837646484
epoch time: 35.9910774231s, input_fired_time: 0.6470542550, output_fired_time: 0.0000000000, propagate_forward_time: 34.0563774109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2876472473
validation[          , genome  5441] predictions:    6018/  10000 (60.18%), best:    6109/10000 (61.09%), error:     11404.02539, best error:     11199.78613 on epoch:    29, epoch:   35/100, mu: 0.9345585108, learning_rate: 0.0002154674, weight_decay: 0.0001135163

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540823703_187_5441.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540823703_187_5441_0_r665708074_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.493527
read mu: 0.937925
read mu_delta: 0.939287
read initial_learning_rate: 0.00233978
read learning_rate: 0.000201274
read learning_rate_delta: 0.934127
read initial_weight_decay: 0.000632063
read weight_decay: 0.000108082
read weight_decay_delta: 0.952126
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0951926
read input_dropout_probability: 0.00115743
read hidden_dropout_probability: 0.0877832
read velocity_reset: 1572
read epoch: 36
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 29
read number_validation_images: 10000
read best_validation_predictions: 6109
read best_validation_error: 11199.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5441
read normal distribution: '1 -0.756985604763031 0.0988513827323914'
generator_str: '2010975448'
read generator: 2010975448
read generated_by_map:
1 add_edge 1
reading 57 nodes.
reading 210 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 971.680419921875s, input_fired_time: 16.8207092285156, output_fired_time: 9.1272611618042, propagate_forward_time: 351.581665039063, propagate_backward_time: 568.645874023438, weight_update_time: 0.865463495254517, other_time: 25.5049438476563
epoch time: 76.289665222168s, input_fired_time: 1.39245748519897, output_fired_time: 0, propagate_forward_time: 70.8811187744141, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.01609039306641
validation[          , genome  5441] predictions:    6051/  10000 (60.51%), best:    6109/10000 (61.09%), error:     11385.11719, best error:     11199.78613 on epoch:    29, epoch:   36/100, mu: 0.9379245639, learning_rate: 0.0002012739, weight_decay: 0.0001080817

epoch time: 951.1488037109s, input_fired_time: 15.3713016510, output_fired_time: 7.8267703056, propagate_forward_time: 345.9393615723, propagate_backward_time: 559.5637817383, weight_update_time: 0.7884121537, other_time: 22.4475708008
epoch time: 71.9359054565s, input_fired_time: 1.1802120209, output_fired_time: 0.0000000000, propagate_forward_time: 67.4109802246, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3447113037
validation[          , genome  5441] predictions:    6099/  10000 (60.99%), best:    6109/10000 (61.09%), error:     11298.88672, best error:     11199.78613 on epoch:    29, epoch:   37/100, mu: 0.9410862327, learning_rate: 0.0001880153, weight_decay: 0.0001029074

epoch time: 952.1563110352s, input_fired_time: 14.5753593445, output_fired_time: 7.2738852501, propagate_forward_time: 349.3554382324, propagate_backward_time: 562.3068237305, weight_update_time: 0.7079765797, other_time: 18.6448364258
epoch time: 74.3848342896s, input_fired_time: 1.2102769613, output_fired_time: 0.0000000000, propagate_forward_time: 70.0038299561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1707305908
validation[          , genome  5441] predictions:    6015/  10000 (60.15%), best:    6109/10000 (61.09%), error:     11345.33496, best error:     11199.78613 on epoch:    29, epoch:   38/100, mu: 0.9440559149, learning_rate: 0.0001756302, weight_decay: 0.0000979808

epoch time: 956.9104003906s, input_fired_time: 14.7117729187, output_fired_time: 7.2910966873, propagate_forward_time: 350.5996398926, propagate_backward_time: 565.0285644531, weight_update_time: 0.7182996273, other_time: 19.2792968750
epoch time: 71.6839675903s, input_fired_time: 1.1448166370, output_fired_time: 0.0000000000, propagate_forward_time: 67.4767456055, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0624084473
validation[          , genome  5441] predictions:    5969/  10000 (59.69%), best:    6109/10000 (61.09%), error:     11446.27734, best error:     11199.78613 on epoch:    29, epoch:   39/100, mu: 0.9468453526, learning_rate: 0.0001640609, weight_decay: 0.0000932900

epoch time: 907.5435180664s, input_fired_time: 13.7760620117, output_fired_time: 6.7021861076, propagate_forward_time: 331.6951293945, propagate_backward_time: 538.6577758789, weight_update_time: 0.6629354358, other_time: 16.7123413086
epoch time: 69.9937438965s, input_fired_time: 1.0888869762, output_fired_time: 0.0000000000, propagate_forward_time: 66.0739746094, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8308792114
validation[          , genome  5441] predictions:    5882/  10000 (58.82%), best:    6109/10000 (61.09%), error:     11656.96191, best error:     11199.78613 on epoch:    29, epoch:   40/100, mu: 0.9494653940, learning_rate: 0.0001532537, weight_decay: 0.0000888238

epoch time: 909.8976440430s, input_fired_time: 13.8340711594, output_fired_time: 6.6949310303, propagate_forward_time: 333.4493713379, propagate_backward_time: 539.2817382813, weight_update_time: 0.6682637930, other_time: 16.6375732422
epoch time: 69.5994873047s, input_fired_time: 1.0942091942, output_fired_time: 0.0000000000, propagate_forward_time: 65.6859741211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8193054199
validation[          , genome  5441] predictions:    6053/  10000 (60.53%), best:    6109/10000 (61.09%), error:     11377.95410, best error:     11199.78613 on epoch:    29, epoch:   41/100, mu: 0.9519263506, learning_rate: 0.0001431584, weight_decay: 0.0000845714

epoch time: 909.7539672852s, input_fired_time: 13.8428182602, output_fired_time: 6.6949534416, propagate_forward_time: 333.1828613281, propagate_backward_time: 539.3490600586, weight_update_time: 0.6689709425, other_time: 16.6842651367
epoch time: 69.5873489380s, input_fired_time: 1.0880403519, output_fired_time: 0.0000000000, propagate_forward_time: 65.7251892090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7741165161
validation[          , genome  5441] predictions:    5989/  10000 (59.89%), best:    6109/10000 (61.09%), error:     11463.94238, best error:     11199.78613 on epoch:    29, epoch:   42/100, mu: 0.9542379379, learning_rate: 0.0001337281, weight_decay: 0.0000805226

epoch time: 906.4986572266s, input_fired_time: 13.7519483566, output_fired_time: 6.6857981682, propagate_forward_time: 330.2397155762, propagate_backward_time: 539.1449584961, weight_update_time: 0.6728273034, other_time: 16.6762084961
epoch time: 71.3530349731s, input_fired_time: 1.1115690470, output_fired_time: 0.0000000000, propagate_forward_time: 67.3690567017, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8724060059
validation[          , genome  5441] predictions:    6168/  10000 (61.68%), best:    6168/10000 (61.68%), error:     11089.77734, best error:     11089.77734 on epoch:    43, epoch:   43/100, mu: 0.9564091563, learning_rate: 0.0001249190, weight_decay: 0.0000766676

epoch time: 922.8612060547s, input_fired_time: 13.8277425766, output_fired_time: 6.7327833176, propagate_forward_time: 337.9690246582, propagate_backward_time: 547.7956542969, weight_update_time: 0.6727191806, other_time: 16.5360107422
epoch time: 71.2517471313s, input_fired_time: 1.1353197098, output_fired_time: 0.0000000000, propagate_forward_time: 67.2985076904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8179168701
validation[          , genome  5441] predictions:    6159/  10000 (61.59%), best:    6159/10000 (61.59%), error:     11018.95996, best error:     11018.95996 on epoch:    44, epoch:   44/100, mu: 0.9584485888, learning_rate: 0.0001166902, weight_decay: 0.0000729972

epoch time: 906.7443237305s, input_fired_time: 13.6585378647, output_fired_time: 6.5813603401, propagate_forward_time: 332.1616516113, propagate_backward_time: 538.7573242188, weight_update_time: 0.6556164026, other_time: 15.5854492188
epoch time: 70.9542312622s, input_fired_time: 1.0760720968, output_fired_time: 0.0000000000, propagate_forward_time: 67.4001693726, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4779891968
validation[          , genome  5441] predictions:    6189/  10000 (61.89%), best:    6189/10000 (61.89%), error:     11011.92285, best error:     11011.92285 on epoch:    45, epoch:   45/100, mu: 0.9603641629, learning_rate: 0.0001090035, weight_decay: 0.0000695025

epoch time: 933.4589233398s, input_fired_time: 13.6927499771, output_fired_time: 6.7109065056, propagate_forward_time: 343.1383972168, propagate_backward_time: 554.5171508789, weight_update_time: 0.6534022689, other_time: 15.3997192383
epoch time: 70.6496582031s, input_fired_time: 1.0781356096, output_fired_time: 0.0000000000, propagate_forward_time: 67.2714614868, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3000640869
validation[          , genome  5441] predictions:    6048/  10000 (60.48%), best:    6189/10000 (61.89%), error:     11447.06738, best error:     11011.92285 on epoch:    45, epoch:   46/100, mu: 0.9621634483, learning_rate: 0.0001018231, weight_decay: 0.0000661751

epoch time: 910.1673583984s, input_fired_time: 13.4666461945, output_fired_time: 6.4642715454, propagate_forward_time: 335.2046203613, propagate_backward_time: 540.6300048828, weight_update_time: 0.6351125240, other_time: 14.4017333984
epoch time: 68.7206878662s, input_fired_time: 1.0527290106, output_fired_time: 0.0000000000, propagate_forward_time: 65.2335968018, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4343643188
validation[          , genome  5441] predictions:    6080/  10000 (60.80%), best:    6189/10000 (61.89%), error:     11208.65137, best error:     11011.92285 on epoch:    45, epoch:   47/100, mu: 0.9638534784, learning_rate: 0.0000951157, weight_decay: 0.0000630070

epoch time: 884.9392089844s, input_fired_time: 13.4491920471, output_fired_time: 6.3629283905, propagate_forward_time: 325.0666809082, propagate_backward_time: 525.2332153320, weight_update_time: 0.6371834874, other_time: 14.8272094727
epoch time: 69.1023406982s, input_fired_time: 1.0609283447, output_fired_time: 0.0000000000, propagate_forward_time: 65.6520919800, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3893203735
validation[          , genome  5441] predictions:    6141/  10000 (61.41%), best:    6189/10000 (61.89%), error:     11098.92480, best error:     11011.92285 on epoch:    45, epoch:   48/100, mu: 0.9654409289, learning_rate: 0.0000888501, weight_decay: 0.0000599906

epoch time: 911.9759521484s, input_fired_time: 13.5802602768, output_fired_time: 6.5444288254, propagate_forward_time: 335.6878662109, propagate_backward_time: 541.2847290039, weight_update_time: 0.6416344643, other_time: 14.8786621094
epoch time: 71.7170028687s, input_fired_time: 1.0643312931, output_fired_time: 0.0000000000, propagate_forward_time: 68.4092712402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2434005737
validation[          , genome  5441] predictions:    6195/  10000 (61.95%), best:    6195/10000 (61.95%), error:     10976.30273, best error:     10976.30273 on epoch:    49, epoch:   49/100, mu: 0.9669319987, learning_rate: 0.0000829973, weight_decay: 0.0000571185

epoch time: 918.4122314453s, input_fired_time: 13.4207124710, output_fired_time: 6.4417018890, propagate_forward_time: 339.9893798828, propagate_backward_time: 545.3508911133, weight_update_time: 0.6288508177, other_time: 13.2095336914
epoch time: 70.6311798096s, input_fired_time: 1.0365071297, output_fired_time: 0.0000000000, propagate_forward_time: 67.6323471069, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9623260498
validation[          , genome  5441] predictions:    6196/  10000 (61.96%), best:    6196/10000 (61.96%), error:     10922.97266, best error:     10922.97266 on epoch:    50, epoch:   50/100, mu: 0.9683325291, learning_rate: 0.0000775300, weight_decay: 0.0000543840

epoch time: 914.9800415039s, input_fired_time: 13.4212417603, output_fired_time: 6.4550313950, propagate_forward_time: 338.6300048828, propagate_backward_time: 543.6317138672, weight_update_time: 0.6237758994, other_time: 12.8420410156
epoch time: 70.2676773071s, input_fired_time: 1.0378904343, output_fired_time: 0.0000000000, propagate_forward_time: 67.0979385376, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1318511963
validation[          , genome  5441] predictions:    6180/  10000 (61.80%), best:    6196/10000 (61.96%), error:     11035.05469, best error:     10922.97266 on epoch:    50, epoch:   51/100, mu: 0.9696480036, learning_rate: 0.0000724229, weight_decay: 0.0000517804

epoch time: 899.8768920898s, input_fired_time: 13.2282419205, output_fired_time: 6.3163242340, propagate_forward_time: 333.1062927246, propagate_backward_time: 534.8571166992, weight_update_time: 0.6168605089, other_time: 12.3688354492
epoch time: 67.8509063721s, input_fired_time: 1.0202617645, output_fired_time: 0.0000000000, propagate_forward_time: 64.8311920166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9994506836
validation[          , genome  5441] predictions:    6245/  10000 (62.45%), best:    6245/10000 (62.45%), error:     10907.44531, best error:     10907.44531 on epoch:    52, epoch:   52/100, mu: 0.9708836079, learning_rate: 0.0000676521, weight_decay: 0.0000493015

epoch time: 891.0072021484s, input_fired_time: 13.1107711792, output_fired_time: 6.1732606888, propagate_forward_time: 329.1563720703, propagate_backward_time: 530.4069213867, weight_update_time: 0.6106122732, other_time: 12.1598510742
epoch time: 68.1924972534s, input_fired_time: 1.0151380301, output_fired_time: 0.0000000000, propagate_forward_time: 65.1827697754, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9945907593
validation[          , genome  5441] predictions:    6247/  10000 (62.47%), best:    6245/10000 (62.45%), error:     10936.01855, best error:     10907.44531 on epoch:    52, epoch:   53/100, mu: 0.9720442295, learning_rate: 0.0000631957, weight_decay: 0.0000469412

epoch time: 896.5245361328s, input_fired_time: 13.1189041138, output_fired_time: 6.1799616814, propagate_forward_time: 332.1296081543, propagate_backward_time: 533.2926025391, weight_update_time: 0.6040074825, other_time: 11.8034667969
epoch time: 69.5439605713s, input_fired_time: 1.0374038219, output_fired_time: 0.0000000000, propagate_forward_time: 66.5624771118, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9440765381
validation[          , genome  5441] predictions:    6167/  10000 (61.67%), best:    6245/10000 (62.45%), error:     11007.68164, best error:     10907.44531 on epoch:    52, epoch:   54/100, mu: 0.9731343985, learning_rate: 0.0000590328, weight_decay: 0.0000446939

epoch time: 900.6198120117s, input_fired_time: 13.1738967896, output_fired_time: 6.2762498856, propagate_forward_time: 333.3238525391, propagate_backward_time: 535.8324584961, weight_update_time: 0.6110317707, other_time: 12.0133666992
epoch time: 68.9602584839s, input_fired_time: 1.0189647675, output_fired_time: 0.0000000000, propagate_forward_time: 66.0373153687, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9039764404
validation[          , genome  5441] predictions:    6234/  10000 (62.34%), best:    6234/10000 (62.34%), error:     10881.36035, best error:     10881.36035 on epoch:    55, epoch:   55/100, mu: 0.9741583467, learning_rate: 0.0000551441, weight_decay: 0.0000425542

epoch time: 882.4827880859s, input_fired_time: 13.0675764084, output_fired_time: 6.1321196556, propagate_forward_time: 325.9114990234, propagate_backward_time: 525.5989990234, weight_update_time: 0.6001120806, other_time: 11.7725830078
epoch time: 66.0064620972s, input_fired_time: 0.9878475070, output_fired_time: 0.0000000000, propagate_forward_time: 63.1217842102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8968315125
validation[          , genome  5441] predictions:    6263/  10000 (62.63%), best:    6263/10000 (62.63%), error:     10877.61621, best error:     10877.61621 on epoch:    56, epoch:   56/100, mu: 0.9751201272, learning_rate: 0.0000515116, weight_decay: 0.0000405169

epoch time: 860.7661132813s, input_fired_time: 12.9660644531, output_fired_time: 6.0149598122, propagate_forward_time: 317.1369018555, propagate_backward_time: 513.1544799805, weight_update_time: 0.5889746547, other_time: 11.4937133789
epoch time: 66.8556671143s, input_fired_time: 1.0026936531, output_fired_time: 0.0000000000, propagate_forward_time: 63.9337882996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9191856384
validation[          , genome  5441] predictions:    6133/  10000 (61.33%), best:    6263/10000 (62.63%), error:     11117.72168, best error:     10877.61621 on epoch:    56, epoch:   57/100, mu: 0.9760235548, learning_rate: 0.0000481184, weight_decay: 0.0000385772

epoch time: 844.3738403320s, input_fired_time: 12.8502330780, output_fired_time: 5.8496165276, propagate_forward_time: 310.4557495117, propagate_backward_time: 503.9586486816, weight_update_time: 0.5818595290, other_time: 11.2596130371
epoch time: 62.9733200073s, input_fired_time: 0.9758828282, output_fired_time: 0.0000000000, propagate_forward_time: 60.1338729858, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8635635376
validation[          , genome  5441] predictions:    6274/  10000 (62.74%), best:    6274/10000 (62.74%), error:     10814.75781, best error:     10814.75781 on epoch:    58, epoch:   58/100, mu: 0.9768720865, learning_rate: 0.0000449487, weight_decay: 0.0000367304

epoch time: 834.6732788086s, input_fired_time: 12.7727565765, output_fired_time: 5.7883663177, propagate_forward_time: 305.2793273926, propagate_backward_time: 499.6999206543, weight_update_time: 0.5786034465, other_time: 11.1328735352
epoch time: 62.2968597412s, input_fired_time: 0.9679948092, output_fired_time: 0.0000000000, propagate_forward_time: 59.5677146912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7611503601
validation[          , genome  5441] predictions:    6277/  10000 (62.77%), best:    6274/10000 (62.74%), error:     10821.67090, best error:     10814.75781 on epoch:    58, epoch:   59/100, mu: 0.9776691198, learning_rate: 0.0000419878, weight_decay: 0.0000349719

epoch time: 793.4012451172s, input_fired_time: 12.5600624084, output_fired_time: 5.5152297020, propagate_forward_time: 290.3309326172, propagate_backward_time: 474.1645507813, weight_update_time: 0.5631714463, other_time: 10.8304443359
epoch time: 56.9702644348s, input_fired_time: 0.9003573060, output_fired_time: 0.0000000000, propagate_forward_time: 54.3750114441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6948966980
validation[          , genome  5441] predictions:    6187/  10000 (61.87%), best:    6274/10000 (62.74%), error:     10907.48145, best error:     10814.75781 on epoch:    58, epoch:   60/100, mu: 0.9784177542, learning_rate: 0.0000392219, weight_decay: 0.0000332976

epoch time: 803.1885986328s, input_fired_time: 12.6174659729, output_fired_time: 5.5993824005, propagate_forward_time: 294.2004394531, propagate_backward_time: 479.7858886719, weight_update_time: 0.5675996542, other_time: 10.9854125977
epoch time: 60.0819892883s, input_fired_time: 0.9277265668, output_fired_time: 0.0000000000, propagate_forward_time: 57.4008178711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7534446716
validation[          , genome  5441] predictions:    6266/  10000 (62.66%), best:    6266/10000 (62.66%), error:     10796.14355, best error:     10796.14355 on epoch:    61, epoch:   61/100, mu: 0.9791209698, learning_rate: 0.0000366382, weight_decay: 0.0000317035

epoch time: 784.9686889648s, input_fired_time: 12.4633760452, output_fired_time: 5.4294524193, propagate_forward_time: 287.0598144531, propagate_backward_time: 469.2561950684, weight_update_time: 0.5598182678, other_time: 10.7598571777
epoch time: 59.2446403503s, input_fired_time: 0.9310573339, output_fired_time: 0.0000000000, propagate_forward_time: 56.5994071960, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7141761780
validation[          , genome  5441] predictions:    6240/  10000 (62.40%), best:    6266/10000 (62.66%), error:     10890.35156, best error:     10796.14355 on epoch:    61, epoch:   62/100, mu: 0.9797814488, learning_rate: 0.0000342248, weight_decay: 0.0000301857

epoch time: 767.9832763672s, input_fired_time: 12.4422826767, output_fired_time: 5.3786578178, propagate_forward_time: 276.5071105957, propagate_backward_time: 463.1051025391, weight_update_time: 0.5457499027, other_time: 10.5501403809
epoch time: 56.8612518311s, input_fired_time: 0.9055798054, output_fired_time: 0.0000000000, propagate_forward_time: 54.2737274170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6819458008
validation[          , genome  5441] predictions:    6180/  10000 (61.80%), best:    6266/10000 (62.66%), error:     11069.71777, best error:     10796.14355 on epoch:    61, epoch:   63/100, mu: 0.9804018736, learning_rate: 0.0000319703, weight_decay: 0.0000287406

epoch time: 774.9956665039s, input_fired_time: 12.3971767426, output_fired_time: 5.2684369087, propagate_forward_time: 282.0122070313, propagate_backward_time: 464.6190490723, weight_update_time: 0.5559499264, other_time: 10.6988220215
epoch time: 62.5673065186s, input_fired_time: 0.9657513499, output_fired_time: 0.0000000000, propagate_forward_time: 59.8234176636, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7781372070
validation[          , genome  5441] predictions:    6331/  10000 (63.31%), best:    6331/10000 (63.31%), error:     10714.02832, best error:     10714.02832 on epoch:    64, epoch:   64/100, mu: 0.9809846282, learning_rate: 0.0000298643, weight_decay: 0.0000273647

epoch time: 784.6834106445s, input_fired_time: 12.4979162216, output_fired_time: 5.4783592224, propagate_forward_time: 285.7127380371, propagate_backward_time: 470.1975402832, weight_update_time: 0.5636076927, other_time: 10.7968750000
epoch time: 58.8544654846s, input_fired_time: 0.9299532175, output_fired_time: 0.0000000000, propagate_forward_time: 56.1813926697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7431182861
validation[          , genome  5441] predictions:    6248/  10000 (62.48%), best:    6331/10000 (63.31%), error:     10848.47168, best error:     10714.02832 on epoch:    64, epoch:   65/100, mu: 0.9815319777, learning_rate: 0.0000278970, weight_decay: 0.0000260546

epoch time: 765.8776855469s, input_fired_time: 12.4304475784, output_fired_time: 5.3569798470, propagate_forward_time: 278.1433715820, propagate_backward_time: 459.3504028320, weight_update_time: 0.5506289601, other_time: 10.5964965820
epoch time: 60.8063201904s, input_fired_time: 0.9486141801, output_fired_time: 0.0000000000, propagate_forward_time: 58.0952339172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7624702454
validation[          , genome  5441] predictions:    6276/  10000 (62.76%), best:    6331/10000 (63.31%), error:     10793.33984, best error:     10714.02832 on epoch:    64, epoch:   66/100, mu: 0.9820461273, learning_rate: 0.0000260594, weight_decay: 0.0000248073

epoch time: 771.2916259766s, input_fired_time: 12.4493160248, output_fired_time: 5.4015679359, propagate_forward_time: 281.0281982422, propagate_backward_time: 461.7052001953, weight_update_time: 0.5548459291, other_time: 10.7073364258
epoch time: 54.4777107239s, input_fired_time: 0.8750952482, output_fired_time: 0.0000000000, propagate_forward_time: 51.9722061157, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6304092407
validation[          , genome  5441] predictions:    6319/  10000 (63.19%), best:    6319/10000 (63.19%), error:     10710.57227, best error:     10710.57227 on epoch:    67, epoch:   67/100, mu: 0.9825290442, learning_rate: 0.0000243428, weight_decay: 0.0000236196

epoch time: 770.3587646484s, input_fired_time: 12.4620380402, output_fired_time: 5.4300675392, propagate_forward_time: 280.5701599121, propagate_backward_time: 461.2159118652, weight_update_time: 0.5521897078, other_time: 10.6806030273
epoch time: 55.6114120483s, input_fired_time: 0.9009873867, output_fired_time: 0.0000000000, propagate_forward_time: 53.0242538452, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6861724854
validation[          , genome  5441] predictions:    6353/  10000 (63.53%), best:    6353/10000 (63.53%), error:     10653.19531, best error:     10653.19531 on epoch:    68, epoch:   68/100, mu: 0.9829826355, learning_rate: 0.0000227392, weight_decay: 0.0000224888

epoch time: 759.9536743164s, input_fired_time: 12.4326858521, output_fired_time: 5.3125700951, propagate_forward_time: 276.1982727051, propagate_backward_time: 455.5148620605, weight_update_time: 0.5433530807, other_time: 10.4953002930
epoch time: 57.1448287964s, input_fired_time: 0.9046222568, output_fired_time: 0.0000000000, propagate_forward_time: 54.5532188416, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6869888306
validation[          , genome  5441] predictions:    6272/  10000 (62.72%), best:    6353/10000 (63.53%), error:     10818.92676, best error:     10653.19531 on epoch:    68, epoch:   69/100, mu: 0.9834086895, learning_rate: 0.0000212413, weight_decay: 0.0000214122

epoch time: 742.1604003906s, input_fired_time: 12.2812728882, output_fired_time: 5.1879343987, propagate_forward_time: 265.5013427734, propagate_backward_time: 448.7933349609, weight_update_time: 0.5384344459, other_time: 10.3965454102
epoch time: 55.8758392334s, input_fired_time: 0.8965239525, output_fired_time: 0.0000000000, propagate_forward_time: 53.3181343079, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6611824036
validation[          , genome  5441] predictions:    6367/  10000 (63.67%), best:    6367/10000 (63.67%), error:     10631.52734, best error:     10631.52734 on epoch:    70, epoch:   70/100, mu: 0.9838088751, learning_rate: 0.0000198421, weight_decay: 0.0000203871

epoch time: 746.2517089844s, input_fired_time: 12.3220529556, output_fired_time: 5.2331919670, propagate_forward_time: 268.9562988281, propagate_backward_time: 449.2472839355, weight_update_time: 0.5440208316, other_time: 10.4928283691
epoch time: 57.6886940002s, input_fired_time: 0.9243423939, output_fired_time: 0.0000000000, propagate_forward_time: 55.0570030212, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7073478699
validation[          , genome  5441] predictions:    6267/  10000 (62.67%), best:    6367/10000 (63.67%), error:     10724.82227, best error:     10631.52734 on epoch:    70, epoch:   71/100, mu: 0.9841847420, learning_rate: 0.0000185350, weight_decay: 0.0000194111

epoch time: 758.4328002930s, input_fired_time: 12.3669233322, output_fired_time: 5.2245593071, propagate_forward_time: 273.4223022461, propagate_backward_time: 456.8336181641, weight_update_time: 0.5480615497, other_time: 10.5853881836
epoch time: 60.0167465210s, input_fired_time: 0.9404912591, output_fired_time: 0.0000000000, propagate_forward_time: 57.3501548767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7261009216
validation[          , genome  5441] predictions:    6315/  10000 (63.15%), best:    6367/10000 (63.67%), error:     10706.25977, best error:     10631.52734 on epoch:    70, epoch:   72/100, mu: 0.9845377803, learning_rate: 0.0000173141, weight_decay: 0.0000184818

epoch time: 783.8740234375s, input_fired_time: 12.5768375397, output_fired_time: 5.5550546646, propagate_forward_time: 284.9000854492, propagate_backward_time: 469.8980407715, weight_update_time: 0.5647784472, other_time: 10.9440002441
epoch time: 57.6308784485s, input_fired_time: 0.9261290431, output_fired_time: 0.0000000000, propagate_forward_time: 54.9778747559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7268753052
validation[          , genome  5441] predictions:    6375/  10000 (63.75%), best:    6367/10000 (63.67%), error:     10658.00293, best error:     10631.52734 on epoch:    70, epoch:   73/100, mu: 0.9848694205, learning_rate: 0.0000161735, weight_decay: 0.0000175970

epoch time: 794.0792236328s, input_fired_time: 12.6159620285, output_fired_time: 5.5183043480, propagate_forward_time: 288.4825744629, propagate_backward_time: 476.6652832031, weight_update_time: 0.5638515353, other_time: 10.7970886230
epoch time: 61.8890876770s, input_fired_time: 0.9675634503, output_fired_time: 0.0000000000, propagate_forward_time: 59.1115608215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8099632263
validation[          , genome  5441] predictions:    6354/  10000 (63.54%), best:    6354/10000 (63.54%), error:     10599.42285, best error:     10599.42285 on epoch:    74, epoch:   74/100, mu: 0.9851809144, learning_rate: 0.0000151081, weight_decay: 0.0000167545

epoch time: 808.9639282227s, input_fired_time: 12.6724767685, output_fired_time: 5.6775283813, propagate_forward_time: 294.3894958496, propagate_backward_time: 485.1488952637, weight_update_time: 0.5741714239, other_time: 11.0755004883
epoch time: 59.6728591919s, input_fired_time: 0.9506105781, output_fired_time: 0.0000000000, propagate_forward_time: 56.9738731384, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7483749390
validation[          , genome  5441] predictions:    6356/  10000 (63.56%), best:    6354/10000 (63.54%), error:     10606.12793, best error:     10599.42285 on epoch:    74, epoch:   75/100, mu: 0.9854735136, learning_rate: 0.0000141129, weight_decay: 0.0000159524

epoch time: 794.0802001953s, input_fired_time: 12.6190471649, output_fired_time: 5.5846295357, propagate_forward_time: 289.4897766113, propagate_backward_time: 475.5245056152, weight_update_time: 0.5663459897, other_time: 10.8622436523
epoch time: 59.3670959473s, input_fired_time: 0.9358647466, output_fired_time: 0.0000000000, propagate_forward_time: 56.7596969604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6715354919
validation[          , genome  5441] predictions:    6263/  10000 (62.63%), best:    6354/10000 (63.54%), error:     10732.48633, best error:     10599.42285 on epoch:    74, epoch:   76/100, mu: 0.9857483506, learning_rate: 0.0000131833, weight_decay: 0.0000151887

epoch time: 797.0941162109s, input_fired_time: 12.5964260101, output_fired_time: 5.5564393997, propagate_forward_time: 291.8502502441, propagate_backward_time: 476.2217407227, weight_update_time: 0.5701463223, other_time: 10.8692321777
epoch time: 58.4834327698s, input_fired_time: 0.9384314418, output_fired_time: 0.0000000000, propagate_forward_time: 55.8616714478, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6833305359
validation[          , genome  5441] predictions:    6396/  10000 (63.96%), best:    6396/10000 (63.96%), error:     10559.23730, best error:     10559.23730 on epoch:    77, epoch:   77/100, mu: 0.9860064983, learning_rate: 0.0000123148, weight_decay: 0.0000144616

epoch time: 781.3506469727s, input_fired_time: 12.5200605392, output_fired_time: 5.4550814629, propagate_forward_time: 284.9086914063, propagate_backward_time: 467.8264770508, weight_update_time: 0.5560256839, other_time: 10.6403198242
epoch time: 56.9480590820s, input_fired_time: 0.9201845527, output_fired_time: 0.0000000000, propagate_forward_time: 54.3447723389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6831016541
validation[          , genome  5441] predictions:    6376/  10000 (63.76%), best:    6396/10000 (63.96%), error:     10573.18555, best error:     10559.23730 on epoch:    77, epoch:   78/100, mu: 0.9862489700, learning_rate: 0.0000115036, weight_decay: 0.0000137692

epoch time: 787.7927856445s, input_fired_time: 12.5742340088, output_fired_time: 5.5434818268, propagate_forward_time: 288.2434997559, propagate_backward_time: 470.6468811035, weight_update_time: 0.5620952249, other_time: 10.7847290039
epoch time: 59.3598022461s, input_fired_time: 0.9419830441, output_fired_time: 0.0000000000, propagate_forward_time: 56.6935844421, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7242355347
validation[          , genome  5441] predictions:    6374/  10000 (63.74%), best:    6396/10000 (63.96%), error:     10596.99609, best error:     10559.23730 on epoch:    77, epoch:   79/100, mu: 0.9864767194, learning_rate: 0.0000107458, weight_decay: 0.0000131100

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540823703_187_5441.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540823703_187_5441_0_r665708074_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.493527
read mu: 0.986691
read mu_delta: 0.939287
read initial_learning_rate: 0.00233978
read learning_rate: 1.0038e-05
read learning_rate_delta: 0.934127
read initial_weight_decay: 0.000632063
read weight_decay: 1.24824e-05
read weight_decay_delta: 0.952126
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0951926
read input_dropout_probability: 0.00115743
read hidden_dropout_probability: 0.0877832
read velocity_reset: 1572
read epoch: 80
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 77
read number_validation_images: 10000
read best_validation_predictions: 6396
read best_validation_error: 10559.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5441
read normal distribution: '1 -0.756985604763031 0.0988513827323914'
generator_str: '2058794093'
read generator: 2058794093
read generated_by_map:
1 add_edge 1
reading 57 nodes.
reading 210 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 919.619689941406s, input_fired_time: 20.2937450408936, output_fired_time: 11.6023511886597, propagate_forward_time: 317.105865478516, propagate_backward_time: 534.503112792969, weight_update_time: 1.09403777122498, other_time: 36.1145629882813
epoch time: 71.6776809692383s, input_fired_time: 2.14658617973328, output_fired_time: 0, propagate_forward_time: 62.5980186462402, propagate_backward_time: 0, weight_update_time: 0, other_time: 6.93307876586914
validation[          , genome  5441] predictions:    6382/  10000 (63.82%), best:    6382/10000 (63.82%), error:     10541.36426, best error:     10541.36426 on epoch:    80, epoch:   80/100, mu: 0.9866906404, learning_rate: 0.0000100380, weight_decay: 0.0000124824

epoch time: 934.6707153320s, input_fired_time: 20.3321285248, output_fired_time: 11.4426269531, propagate_forward_time: 321.0545043945, propagate_backward_time: 545.1727905273, weight_update_time: 1.1149214506, other_time: 36.6686401367
epoch time: 73.4122390747s, input_fired_time: 2.1043398380, output_fired_time: 0.0000000000, propagate_forward_time: 64.6420898438, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6658096313
validation[          , genome  5441] predictions:    6352/  10000 (63.52%), best:    6382/10000 (63.82%), error:     10622.69727, best error:     10541.36426 on epoch:    80, epoch:   81/100, mu: 0.9868915677, learning_rate: 0.0000093767, weight_decay: 0.0000118848

epoch time: 955.2752075195s, input_fired_time: 20.9093551636, output_fired_time: 11.8610143661, propagate_forward_time: 332.4473571777, propagate_backward_time: 552.5354003906, weight_update_time: 1.1442010403, other_time: 37.5220947266
epoch time: 73.3026657104s, input_fired_time: 2.1536478996, output_fired_time: 0.0000000000, propagate_forward_time: 64.8048248291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3441925049
validation[          , genome  5441] predictions:    6339/  10000 (63.39%), best:    6382/10000 (63.82%), error:     10562.91699, best error:     10541.36426 on epoch:    80, epoch:   82/100, mu: 0.9870802760, learning_rate: 0.0000087591, weight_decay: 0.0000113158

epoch time: 945.8665161133s, input_fired_time: 20.4451465607, output_fired_time: 11.5037078857, propagate_forward_time: 329.0090637207, propagate_backward_time: 548.8509521484, weight_update_time: 1.0826545954, other_time: 36.0576171875
epoch time: 73.0651855469s, input_fired_time: 2.0755467415, output_fired_time: 0.0000000000, propagate_forward_time: 64.6809463501, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3086929321
validation[          , genome  5441] predictions:    6367/  10000 (63.67%), best:    6382/10000 (63.82%), error:     10561.93750, best error:     10541.36426 on epoch:    80, epoch:   83/100, mu: 0.9872575402, learning_rate: 0.0000081821, weight_decay: 0.0000107741

epoch time: 952.5773315430s, input_fired_time: 20.5260772705, output_fired_time: 11.7506074905, propagate_forward_time: 328.7088928223, propagate_backward_time: 554.1580810547, weight_update_time: 1.1081193686, other_time: 37.4337158203
epoch time: 69.7331542969s, input_fired_time: 1.8906368017, output_fired_time: 0.0000000000, propagate_forward_time: 62.4433784485, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3991355896
validation[          , genome  5441] predictions:    6356/  10000 (63.56%), best:    6382/10000 (63.82%), error:     10567.23535, best error:     10541.36426 on epoch:    80, epoch:   84/100, mu: 0.9874240160, learning_rate: 0.0000076431, weight_decay: 0.0000102583

epoch time: 909.4333496094s, input_fired_time: 20.0191078186, output_fired_time: 11.1168727875, propagate_forward_time: 314.6104431152, propagate_backward_time: 528.1193847656, weight_update_time: 1.0653636456, other_time: 35.5675048828
epoch time: 72.3350143433s, input_fired_time: 2.1737179756, output_fired_time: 0.0000000000, propagate_forward_time: 62.9198341370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2414588928
validation[          , genome  5441] predictions:    6319/  10000 (63.19%), best:    6382/10000 (63.82%), error:     10674.46973, best error:     10541.36426 on epoch:    80, epoch:   85/100, mu: 0.9875804186, learning_rate: 0.0000071396, weight_decay: 0.0000097672

epoch time: 908.2686767578s, input_fired_time: 20.1740684509, output_fired_time: 11.5457963943, propagate_forward_time: 310.6896972656, propagate_backward_time: 528.6635131836, weight_update_time: 1.1063849926, other_time: 37.1956176758
epoch time: 67.5835113525s, input_fired_time: 1.9598084688, output_fired_time: 0.0000000000, propagate_forward_time: 59.2649078369, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3587951660
validation[          , genome  5441] predictions:    6370/  10000 (63.70%), best:    6382/10000 (63.82%), error:     10555.53711, best error:     10541.36426 on epoch:    80, epoch:   86/100, mu: 0.9877273440, learning_rate: 0.0000066693, weight_decay: 0.0000092996

epoch time: 908.1644287109s, input_fired_time: 20.2593688965, output_fired_time: 11.5080413818, propagate_forward_time: 312.2045898438, propagate_backward_time: 527.9541625977, weight_update_time: 1.0814242363, other_time: 36.2382202148
epoch time: 67.8816528320s, input_fired_time: 1.9373610020, output_fired_time: 0.0000000000, propagate_forward_time: 59.7420539856, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2022361755
validation[          , genome  5441] predictions:    6387/  10000 (63.87%), best:    6382/10000 (63.82%), error:     10567.92871, best error:     10541.36426 on epoch:    80, epoch:   87/100, mu: 0.9878653288, learning_rate: 0.0000062300, weight_decay: 0.0000088544

epoch time: 905.0756835938s, input_fired_time: 20.0986900330, output_fired_time: 11.5120725632, propagate_forward_time: 308.3239440918, propagate_backward_time: 528.9837036133, weight_update_time: 1.0786339045, other_time: 36.1572875977
epoch time: 70.8046264648s, input_fired_time: 2.1563425064, output_fired_time: 0.0000000000, propagate_forward_time: 61.7973594666, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8509254456
validation[          , genome  5441] predictions:    6351/  10000 (63.51%), best:    6382/10000 (63.82%), error:     10588.35742, best error:     10541.36426 on epoch:    80, epoch:   88/100, mu: 0.9879949093, learning_rate: 0.0000058196, weight_decay: 0.0000084305

epoch time: 903.9472656250s, input_fired_time: 20.3435497284, output_fired_time: 11.4208068848, propagate_forward_time: 312.5020751953, propagate_backward_time: 523.4657592773, weight_update_time: 1.0834101439, other_time: 36.2150268555
epoch time: 65.3048706055s, input_fired_time: 1.9507514238, output_fired_time: 0.0000000000, propagate_forward_time: 57.5122108459, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8419075012
validation[          , genome  5441] predictions:    6325/  10000 (63.25%), best:    6382/10000 (63.82%), error:     10732.45703, best error:     10541.36426 on epoch:    80, epoch:   89/100, mu: 0.9881166220, learning_rate: 0.0000054362, weight_decay: 0.0000080269

epoch time: 916.2042846680s, input_fired_time: 20.5170078278, output_fired_time: 11.8120908737, propagate_forward_time: 312.4228515625, propagate_backward_time: 534.6605224609, weight_update_time: 1.0959244967, other_time: 36.7918090820
epoch time: 69.0098953247s, input_fired_time: 2.0349295139, output_fired_time: 0.0000000000, propagate_forward_time: 60.6306152344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3443527222
validation[          , genome  5441] predictions:    6350/  10000 (63.50%), best:    6382/10000 (63.82%), error:     10612.29102, best error:     10541.36426 on epoch:    80, epoch:   90/100, mu: 0.9882309437, learning_rate: 0.0000050781, weight_decay: 0.0000076426

epoch time: 900.0241088867s, input_fired_time: 19.8997173309, output_fired_time: 11.3580703735, propagate_forward_time: 305.3670654297, propagate_backward_time: 526.7149658203, weight_update_time: 1.0779800415, other_time: 36.6842651367
epoch time: 68.6327819824s, input_fired_time: 2.0609123707, output_fired_time: 0.0000000000, propagate_forward_time: 59.9230003357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6488685608
validation[          , genome  5441] predictions:    6374/  10000 (63.74%), best:    6382/10000 (63.82%), error:     10692.85449, best error:     10541.36426 on epoch:    80, epoch:   91/100, mu: 0.9883383512, learning_rate: 0.0000047436, weight_decay: 0.0000072767

epoch time: 894.1433715820s, input_fired_time: 20.1897201538, output_fired_time: 11.4442367554, propagate_forward_time: 306.7147216797, propagate_backward_time: 519.2443237305, weight_update_time: 1.0968747139, other_time: 36.5504150391
epoch time: 66.9724044800s, input_fired_time: 1.9903267622, output_fired_time: 0.0000000000, propagate_forward_time: 58.8267936707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1552848816
validation[          , genome  5441] predictions:    6359/  10000 (63.59%), best:    6382/10000 (63.82%), error:     10558.85547, best error:     10541.36426 on epoch:    80, epoch:   92/100, mu: 0.9884392619, learning_rate: 0.0000044312, weight_decay: 0.0000069283

epoch time: 889.2301635742s, input_fired_time: 20.3553714752, output_fired_time: 11.6055889130, propagate_forward_time: 306.0094604492, propagate_backward_time: 514.6330566406, weight_update_time: 1.0823180676, other_time: 36.6267089844
epoch time: 69.0304183960s, input_fired_time: 2.0443267822, output_fired_time: 0.0000000000, propagate_forward_time: 60.5235443115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4625473022
validation[          , genome  5441] predictions:    6320/  10000 (63.20%), best:    6382/10000 (63.82%), error:     10686.30566, best error:     10541.36426 on epoch:    80, epoch:   93/100, mu: 0.9885340333, learning_rate: 0.0000041393, weight_decay: 0.0000065966

epoch time: 880.2030639648s, input_fired_time: 20.2976608276, output_fired_time: 11.5709238052, propagate_forward_time: 299.2778015137, propagate_backward_time: 513.4896240234, weight_update_time: 1.0670894384, other_time: 35.5670166016
epoch time: 70.9478073120s, input_fired_time: 2.1821210384, output_fired_time: 0.0000000000, propagate_forward_time: 61.8434410095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9222450256
validation[          , genome  5441] predictions:    6416/  10000 (64.16%), best:    6416/10000 (64.16%), error:     10505.91504, best error:     10505.91504 on epoch:    94, epoch:   94/100, mu: 0.9886230230, learning_rate: 0.0000038666, weight_decay: 0.0000062808

epoch time: 843.7542114258s, input_fired_time: 19.6977767944, output_fired_time: 11.1841306686, propagate_forward_time: 287.6915283203, propagate_backward_time: 489.6530456543, weight_update_time: 1.0623730421, other_time: 35.5277404785
epoch time: 60.3795471191s, input_fired_time: 1.9233920574, output_fired_time: 0.0000000000, propagate_forward_time: 52.6686630249, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7874908447
validation[          , genome  5441] predictions:    6389/  10000 (63.89%), best:    6416/10000 (64.16%), error:     10556.06250, best error:     10505.91504 on epoch:    94, epoch:   95/100, mu: 0.9887066483, learning_rate: 0.0000036119, weight_decay: 0.0000059801

epoch time: 829.8448486328s, input_fired_time: 19.6577491760, output_fired_time: 10.9503154755, propagate_forward_time: 282.7498168945, propagate_backward_time: 483.4043273926, weight_update_time: 1.0339888334, other_time: 33.0826110840
epoch time: 65.9081573486s, input_fired_time: 2.0117001534, output_fired_time: 0.0000000000, propagate_forward_time: 57.5046119690, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3918457031
validation[          , genome  5441] predictions:    6394/  10000 (63.94%), best:    6416/10000 (64.16%), error:     10525.56055, best error:     10505.91504 on epoch:    94, epoch:   96/100, mu: 0.9887851477, learning_rate: 0.0000033740, weight_decay: 0.0000056938

epoch time: 815.3866577148s, input_fired_time: 19.8904762268, output_fired_time: 10.9796962738, propagate_forward_time: 277.8394165039, propagate_backward_time: 473.2167053223, weight_update_time: 1.0172147751, other_time: 33.4603576660
epoch time: 59.0134429932s, input_fired_time: 1.8125350475, output_fired_time: 0.0000000000, propagate_forward_time: 51.6891212463, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5117874146
validation[          , genome  5441] predictions:    6396/  10000 (63.96%), best:    6416/10000 (64.16%), error:     10558.38770, best error:     10505.91504 on epoch:    94, epoch:   97/100, mu: 0.9888588786, learning_rate: 0.0000031517, weight_decay: 0.0000054213

epoch time: 793.9984741211s, input_fired_time: 19.4609756470, output_fired_time: 10.7753667831, propagate_forward_time: 271.0782775879, propagate_backward_time: 460.7075805664, weight_update_time: 0.9987779260, other_time: 31.9762268066
epoch time: 55.6016883850s, input_fired_time: 1.8920903206, output_fired_time: 0.0000000000, propagate_forward_time: 48.5390815735, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1705169678
validation[          , genome  5441] predictions:    6410/  10000 (64.10%), best:    6416/10000 (64.16%), error:     10533.99414, best error:     10505.91504 on epoch:    94, epoch:   98/100, mu: 0.9889281392, learning_rate: 0.0000029441, weight_decay: 0.0000051617

epoch time: 806.6998901367s, input_fired_time: 20.0246620178, output_fired_time: 11.1473484039, propagate_forward_time: 274.8519897461, propagate_backward_time: 467.4707641602, weight_update_time: 1.0249367952, other_time: 33.2051391602
epoch time: 52.3250236511s, input_fired_time: 1.7480283976, output_fired_time: 0.0000000000, propagate_forward_time: 45.9112663269, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6657295227
validation[          , genome  5441] predictions:    6417/  10000 (64.17%), best:    6416/10000 (64.16%), error:     10514.42188, best error:     10505.91504 on epoch:    94, epoch:   99/100, mu: 0.9889932275, learning_rate: 0.0000027502, weight_decay: 0.0000049146

epoch time: 820.1145629883s, input_fired_time: 19.8465862274, output_fired_time: 11.3912048340, propagate_forward_time: 276.9064941406, propagate_backward_time: 477.9012145996, weight_update_time: 1.0538959503, other_time: 34.0691223145
epoch time: 56.5065155029s, input_fired_time: 2.0583584309, output_fired_time: 0.0000000000, propagate_forward_time: 49.0320625305, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4160957336
validation[          , genome  5441] predictions:    6378/  10000 (63.78%), best:    6416/10000 (64.16%), error:     10537.93359, best error:     10505.91504 on epoch:    94, epoch:  100/100, mu: 0.9890543222, learning_rate: 0.0000025690, weight_decay: 0.0000046793

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 309.6602478027s, input_fired_time: 10.8577461243, output_fired_time: 0.0000000000, propagate_forward_time: 268.6298522949, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 30.1726379395
best training[          , genome  5441] predictions:   34224/  50000 (68.45%), best:    6416/10000 (64.16%), error:     47185.42578, best error:     10505.91504 on epoch:    94, epoch:  101/100, mu: 0.9891117215, learning_rate: 0.0000023998, weight_decay: 0.0000044553
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 72.5337295532s, input_fired_time: 2.0974044800, output_fired_time: 0.0000000000, propagate_forward_time: 62.9536743164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4826507568
   testing[          , genome  5441] predictions:    6416/  10000 (64.16%), best:    6416/10000 (64.16%), error:     10505.91504, best error:     10505.91504 on epoch:    94, epoch:  101/100, mu: 0.9891117215, learning_rate: 0.0000023998, weight_decay: 0.0000044553
backpropagation finished successfully!
18:51:05 (46868): called boinc_finish(0)

</stderr_txt>
]]>