Nameexact_genome_1542334203_178_6035_1
Workunit4092240
Created16 Nov 2018, 4:22:15 UTC
Sent16 Nov 2018, 4:59:53 UTC
Report deadline18 Jan 2019, 5:16:16 UTC
Received23 Nov 2018, 5:36:28 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID21895
Run time15 hours 24 min 51 sec
CPU time15 hours 7 min 11 sec
Validate stateValid
Credit15,752.85
Device peak FLOPS4.37 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size551.42 MB
Peak swap size549.73 MB
Peak disk usage14.31 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1981724709
post shuffle 1: 1631507365
epoch time: 1102.47741699219s, input_fired_time: 15.5725002288818, output_fired_time: 5.83204936981201, propagate_forward_time: 404.687805175781, propagate_backward_time: 665.33984375, weight_update_time: 0.451124578714371, other_time: 11.0452270507813
epoch time: 161.412948608398s, input_fired_time: 0.973211765289307, output_fired_time: 0, propagate_forward_time: 158.631958007813, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.80778503417969
validation[          , genome  6035] predictions:    4026/  10000 (40.26%), best:    4026/10000 (40.26%), error:     18467.80078, best error:     18467.80078 on epoch:     0, epoch:    0/50, mu: 0.4740531743, learning_rate: 0.0006946139, weight_decay: 0.0004115002

epoch time: 1105.8779296875s, input_fired_time: 14.6404027939, output_fired_time: 5.3773517609, propagate_forward_time: 442.1958312988, propagate_backward_time: 633.7280883789, weight_update_time: 0.4295561314, other_time: 9.9363403320
epoch time: 102.3577270508s, input_fired_time: 1.0394577980, output_fired_time: 0.0000000000, propagate_forward_time: 99.2947387695, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0235290527
validation[          , genome  6035] predictions:    5618/  10000 (56.18%), best:    5618/10000 (56.18%), error:     12630.94043, best error:     12630.94043 on epoch:     1, epoch:    1/50, mu: 0.4932180941, learning_rate: 0.0006464908, weight_decay: 0.0003796258

epoch time: 987.0700073242s, input_fired_time: 14.6368484497, output_fired_time: 5.3665332794, propagate_forward_time: 367.3371582031, propagate_backward_time: 589.8648071289, weight_update_time: 0.4289604127, other_time: 9.8646850586
epoch time: 75.8909530640s, input_fired_time: 0.8597375154, output_fired_time: 0.0000000000, propagate_forward_time: 73.4679031372, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5633087158
validation[          , genome  6035] predictions:    5273/  10000 (52.73%), best:    5618/10000 (56.18%), error:     13541.77734, best error:     12630.94043 on epoch:     1, epoch:    2/50, mu: 0.5116711259, learning_rate: 0.0006017017, weight_decay: 0.0003502203

epoch time: 993.0266113281s, input_fired_time: 14.7183771133, output_fired_time: 5.3984961510, propagate_forward_time: 370.7898864746, propagate_backward_time: 592.2217407227, weight_update_time: 0.4276371598, other_time: 9.8981323242
epoch time: 79.7087097168s, input_fired_time: 0.9185217619, output_fired_time: 0.0000000000, propagate_forward_time: 77.1508407593, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6393508911
validation[          , genome  6035] predictions:    5604/  10000 (56.04%), best:    5618/10000 (56.18%), error:     12817.78516, best error:     12630.94043 on epoch:     1, epoch:    3/50, mu: 0.5294387341, learning_rate: 0.0005600156, weight_decay: 0.0003230926

epoch time: 1153.5483398438s, input_fired_time: 15.5731258392, output_fired_time: 5.6920700073, propagate_forward_time: 401.7868957520, propagate_backward_time: 719.9159545898, weight_update_time: 0.4519427717, other_time: 10.5803833008
epoch time: 76.0494613647s, input_fired_time: 0.8669076562, output_fired_time: 0.0000000000, propagate_forward_time: 73.6236801147, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5588760376
validation[          , genome  6035] predictions:    5446/  10000 (54.46%), best:    5618/10000 (56.18%), error:     13153.45996, best error:     12630.94043 on epoch:     1, epoch:    4/50, mu: 0.5465463400, learning_rate: 0.0005212175, weight_decay: 0.0002980662

epoch time: 999.8769531250s, input_fired_time: 14.8749885559, output_fired_time: 5.4538588524, propagate_forward_time: 371.9378662109, propagate_backward_time: 597.5948486328, weight_update_time: 0.4412183166, other_time: 10.0153808594
epoch time: 91.6528930664s, input_fired_time: 0.9021267891, output_fired_time: 0.0000000000, propagate_forward_time: 89.1027069092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6480560303
validation[          , genome  6035] predictions:    5739/  10000 (57.39%), best:    5739/10000 (57.39%), error:     12323.73340, best error:     12323.73340 on epoch:     5, epoch:    5/50, mu: 0.5630185008, learning_rate: 0.0004851074, weight_decay: 0.0002749782

epoch time: 999.5946044922s, input_fired_time: 14.8368396759, output_fired_time: 5.4114279747, propagate_forward_time: 372.9434204102, propagate_backward_time: 596.4469604492, weight_update_time: 0.4316983521, other_time: 9.9559326172
epoch time: 76.1784973145s, input_fired_time: 0.8701307774, output_fired_time: 0.0000000000, propagate_forward_time: 73.7375793457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5707855225
validation[          , genome  6035] predictions:    5816/  10000 (58.16%), best:    5816/10000 (58.16%), error:     11987.06152, best error:     11987.06152 on epoch:     6, epoch:    6/50, mu: 0.5788787603, learning_rate: 0.0004514990, weight_decay: 0.0002536787

epoch time: 1036.4466552734s, input_fired_time: 15.0631666183, output_fired_time: 5.4637908936, propagate_forward_time: 402.2759704590, propagate_backward_time: 603.5013427734, weight_update_time: 0.4819719195, other_time: 10.1423339844
epoch time: 77.6851196289s, input_fired_time: 0.8754610419, output_fired_time: 0.0000000000, propagate_forward_time: 75.2170486450, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5926132202
validation[          , genome  6035] predictions:    5904/  10000 (59.04%), best:    5904/10000 (59.04%), error:     11834.76953, best error:     11834.76953 on epoch:     7, epoch:    7/50, mu: 0.5941498876, learning_rate: 0.0004202190, weight_decay: 0.0002340290

epoch time: 1082.1434326172s, input_fired_time: 15.7616281509, output_fired_time: 5.7244410515, propagate_forward_time: 415.1038208008, propagate_backward_time: 634.7658081055, weight_update_time: 0.4654414654, other_time: 10.7877197266
epoch time: 76.3093872070s, input_fired_time: 0.8665760159, output_fired_time: 0.0000000000, propagate_forward_time: 73.8721084595, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5707015991
validation[          , genome  6035] predictions:    5933/  10000 (59.33%), best:    5933/10000 (59.33%), error:     11832.17090, best error:     11832.17090 on epoch:     8, epoch:    8/50, mu: 0.6088538170, learning_rate: 0.0003911061, weight_decay: 0.0002159013

epoch time: 1040.5742187500s, input_fired_time: 16.1922817230, output_fired_time: 5.5423851013, propagate_forward_time: 394.7483215332, propagate_backward_time: 613.5194091797, weight_update_time: 0.5476077795, other_time: 10.5718994141
epoch time: 77.6765441895s, input_fired_time: 0.8757202029, output_fired_time: 0.0000000000, propagate_forward_time: 75.2065277100, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5942993164
validation[          , genome  6035] predictions:    5715/  10000 (57.15%), best:    5933/10000 (59.33%), error:     12304.29688, best error:     11832.17090 on epoch:     8, epoch:    9/50, mu: 0.6230115294, learning_rate: 0.0003640101, weight_decay: 0.0001991778

epoch time: 1054.6234130859s, input_fired_time: 14.9033737183, output_fired_time: 5.3615388870, propagate_forward_time: 372.6835937500, propagate_backward_time: 651.5567626953, weight_update_time: 0.4372866452, other_time: 10.1181640625
epoch time: 77.3831939697s, input_fired_time: 0.8888251185, output_fired_time: 0.0000000000, propagate_forward_time: 74.9105911255, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5837783813
validation[          , genome  6035] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11392.79492, best error:     11392.79492 on epoch:    10, epoch:   10/50, mu: 0.6366433501, learning_rate: 0.0003387914, weight_decay: 0.0001837497

epoch time: 1007.1062011719s, input_fired_time: 14.7130012512, output_fired_time: 5.3970003128, propagate_forward_time: 380.8897399902, propagate_backward_time: 596.0378417969, weight_update_time: 0.4326110184, other_time: 10.0686035156
epoch time: 75.8591842651s, input_fired_time: 0.8678760529, output_fired_time: 0.0000000000, propagate_forward_time: 73.4389190674, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5523910522
validation[          , genome  6035] predictions:    5933/  10000 (59.33%), best:    6049/10000 (60.49%), error:     11737.44238, best error:     11392.79492 on epoch:    10, epoch:   11/50, mu: 0.6497688293, learning_rate: 0.0003153198, weight_decay: 0.0001695166

epoch time: 1058.8029785156s, input_fired_time: 15.5691213608, output_fired_time: 5.4685277939, propagate_forward_time: 379.3163757324, propagate_backward_time: 648.2603759766, weight_update_time: 0.4529170692, other_time: 10.1885986328
epoch time: 76.2554168701s, input_fired_time: 0.8693660498, output_fired_time: 0.0000000000, propagate_forward_time: 73.8395080566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5465393066
validation[          , genome  6035] predictions:    5949/  10000 (59.49%), best:    6049/10000 (60.49%), error:     11668.91895, best error:     11392.79492 on epoch:    10, epoch:   12/50, mu: 0.6624067426, learning_rate: 0.0002934744, weight_decay: 0.0001563860

epoch time: 984.9346313477s, input_fired_time: 14.5953474045, output_fired_time: 5.3235278130, propagate_forward_time: 366.2226257324, propagate_backward_time: 588.5471801758, weight_update_time: 0.4334937334, other_time: 10.2459106445
epoch time: 77.2547073364s, input_fired_time: 0.8882693648, output_fired_time: 0.0000000000, propagate_forward_time: 74.7951278687, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5713119507
validation[          , genome  6035] predictions:    6005/  10000 (60.05%), best:    6049/10000 (60.49%), error:     11647.86523, best error:     11392.79492 on epoch:    10, epoch:   13/50, mu: 0.6745752096, learning_rate: 0.0002731424, weight_decay: 0.0001442725

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035_1_r838246859_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.474053
read mu: 0.686292
read mu_delta: 0.962855
read initial_learning_rate: 0.000694614
read learning_rate: 0.000254219
read learning_rate_delta: 0.93072
read initial_weight_decay: 0.0004115
read weight_decay: 0.000133097
read weight_decay_delta: 0.922541
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0848982
read input_dropout_probability: 0.0010902
read hidden_dropout_probability: 0.103697
read velocity_reset: 2387
read epoch: 14
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 10
read number_validation_images: 10000
read best_validation_predictions: 6049
read best_validation_error: 11392.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6035
read normal distribution: '0 -0.205468595027924 -1.16918885707855'
generator_str: '679372535'
read generator: 679372535
read generated_by_map:
1 add_node 1
reading 101 nodes.
reading 524 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 983.599975585938s, input_fired_time: 14.7420892715454, output_fired_time: 5.32261323928833, propagate_forward_time: 366.350463867188, propagate_backward_time: 587.398620605469, weight_update_time: 0.428836584091187, other_time: 9.78619384765625
epoch time: 76.4515075683594s, input_fired_time: 0.870911478996277, output_fired_time: 0, propagate_forward_time: 74.0258560180664, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.55474090576172
validation[          , genome  6035] predictions:    5882/  10000 (58.82%), best:    6049/10000 (60.49%), error:     11987.84961, best error:     11392.79492 on epoch:    10, epoch:   14/50, mu: 0.6862916946, learning_rate: 0.0002542190, weight_decay: 0.0001330973

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035_1_r838246859_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.474053
read mu: 0.697573
read mu_delta: 0.962855
read initial_learning_rate: 0.000694614
read learning_rate: 0.000236607
read learning_rate_delta: 0.93072
read initial_weight_decay: 0.0004115
read weight_decay: 0.000122788
read weight_decay_delta: 0.922541
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0848982
read input_dropout_probability: 0.0010902
read hidden_dropout_probability: 0.103697
read velocity_reset: 2387
read epoch: 15
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 10
read number_validation_images: 10000
read best_validation_predictions: 6049
read best_validation_error: 11392.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6035
read normal distribution: '0 -0.205468595027924 -1.16918885707855'
generator_str: '1680962474'
read generator: 1680962474
read generated_by_map:
1 add_node 1
reading 101 nodes.
reading 524 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1309.64636230469s, input_fired_time: 47.9145202636719, output_fired_time: 5.82493686676025, propagate_forward_time: 538.888793945313, propagate_backward_time: 706.209228515625, weight_update_time: 0.47514745593071, other_time: 10.808837890625
epoch time: 76.3747100830078s, input_fired_time: 0.868754923343658, output_fired_time: 0, propagate_forward_time: 73.9732971191406, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.53266143798828
validation[          , genome  6035] predictions:    5985/  10000 (59.85%), best:    6049/10000 (60.49%), error:     11663.38477, best error:     11392.79492 on epoch:    10, epoch:   15/50, mu: 0.6975730062, learning_rate: 0.0002366066, weight_decay: 0.0001227877

epoch time: 1268.5216064453s, input_fired_time: 15.4972400665, output_fired_time: 5.6734480858, propagate_forward_time: 545.9378051758, propagate_backward_time: 691.0800781250, weight_update_time: 0.4330359101, other_time: 10.3330688477
epoch time: 75.3287658691s, input_fired_time: 0.8629823327, output_fired_time: 0.0000000000, propagate_forward_time: 72.9474945068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5182876587
validation[          , genome  6035] predictions:    6071/  10000 (60.71%), best:    6049/10000 (60.49%), error:     11437.07227, best error:     11392.79492 on epoch:    10, epoch:   16/50, mu: 0.7084352374, learning_rate: 0.0002202144, weight_decay: 0.0001132767

epoch time: 970.0606689453s, input_fired_time: 14.4771041870, output_fired_time: 5.2401237488, propagate_forward_time: 361.6873474121, propagate_backward_time: 579.0665283203, weight_update_time: 0.4245073199, other_time: 9.5895996094
epoch time: 75.2093353271s, input_fired_time: 0.8669134378, output_fired_time: 0.0000000000, propagate_forward_time: 72.8133239746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5290985107
validation[          , genome  6035] predictions:    6069/  10000 (60.69%), best:    6069/10000 (60.69%), error:     11387.00879, best error:     11387.00879 on epoch:    17, epoch:   17/50, mu: 0.7188940048, learning_rate: 0.0002049579, weight_decay: 0.0001045024

epoch time: 969.4220581055s, input_fired_time: 14.4840183258, output_fired_time: 5.2449612617, propagate_forward_time: 361.3667602539, propagate_backward_time: 578.7406616211, weight_update_time: 0.4239239097, other_time: 9.5856933594
epoch time: 75.2555313110s, input_fired_time: 0.8663079739, output_fired_time: 0.0000000000, propagate_forward_time: 72.8758621216, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5133590698
validation[          , genome  6035] predictions:    6022/  10000 (60.22%), best:    6069/10000 (60.69%), error:     11523.00391, best error:     11387.00879 on epoch:    17, epoch:   18/50, mu: 0.7289642692, learning_rate: 0.0001907584, weight_decay: 0.0000964077

epoch time: 969.1661376953s, input_fired_time: 14.4713153839, output_fired_time: 5.2430338860, propagate_forward_time: 361.3265686035, propagate_backward_time: 578.5305786133, weight_update_time: 0.4250783324, other_time: 9.5946655273
epoch time: 75.2278671265s, input_fired_time: 0.8658633232, output_fired_time: 0.0000000000, propagate_forward_time: 72.8472137451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5147933960
validation[          , genome  6035] predictions:    6139/  10000 (61.39%), best:    6139/10000 (61.39%), error:     11135.29492, best error:     11135.29492 on epoch:    19, epoch:   19/50, mu: 0.7386604548, learning_rate: 0.0001775425, weight_decay: 0.0000889401

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035_1_r838246859_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.474053
read mu: 0.747997
read mu_delta: 0.962855
read initial_learning_rate: 0.000694614
read learning_rate: 0.000165242
read learning_rate_delta: 0.93072
read initial_weight_decay: 0.0004115
read weight_decay: 8.20509e-05
read weight_decay_delta: 0.922541
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0848982
read input_dropout_probability: 0.0010902
read hidden_dropout_probability: 0.103697
read velocity_reset: 2387
read epoch: 20
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 19
read number_validation_images: 10000
read best_validation_predictions: 6139
read best_validation_error: 11135.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6035
read normal distribution: '0 -0.205468595027924 -1.16918885707855'
generator_str: '1278642480'
read generator: 1278642480
read generated_by_map:
1 add_node 1
reading 101 nodes.
reading 524 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1313.62463378906s, input_fired_time: 14.7018604278564, output_fired_time: 5.34745645523071, propagate_forward_time: 552.5283203125, propagate_backward_time: 731.244506835938, weight_update_time: 0.430327981710434, other_time: 9.802490234375
epoch time: 75.9287490844727s, input_fired_time: 0.882015645503998, output_fired_time: 0, propagate_forward_time: 73.4939651489258, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.55276489257813
validation[          , genome  6035] predictions:    6103/  10000 (61.03%), best:    6139/10000 (61.39%), error:     11357.65723, best error:     11135.29492 on epoch:    19, epoch:   20/50, mu: 0.7479965091, learning_rate: 0.0001652423, weight_decay: 0.0000820509

epoch time: 955.4594116211s, input_fired_time: 14.2820148468, output_fired_time: 5.1908164024, propagate_forward_time: 356.0925292969, propagate_backward_time: 570.4170532227, weight_update_time: 0.4166373610, other_time: 9.4769897461
epoch time: 73.2627563477s, input_fired_time: 0.8422556520, output_fired_time: 0.0000000000, propagate_forward_time: 70.9128265381, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5076751709
validation[          , genome  6035] predictions:    6088/  10000 (60.88%), best:    6139/10000 (61.39%), error:     11419.56641, best error:     11135.29492 on epoch:    19, epoch:   21/50, mu: 0.7569857836, learning_rate: 0.0001537943, weight_decay: 0.0000756953

epoch time: 944.2857055664s, input_fired_time: 14.1218233109, output_fired_time: 5.1002235413, propagate_forward_time: 352.0255126953, propagate_backward_time: 563.7268676758, weight_update_time: 0.4114274979, other_time: 9.3112792969
epoch time: 73.2527465820s, input_fired_time: 0.8505206108, output_fired_time: 0.0000000000, propagate_forward_time: 70.9043350220, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4978942871
validation[          , genome  6035] predictions:    6109/  10000 (61.09%), best:    6139/10000 (61.39%), error:     11250.96777, best error:     11135.29492 on epoch:    19, epoch:   22/50, mu: 0.7656411529, learning_rate: 0.0001431394, weight_decay: 0.0000698320

epoch time: 944.2765502930s, input_fired_time: 14.1167850494, output_fired_time: 5.1061587334, propagate_forward_time: 352.0566711426, propagate_backward_time: 563.6895751953, weight_update_time: 0.4114821255, other_time: 9.3073730469
epoch time: 73.2894821167s, input_fired_time: 0.8461970091, output_fired_time: 0.0000000000, propagate_forward_time: 70.9417953491, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5014877319
validation[          , genome  6035] predictions:    6077/  10000 (60.77%), best:    6139/10000 (61.39%), error:     11211.55566, best error:     11135.29492 on epoch:    19, epoch:   23/50, mu: 0.7739750147, learning_rate: 0.0001332226, weight_decay: 0.0000644229

epoch time: 1154.1636962891s, input_fired_time: 14.2038974762, output_fired_time: 5.1279172897, propagate_forward_time: 353.6497497559, propagate_backward_time: 771.8473510742, weight_update_time: 0.4128375947, other_time: 9.3347778320
epoch time: 73.3076400757s, input_fired_time: 0.8449863791, output_fired_time: 0.0000000000, propagate_forward_time: 70.9614791870, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5011749268
validation[          , genome  6035] predictions:    6183/  10000 (61.83%), best:    6183/10000 (61.83%), error:     10994.62793, best error:     10994.62793 on epoch:    24, epoch:   24/50, mu: 0.7819992900, learning_rate: 0.0001239929, weight_decay: 0.0000594328

epoch time: 943.6788330078s, input_fired_time: 14.1141986847, output_fired_time: 5.1087479591, propagate_forward_time: 351.7712097168, propagate_backward_time: 563.4138183594, weight_update_time: 0.4110189676, other_time: 9.2708740234
epoch time: 73.3078384399s, input_fired_time: 0.8430203199, output_fired_time: 0.0000000000, propagate_forward_time: 70.9546432495, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5101776123
validation[          , genome  6035] predictions:    6196/  10000 (61.96%), best:    6183/10000 (61.83%), error:     11130.80664, best error:     10994.62793 on epoch:    24, epoch:   25/50, mu: 0.7897254825, learning_rate: 0.0001154027, weight_decay: 0.0000548291

epoch time: 943.9956665039s, input_fired_time: 14.1285886765, output_fired_time: 5.1023755074, propagate_forward_time: 351.8405761719, propagate_backward_time: 563.6429443359, weight_update_time: 0.4114340544, other_time: 9.2811889648
epoch time: 73.3086471558s, input_fired_time: 0.8471630812, output_fired_time: 0.0000000000, propagate_forward_time: 70.9616165161, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4998703003
validation[          , genome  6035] predictions:    6267/  10000 (62.67%), best:    6267/10000 (62.67%), error:     10805.87402, best error:     10805.87402 on epoch:    26, epoch:   26/50, mu: 0.7971646786, learning_rate: 0.0001074075, weight_decay: 0.0000505821

epoch time: 943.4916381836s, input_fired_time: 14.1283178329, output_fired_time: 5.0964279175, propagate_forward_time: 351.7383117676, propagate_backward_time: 563.2605590820, weight_update_time: 0.4102934599, other_time: 9.2680053711
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035_1_r838246859_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.474053
read mu: 0.804328
read mu_delta: 0.962855
read initial_learning_rate: 0.000694614
read learning_rate: 9.99663e-05
read learning_rate_delta: 0.93072
read initial_weight_decay: 0.0004115
read weight_decay: 4.66641e-05
read weight_decay_delta: 0.922541
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0848982
read input_dropout_probability: 0.0010902
read hidden_dropout_probability: 0.103697
read velocity_reset: 2387
read epoch: 27
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 26
read number_validation_images: 10000
read best_validation_predictions: 6267
read best_validation_error: 10805.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6035
read normal distribution: '0 -0.205468595027924 -1.16918885707855'
generator_str: '1237813006'
read generator: 1237813006
read generated_by_map:
1 add_node 1
reading 101 nodes.
reading 524 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1649.16162109375s, input_fired_time: 14.6749324798584, output_fired_time: 5.32791900634766, propagate_forward_time: 498.990936279297, propagate_backward_time: 1120.29455566406, weight_update_time: 0.434554010629654, other_time: 9.873291015625
epoch time: 74.8766098022461s, input_fired_time: 0.869401395320892, output_fired_time: 0, propagate_forward_time: 72.4527969360352, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.55441284179688
validation[          , genome  6035] predictions:    6251/  10000 (62.51%), best:    6267/10000 (62.67%), error:     10949.24512, best error:     10805.87402 on epoch:    26, epoch:   27/50, mu: 0.8043275476, learning_rate: 0.0000999663, weight_decay: 0.0000466641

epoch time: 1019.7993774414s, input_fired_time: 14.6892013550, output_fired_time: 5.3521776199, propagate_forward_time: 374.6554870605, propagate_backward_time: 615.2697753906, weight_update_time: 0.4274838865, other_time: 9.8327636719
epoch time: 75.1754837036s, input_fired_time: 0.8792936802, output_fired_time: 0.0000000000, propagate_forward_time: 72.7301635742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5660247803
validation[          , genome  6035] predictions:    6314/  10000 (63.14%), best:    6314/10000 (63.14%), error:     10644.15430, best error:     10644.15430 on epoch:    28, epoch:   28/50, mu: 0.8112244010, learning_rate: 0.0000930406, weight_decay: 0.0000430495

epoch time: 974.3110961914s, input_fired_time: 14.5939245224, output_fired_time: 5.2825989723, propagate_forward_time: 362.7075805664, propagate_backward_time: 581.9721679688, weight_update_time: 0.4259228110, other_time: 9.7548217773
epoch time: 74.0606918335s, input_fired_time: 0.8654364347, output_fired_time: 0.0000000000, propagate_forward_time: 71.6527328491, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5425262451
validation[          , genome  6035] predictions:    6351/  10000 (63.51%), best:    6314/10000 (63.14%), error:     10665.69531, best error:     10644.15430 on epoch:    28, epoch:   29/50, mu: 0.8178650737, learning_rate: 0.0000865947, weight_decay: 0.0000397150

epoch time: 964.3175659180s, input_fired_time: 14.3036069870, output_fired_time: 5.1992955208, propagate_forward_time: 355.3544921875, propagate_backward_time: 579.9697875977, weight_update_time: 0.4169243872, other_time: 9.4904174805
epoch time: 73.7509613037s, input_fired_time: 0.8631524444, output_fired_time: 0.0000000000, propagate_forward_time: 71.3611068726, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5267028809
validation[          , genome  6035] predictions:    6398/  10000 (63.98%), best:    6398/10000 (63.98%), error:     10470.86719, best error:     10470.86719 on epoch:    30, epoch:   30/50, mu: 0.8242590427, learning_rate: 0.0000805954, weight_decay: 0.0000366387

epoch time: 948.3100585938s, input_fired_time: 14.2375192642, output_fired_time: 5.1286449432, propagate_forward_time: 353.5016784668, propagate_backward_time: 566.0854492188, weight_update_time: 0.4126678705, other_time: 9.3566894531
epoch time: 73.5220260620s, input_fired_time: 0.8544890881, output_fired_time: 0.0000000000, propagate_forward_time: 71.1603317261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5072021484
validation[          , genome  6035] predictions:    6341/  10000 (63.41%), best:    6398/10000 (63.98%), error:     10521.34570, best error:     10470.86719 on epoch:    30, epoch:   31/50, mu: 0.8304154873, learning_rate: 0.0000750117, weight_decay: 0.0000338007

epoch time: 947.9146728516s, input_fired_time: 14.2282094955, output_fired_time: 5.1166663170, propagate_forward_time: 353.3851928711, propagate_backward_time: 565.8357543945, weight_update_time: 0.4137568176, other_time: 9.3488769531
epoch time: 73.4850387573s, input_fired_time: 0.8578217030, output_fired_time: 0.0000000000, propagate_forward_time: 71.1201705933, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5070495605
validation[          , genome  6035] predictions:    6314/  10000 (63.14%), best:    6398/10000 (63.98%), error:     10659.12793, best error:     10470.86719 on epoch:    30, epoch:   32/50, mu: 0.8363432884, learning_rate: 0.0000698149, weight_decay: 0.0000311825

epoch time: 947.5571899414s, input_fired_time: 14.2132692337, output_fired_time: 5.1172833443, propagate_forward_time: 353.1242065430, propagate_backward_time: 565.7702026367, weight_update_time: 0.4138955772, other_time: 9.3322143555
epoch time: 73.5533370972s, input_fired_time: 0.8590766788, output_fired_time: 0.0000000000, propagate_forward_time: 71.1842803955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5099792480
validation[          , genome  6035] predictions:    6214/  10000 (62.14%), best:    6398/10000 (63.98%), error:     10976.86328, best error:     10470.86719 on epoch:    30, epoch:   33/50, mu: 0.8420509100, learning_rate: 0.0000649781, weight_decay: 0.0000287671

epoch time: 947.8740234375s, input_fired_time: 14.2443399429, output_fired_time: 5.1285419464, propagate_forward_time: 353.2640075684, propagate_backward_time: 565.9047241211, weight_update_time: 0.4132469893, other_time: 9.3324584961
epoch time: 73.5630950928s, input_fired_time: 0.8587787747, output_fired_time: 0.0000000000, propagate_forward_time: 71.2020339966, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5022811890
validation[          , genome  6035] predictions:    6347/  10000 (63.47%), best:    6398/10000 (63.98%), error:     10514.77539, best error:     10470.86719 on epoch:    30, epoch:   34/50, mu: 0.8475465178, learning_rate: 0.0000604764, weight_decay: 0.0000265389

epoch time: 948.3063964844s, input_fired_time: 14.2487850189, output_fired_time: 5.1295514107, propagate_forward_time: 353.3347778320, propagate_backward_time: 566.2752685547, weight_update_time: 0.4131327868, other_time: 9.3179931641
epoch time: 73.4663314819s, input_fired_time: 0.8606721163, output_fired_time: 0.0000000000, propagate_forward_time: 71.0966339111, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5090255737
validation[          , genome  6035] predictions:    6438/  10000 (64.38%), best:    6438/10000 (64.38%), error:     10269.41504, best error:     10269.41504 on epoch:    35, epoch:   35/50, mu: 0.8528379798, learning_rate: 0.0000562866, weight_decay: 0.0000244832

epoch time: 947.8858642578s, input_fired_time: 14.2748432159, output_fired_time: 5.1209850311, propagate_forward_time: 353.2887878418, propagate_backward_time: 565.8554077148, weight_update_time: 0.4136403203, other_time: 9.3458862305
epoch time: 73.7371978760s, input_fired_time: 0.8738160133, output_fired_time: 0.0000000000, propagate_forward_time: 71.3496170044, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5137634277
validation[          , genome  6035] predictions:    6313/  10000 (63.13%), best:    6438/10000 (64.38%), error:     10616.92285, best error:     10269.41504 on epoch:    35, epoch:   36/50, mu: 0.8579328656, learning_rate: 0.0000523870, weight_decay: 0.0000225867

epoch time: 949.4940185547s, input_fired_time: 14.2592315674, output_fired_time: 5.1287484169, propagate_forward_time: 353.8918762207, propagate_backward_time: 566.8736572266, weight_update_time: 0.4142999947, other_time: 9.3405761719
epoch time: 74.1904296875s, input_fired_time: 0.8748155236, output_fired_time: 0.0000000000, propagate_forward_time: 71.7930679321, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5225448608
validation[          , genome  6035] predictions:    6304/  10000 (63.04%), best:    6438/10000 (64.38%), error:     10616.30566, best error:     10269.41504 on epoch:    35, epoch:   37/50, mu: 0.8628385067, learning_rate: 0.0000487576, weight_decay: 0.0000208372

epoch time: 967.0111694336s, input_fired_time: 14.5157270432, output_fired_time: 5.2299838066, propagate_forward_time: 360.5188598633, propagate_backward_time: 577.1135253906, weight_update_time: 0.4246285856, other_time: 9.6330566406
epoch time: 75.8979873657s, input_fired_time: 0.9011394978, output_fired_time: 0.0000000000, propagate_forward_time: 73.4200286865, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5768203735
validation[          , genome  6035] predictions:    6406/  10000 (64.06%), best:    6438/10000 (64.38%), error:     10479.04785, best error:     10269.41504 on epoch:    35, epoch:   38/50, mu: 0.8675619364, learning_rate: 0.0000453797, weight_decay: 0.0000192232

epoch time: 1531.4462890625s, input_fired_time: 14.7749757767, output_fired_time: 5.3312077522, propagate_forward_time: 469.6761779785, propagate_backward_time: 1031.6397705078, weight_update_time: 0.4327229857, other_time: 10.0241699219
epoch time: 75.1944732666s, input_fired_time: 0.8927844763, output_fired_time: 0.0000000000, propagate_forward_time: 72.7576370239, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5440521240
validation[          , genome  6035] predictions:    6360/  10000 (63.60%), best:    6438/10000 (64.38%), error:     10444.30664, best error:     10269.41504 on epoch:    35, epoch:   39/50, mu: 0.8721098900, learning_rate: 0.0000422358, weight_decay: 0.0000177342

epoch time: 977.9421386719s, input_fired_time: 14.3968029022, output_fired_time: 5.1874718666, propagate_forward_time: 366.9482116699, propagate_backward_time: 581.8917236328, weight_update_time: 0.4185862839, other_time: 9.5179443359
epoch time: 74.7909545898s, input_fired_time: 0.8835924864, output_fired_time: 0.0000000000, propagate_forward_time: 72.3472900391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5600738525
validation[          , genome  6035] predictions:    6435/  10000 (64.35%), best:    6435/10000 (64.35%), error:     10250.95703, best error:     10250.95703 on epoch:    40, epoch:   40/50, mu: 0.8764889240, learning_rate: 0.0000393096, weight_decay: 0.0000163605

epoch time: 983.6141357422s, input_fired_time: 14.3335666656, output_fired_time: 5.1637482643, propagate_forward_time: 355.1052246094, propagate_backward_time: 589.2857666016, weight_update_time: 0.4255570471, other_time: 19.7258300781
epoch time: 73.8416824341s, input_fired_time: 0.8703744411, output_fired_time: 0.0000000000, propagate_forward_time: 71.4667892456, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5045166016
validation[          , genome  6035] predictions:    6375/  10000 (63.75%), best:    6435/10000 (64.35%), error:     10512.19238, best error:     10250.95703 on epoch:    40, epoch:   41/50, mu: 0.8807052970, learning_rate: 0.0000365863, weight_decay: 0.0000150932

epoch time: 947.2082519531s, input_fired_time: 14.2644453049, output_fired_time: 5.1183834076, propagate_forward_time: 353.0078125000, propagate_backward_time: 565.4924316406, weight_update_time: 0.4124124348, other_time: 9.3251342773
epoch time: 73.4587936401s, input_fired_time: 0.8637409210, output_fired_time: 0.0000000000, propagate_forward_time: 71.0905761719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5044784546
validation[          , genome  6035] predictions:    6460/  10000 (64.60%), best:    6435/10000 (64.35%), error:     10289.50879, best error:     10250.95703 on epoch:    40, epoch:   42/50, mu: 0.8847650886, learning_rate: 0.0000340516, weight_decay: 0.0000139241

epoch time: 947.9652709961s, input_fired_time: 14.2609319687, output_fired_time: 5.1366977692, propagate_forward_time: 353.2620849609, propagate_backward_time: 565.9918212891, weight_update_time: 0.4128107131, other_time: 9.3137207031
epoch time: 73.5383834839s, input_fired_time: 0.8707228303, output_fired_time: 0.0000000000, propagate_forward_time: 71.1668472290, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5008163452
validation[          , genome  6035] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10162.64648, best error:     10162.64648 on epoch:    43, epoch:   43/50, mu: 0.8886740804, learning_rate: 0.0000316925, weight_decay: 0.0000128456

epoch time: 949.1219482422s, input_fired_time: 14.2915554047, output_fired_time: 5.1249995232, propagate_forward_time: 353.7693481445, propagate_backward_time: 566.5496215820, weight_update_time: 0.4142595530, other_time: 9.3864135742
epoch time: 73.4903488159s, input_fired_time: 0.8688818812, output_fired_time: 0.0000000000, propagate_forward_time: 71.1161041260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5053634644
validation[          , genome  6035] predictions:    6543/  10000 (65.43%), best:    6543/10000 (65.43%), error:     10089.68750, best error:     10089.68750 on epoch:    44, epoch:   44/50, mu: 0.8924378753, learning_rate: 0.0000294968, weight_decay: 0.0000118506

epoch time: 946.2168579102s, input_fired_time: 14.2502937317, output_fired_time: 5.1086215973, propagate_forward_time: 352.7017517090, propagate_backward_time: 564.8485107422, weight_update_time: 0.4214586318, other_time: 9.3076171875
epoch time: 73.4660339355s, input_fired_time: 0.8678375483, output_fired_time: 0.0000000000, propagate_forward_time: 71.0964965820, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5017013550
validation[          , genome  6035] predictions:    6504/  10000 (65.04%), best:    6543/10000 (65.43%), error:     10150.85938, best error:     10089.68750 on epoch:    44, epoch:   45/50, mu: 0.8960618377, learning_rate: 0.0000274532, weight_decay: 0.0000109326

epoch time: 950.3605346680s, input_fired_time: 14.3202304840, output_fired_time: 5.1360311508, propagate_forward_time: 354.1333923340, propagate_backward_time: 567.3925170898, weight_update_time: 0.4146733284, other_time: 9.3783569336
epoch time: 76.1695861816s, input_fired_time: 0.9006112218, output_fired_time: 0.0000000000, propagate_forward_time: 73.7156219482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5533523560
validation[          , genome  6035] predictions:    6457/  10000 (64.57%), best:    6543/10000 (65.43%), error:     10213.30664, best error:     10089.68750 on epoch:    44, epoch:   46/50, mu: 0.8995512128, learning_rate: 0.0000255513, weight_decay: 0.0000100858

epoch time: 1227.7391357422s, input_fired_time: 14.4456930161, output_fired_time: 5.2054491043, propagate_forward_time: 377.9328918457, propagate_backward_time: 820.6644897461, weight_update_time: 0.4212637842, other_time: 9.4906616211
epoch time: 73.6265106201s, input_fired_time: 0.8722862601, output_fired_time: 0.0000000000, propagate_forward_time: 71.2389221191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5153045654
validation[          , genome  6035] predictions:    6583/  10000 (65.83%), best:    6583/10000 (65.83%), error:     10021.40039, best error:     10021.40039 on epoch:    47, epoch:   47/50, mu: 0.9029109478, learning_rate: 0.0000237811, weight_decay: 0.0000093046

epoch time: 944.5075073242s, input_fired_time: 14.2399034500, output_fired_time: 5.1127347946, propagate_forward_time: 352.1131896973, propagate_backward_time: 563.7413330078, weight_update_time: 0.4111181796, other_time: 9.3002929688
epoch time: 73.6412277222s, input_fired_time: 0.8738504052, output_fired_time: 0.0000000000, propagate_forward_time: 71.2533493042, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5140304565
validation[          , genome  6035] predictions:    6526/  10000 (65.26%), best:    6583/10000 (65.83%), error:     10156.43652, best error:     10021.40039 on epoch:    47, epoch:   48/50, mu: 0.9061458707, learning_rate: 0.0000221335, weight_decay: 0.0000085838

epoch time: 945.6108398438s, input_fired_time: 14.2554807663, output_fired_time: 5.1186037064, propagate_forward_time: 352.4627685547, propagate_backward_time: 564.4611816406, weight_update_time: 0.4113352299, other_time: 9.3128051758
epoch time: 73.2629928589s, input_fired_time: 0.8690629601, output_fired_time: 0.0000000000, propagate_forward_time: 70.8893585205, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5045700073
validation[          , genome  6035] predictions:    6558/  10000 (65.58%), best:    6558/10000 (65.58%), error:      9973.95703, best error:      9973.95703 on epoch:    49, epoch:   49/50, mu: 0.9092606306, learning_rate: 0.0000206001, weight_decay: 0.0000079189

epoch time: 943.0835571289s, input_fired_time: 14.2346324921, output_fired_time: 5.1005811691, propagate_forward_time: 351.5193176270, propagate_backward_time: 562.9373168945, weight_update_time: 0.4113669991, other_time: 9.2916870117
epoch time: 73.3134307861s, input_fired_time: 0.8738790154, output_fired_time: 0.0000000000, propagate_forward_time: 70.9436798096, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4958724976
validation[          , genome  6035] predictions:    6578/  10000 (65.78%), best:    6578/10000 (65.78%), error:      9950.44922, best error:      9950.44922 on epoch:    50, epoch:   50/50, mu: 0.9122596979, learning_rate: 0.0000191729, weight_decay: 0.0000073056

evaluating best weights on full training data.
evaluting training set with running mean/variance:
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_178_6035_1_r838246859_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.474053
read mu: 0.915147
read mu_delta: 0.962855
read initial_learning_rate: 0.000694614
read learning_rate: 1.78446e-05
read learning_rate_delta: 0.93072
read initial_weight_decay: 0.0004115
read weight_decay: 6.73967e-06
read weight_decay_delta: 0.922541
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0848982
read input_dropout_probability: 0.0010902
read hidden_dropout_probability: 0.103697
read velocity_reset: 2387
read epoch: 51
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 50
read number_validation_images: 10000
read best_validation_predictions: 6578
read best_validation_error: 9950.45
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6035
read normal distribution: '0 -0.205468595027924 -1.16918885707855'
generator_str: '394807461'
read generator: 394807461
read generated_by_map:
1 add_node 1
reading 101 nodes.
reading 524 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1576.49475097656s, input_fired_time: 14.8401699066162, output_fired_time: 77.8619079589844, propagate_forward_time: 570.024108886719, propagate_backward_time: 903.916198730469, weight_update_time: 0.429724752902985, other_time: 9.852294921875
epoch time: 77.1092300415039s, input_fired_time: 0.917793869972229, output_fired_time: 0, propagate_forward_time: 74.6073684692383, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.58406829833984
validation[          , genome  6035] predictions:    6519/  10000 (65.19%), best:    6578/10000 (65.78%), error:      9993.65918, best error:      9950.44922 on epoch:    50, epoch:   51/50, mu: 0.9151473641, learning_rate: 0.0000178446, weight_decay: 0.0000067397

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 407.2023315430s, input_fired_time: 4.5234971046, output_fired_time: 0.0000000000, propagate_forward_time: 384.5695800781, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 18.1092529297
best training[          , genome  6035] predictions:   35652/  50000 (71.30%), best:    6578/10000 (65.78%), error:     43004.33984, best error:      9950.44922 on epoch:    50, epoch:   52/50, mu: 0.9179278016, learning_rate: 0.0000166083, weight_decay: 0.0000062176
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 74.7411727905s, input_fired_time: 0.8902306557, output_fired_time: 0.0000000000, propagate_forward_time: 72.3168182373, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5341262817
   testing[          , genome  6035] predictions:    6578/  10000 (65.78%), best:    6578/10000 (65.78%), error:      9950.44922, best error:      9950.44922 on epoch:    50, epoch:   52/50, mu: 0.9179278016, learning_rate: 0.0000166083, weight_decay: 0.0000062176
backpropagation finished successfully!
22:35:50 (4120): called boinc_finish(0)

</stderr_txt>
]]>