Nameexact_genome_1542628805_181_6313_1
Workunit4098779
Created19 Nov 2018, 14:42:53 UTC
Sent19 Nov 2018, 14:43:31 UTC
Report deadline11 Jan 2019, 8:09:07 UTC
Received21 Nov 2018, 11:38:34 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time22 hours 13 min 47 sec
CPU time21 hours 46 min 26 sec
Validate stateValid
Credit13,181.53
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size553.86 MB
Peak swap size573.77 MB
Peak disk usage11.55 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313_1_r1922023279_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485337
read mu: 0.485337
read mu_delta: 0.94439
read initial_learning_rate: 0.000457924
read learning_rate: 0.000457924
read learning_rate_delta: 0.918178
read initial_weight_decay: 0.000480024
read weight_decay: 0.000480024
read weight_decay_delta: 0.932364
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0683922
read input_dropout_probability: 0.0013135
read hidden_dropout_probability: 0.111598
read velocity_reset: 1258
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 0 0'
generator_str: '925019763'
read generator: 925019763
read generated_by_map:
1 enable_edge 1
reading 94 nodes.
reading 350 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1173036108
post shuffle 1: 840293200
epoch time: 1666.0986328125s, input_fired_time: 16.8186874389648, output_fired_time: 7.35493469238281, propagate_forward_time: 654.865783691406, propagate_backward_time: 962.182250976563, weight_update_time: 0.407997757196426, other_time: 24.8768920898438
epoch time: 138.294204711914s, input_fired_time: 1.25123679637909, output_fired_time: 0, propagate_forward_time: 133.166015625, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.876953125
validation[          , genome  6313] predictions:    5565/  10000 (55.65%), best:    5565/10000 (55.65%), error:     12751.25000, best error:     12751.25000 on epoch:     0, epoch:    0/50, mu: 0.4853372574, learning_rate: 0.0004579243, weight_decay: 0.0004800236

epoch time: 1648.0036621094s, input_fired_time: 16.4152355194, output_fired_time: 7.2492513657, propagate_forward_time: 647.4572143555, propagate_backward_time: 950.6387939453, weight_update_time: 0.3852254450, other_time: 26.2431030273
epoch time: 139.3680114746s, input_fired_time: 1.2938382626, output_fired_time: 0.0000000000, propagate_forward_time: 134.4477844238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6263885498
validation[          , genome  6313] predictions:    5697/  10000 (56.97%), best:    5697/10000 (56.97%), error:     12532.63281, best error:     12532.63281 on epoch:     1, epoch:    1/50, mu: 0.5134015083, learning_rate: 0.0004204560, weight_decay: 0.0004475566

epoch time: 1683.9581298828s, input_fired_time: 16.8583011627, output_fired_time: 7.3510918617, propagate_forward_time: 661.6262817383, propagate_backward_time: 967.0587158203, weight_update_time: 0.3995942771, other_time: 31.0637817383
epoch time: 138.1700744629s, input_fired_time: 1.2483743429, output_fired_time: 0.0000000000, propagate_forward_time: 133.1908874512, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7308197021
validation[          , genome  6313] predictions:    5532/  10000 (55.32%), best:    5697/10000 (56.97%), error:     12934.76270, best error:     12532.63281 on epoch:     1, epoch:    2/50, mu: 0.5399050713, learning_rate: 0.0003860535, weight_decay: 0.0004172856

epoch time: 1719.7050781250s, input_fired_time: 17.9093513489, output_fired_time: 7.6402068138, propagate_forward_time: 671.9921875000, propagate_backward_time: 978.2340087891, weight_update_time: 0.3846892118, other_time: 43.9293212891
epoch time: 141.6478576660s, input_fired_time: 1.2924079895, output_fired_time: 0.0000000000, propagate_forward_time: 137.7544708252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6009826660
validation[          , genome  6313] predictions:    5551/  10000 (55.51%), best:    5697/10000 (56.97%), error:     12987.59766, best error:     12532.63281 on epoch:     1, epoch:    3/50, mu: 0.5649347901, learning_rate: 0.0003544658, weight_decay: 0.0003890620

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313_1_r1922023279_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485337
read mu: 0.588573
read mu_delta: 0.94439
read initial_learning_rate: 0.000457924
read learning_rate: 0.000325463
read learning_rate_delta: 0.918178
read initial_weight_decay: 0.000480024
read weight_decay: 0.000362747
read weight_decay_delta: 0.932364
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0683922
read input_dropout_probability: 0.0013135
read hidden_dropout_probability: 0.111598
read velocity_reset: 1258
read epoch: 4
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 5697
read best_validation_error: 12532.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 0 0'
generator_str: '553858844'
read generator: 553858844
read generated_by_map:
1 enable_edge 1
reading 94 nodes.
reading 350 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1659.62927246094s, input_fired_time: 16.2020473480225, output_fired_time: 7.17672634124756, propagate_forward_time: 682.401916503906, propagate_backward_time: 919.909912109375, weight_update_time: 0.350397616624832, other_time: 33.9386596679688
epoch time: 141.538986206055s, input_fired_time: 1.18189930915833, output_fired_time: 0, propagate_forward_time: 136.527755737305, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.82933044433594
validation[          , genome  6313] predictions:    5496/  10000 (54.96%), best:    5697/10000 (56.97%), error:     12989.21387, best error:     12532.63281 on epoch:     1, epoch:    4/50, mu: 0.5885726213, learning_rate: 0.0003254627, weight_decay: 0.0003627473

epoch time: 1647.3956298828s, input_fired_time: 16.2228984833, output_fired_time: 7.1470594406, propagate_forward_time: 688.2132568359, propagate_backward_time: 921.0914916992, weight_update_time: 0.3523022830, other_time: 14.7208862305
epoch time: 137.7396240234s, input_fired_time: 1.1640582085, output_fired_time: 0.0000000000, propagate_forward_time: 134.0046081543, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5709533691
validation[          , genome  6313] predictions:    5364/  10000 (53.64%), best:    5697/10000 (56.97%), error:     13639.43555, best error:     12532.63281 on epoch:     1, epoch:    5/50, mu: 0.6108959317, learning_rate: 0.0002988327, weight_decay: 0.0003382125

epoch time: 1693.9735107422s, input_fired_time: 16.7551708221, output_fired_time: 7.3072571754, propagate_forward_time: 719.2355346680, propagate_backward_time: 931.3773193359, weight_update_time: 0.3620591164, other_time: 19.2982788086
epoch time: 156.6230163574s, input_fired_time: 1.2623901367, output_fired_time: 0.0000000000, propagate_forward_time: 151.2353210449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1253051758
validation[          , genome  6313] predictions:    5665/  10000 (56.65%), best:    5697/10000 (56.97%), error:     12720.91504, best error:     12532.63281 on epoch:     1, epoch:    6/50, mu: 0.6319778562, learning_rate: 0.0002743816, weight_decay: 0.0003153371

epoch time: 1618.4685058594s, input_fired_time: 16.0655212402, output_fired_time: 6.8805704117, propagate_forward_time: 665.7337036133, propagate_backward_time: 894.4353637695, weight_update_time: 0.3527076542, other_time: 35.3532714844
epoch time: 118.1749343872s, input_fired_time: 1.0545095205, output_fired_time: 0.0000000000, propagate_forward_time: 114.8483657837, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2720565796
validation[          , genome  6313] predictions:    5832/  10000 (58.32%), best:    5832/10000 (58.32%), error:     12444.59570, best error:     12444.59570 on epoch:     7, epoch:    7/50, mu: 0.6518874168, learning_rate: 0.0002519312, weight_decay: 0.0002940089

epoch time: 1453.3763427734s, input_fired_time: 15.3148317337, output_fired_time: 6.2980589867, propagate_forward_time: 599.5383300781, propagate_backward_time: 818.9207763672, weight_update_time: 0.3311615586, other_time: 13.3043212891
epoch time: 122.3145523071s, input_fired_time: 1.0733519793, output_fired_time: 0.0000000000, propagate_forward_time: 119.0145721436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2266311646
validation[          , genome  6313] predictions:    5694/  10000 (56.94%), best:    5832/10000 (58.32%), error:     12549.19629, best error:     12444.59570 on epoch:     7, epoch:    8/50, mu: 0.6706898212, learning_rate: 0.0002313177, weight_decay: 0.0002741232

epoch time: 1317.1864013672s, input_fired_time: 14.8626461029, output_fired_time: 5.7579646111, propagate_forward_time: 528.6973876953, propagate_backward_time: 754.4149780273, weight_update_time: 0.3057251573, other_time: 13.4534301758
epoch time: 87.9910202026s, input_fired_time: 0.8257288933, output_fired_time: 0.0000000000, propagate_forward_time: 84.7168350220, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4484558105
validation[          , genome  6313] predictions:    5451/  10000 (54.51%), best:    5832/10000 (58.32%), error:     13193.42383, best error:     12444.59570 on epoch:     7, epoch:    9/50, mu: 0.6884466410, learning_rate: 0.0002123908, weight_decay: 0.0002555826

epoch time: 1072.4344482422s, input_fired_time: 13.8266925812, output_fired_time: 4.7205882072, propagate_forward_time: 419.9159545898, propagate_backward_time: 619.9512939453, weight_update_time: 0.2535116673, other_time: 14.0199584961
epoch time: 87.3687133789s, input_fired_time: 0.8315355182, output_fired_time: 0.0000000000, propagate_forward_time: 84.1105041504, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4266738892
validation[          , genome  6313] predictions:    5556/  10000 (55.56%), best:    5832/10000 (58.32%), error:     12869.49512, best error:     12444.59570 on epoch:     7, epoch:   10/50, mu: 0.7052159905, learning_rate: 0.0001950126, weight_decay: 0.0002382960

epoch time: 1065.1121826172s, input_fired_time: 13.8321161270, output_fired_time: 4.6805305481, propagate_forward_time: 415.1513977051, propagate_backward_time: 617.4075317383, weight_update_time: 0.2470023483, other_time: 14.0405883789
epoch time: 88.1479339600s, input_fired_time: 0.8469337225, output_fired_time: 0.0000000000, propagate_forward_time: 84.8539581299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4470443726
validation[          , genome  6313] predictions:    5653/  10000 (56.53%), best:    5832/10000 (58.32%), error:     12646.07617, best error:     12444.59570 on epoch:     7, epoch:   11/50, mu: 0.7210528255, learning_rate: 0.0001790562, weight_decay: 0.0002221785

epoch time: 1069.4747314453s, input_fired_time: 13.7983045578, output_fired_time: 4.6680617332, propagate_forward_time: 418.7660827637, propagate_backward_time: 618.2686767578, weight_update_time: 0.2526117861, other_time: 13.9735107422
epoch time: 87.7649078369s, input_fired_time: 0.8437811136, output_fired_time: 0.0000000000, propagate_forward_time: 84.4184494019, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5026779175
validation[          , genome  6313] predictions:    5707/  10000 (57.07%), best:    5832/10000 (58.32%), error:     12580.68652, best error:     12444.59570 on epoch:     7, epoch:   12/50, mu: 0.7360089421, learning_rate: 0.0001644055, weight_decay: 0.0002071512

epoch time: 1002.8696289063s, input_fired_time: 13.6286764145, output_fired_time: 4.4164609909, propagate_forward_time: 388.5545043945, propagate_backward_time: 582.2578125000, weight_update_time: 0.2341572642, other_time: 14.0122070313
epoch time: 74.9121017456s, input_fired_time: 0.7594249249, output_fired_time: 0.0000000000, propagate_forward_time: 71.6524581909, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5002212524
validation[          , genome  6313] predictions:    5650/  10000 (56.50%), best:    5832/10000 (58.32%), error:     12713.51953, best error:     12444.59570 on epoch:     7, epoch:   13/50, mu: 0.7501333356, learning_rate: 0.0001509535, weight_decay: 0.0001931403

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313_1_r1922023279_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485337
read mu: 0.763472
read mu_delta: 0.94439
read initial_learning_rate: 0.000457924
read learning_rate: 0.000138602
read learning_rate_delta: 0.918178
read initial_weight_decay: 0.000480024
read weight_decay: 0.000180077
read weight_decay_delta: 0.932364
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0683922
read input_dropout_probability: 0.0013135
read hidden_dropout_probability: 0.111598
read velocity_reset: 1258
read epoch: 14
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 7
read number_validation_images: 10000
read best_validation_predictions: 5832
read best_validation_error: 12444.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 0 0'
generator_str: '1105554671'
read generator: 1105554671
read generated_by_map:
1 enable_edge 1
reading 94 nodes.
reading 350 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1472.32092285156s, input_fired_time: 15.5940418243408, output_fired_time: 6.69960069656372, propagate_forward_time: 581.544067382813, propagate_backward_time: 855.604309082031, weight_update_time: 0.344915628433228, other_time: 12.8789672851563
epoch time: 119.114433288574s, input_fired_time: 1.1336704492569, output_fired_time: 0, propagate_forward_time: 115.740432739258, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.24033355712891
validation[          , genome  6313] predictions:    5729/  10000 (57.29%), best:    5832/10000 (58.32%), error:     12478.42188, best error:     12444.59570 on epoch:     7, epoch:   14/50, mu: 0.7634723186, learning_rate: 0.0001386022, weight_decay: 0.0001800770

epoch time: 1494.7921142578s, input_fired_time: 15.5084800720, output_fired_time: 6.7196745872, propagate_forward_time: 589.4992065430, propagate_backward_time: 869.9205322266, weight_update_time: 0.3432866335, other_time: 13.1442260742
epoch time: 124.3986434937s, input_fired_time: 1.1399153471, output_fired_time: 0.0000000000, propagate_forward_time: 120.8138122559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4449157715
validation[          , genome  6313] predictions:    5818/  10000 (58.18%), best:    5818/10000 (58.18%), error:     12230.41602, best error:     12230.41602 on epoch:    15, epoch:   15/50, mu: 0.7760695219, learning_rate: 0.0001272615, weight_decay: 0.0001678973

epoch time: 1499.1868896484s, input_fired_time: 15.5426502228, output_fired_time: 6.7449049950, propagate_forward_time: 591.6367797852, propagate_backward_time: 872.1825561523, weight_update_time: 0.3425934911, other_time: 13.0800781250
epoch time: 118.7822189331s, input_fired_time: 1.1432545185, output_fired_time: 0.0000000000, propagate_forward_time: 115.4460906982, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1928710938
validation[          , genome  6313] predictions:    5805/  10000 (58.05%), best:    5818/10000 (58.18%), error:     12399.62598, best error:     12230.41602 on epoch:    15, epoch:   16/50, mu: 0.7879661918, learning_rate: 0.0001168487, weight_decay: 0.0001565414

epoch time: 1506.4915771484s, input_fired_time: 15.5352106094, output_fired_time: 6.7735729218, propagate_forward_time: 594.1881713867, propagate_backward_time: 876.5917358398, weight_update_time: 0.3459143639, other_time: 13.4029541016
epoch time: 122.8438644409s, input_fired_time: 1.1558041573, output_fired_time: 0.0000000000, propagate_forward_time: 119.3671569824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3208999634
validation[          , genome  6313] predictions:    5860/  10000 (58.60%), best:    5860/10000 (58.60%), error:     12058.26660, best error:     12058.26660 on epoch:    17, epoch:   17/50, mu: 0.7992012501, learning_rate: 0.0001072879, weight_decay: 0.0001459535

epoch time: 1505.5273437500s, input_fired_time: 15.5017795563, output_fired_time: 6.7443814278, propagate_forward_time: 593.3951416016, propagate_backward_time: 876.3010253906, weight_update_time: 0.3489537835, other_time: 13.5849609375
epoch time: 126.2113571167s, input_fired_time: 1.1330714226, output_fired_time: 0.0000000000, propagate_forward_time: 122.6221771240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4561080933
validation[          , genome  6313] predictions:    5684/  10000 (56.84%), best:    5860/10000 (58.60%), error:     12626.08496, best error:     12058.26660 on epoch:    17, epoch:   18/50, mu: 0.8098115325, learning_rate: 0.0000985094, weight_decay: 0.0001360818

epoch time: 1520.7208251953s, input_fired_time: 15.4937200546, output_fired_time: 6.7820277214, propagate_forward_time: 599.8513793945, propagate_backward_time: 885.0533447266, weight_update_time: 0.3404519558, other_time: 13.5403442383
epoch time: 125.5036849976s, input_fired_time: 1.1606950760, output_fired_time: 0.0000000000, propagate_forward_time: 121.9255065918, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4174804688
validation[          , genome  6313] predictions:    5786/  10000 (57.86%), best:    5860/10000 (58.60%), error:     12223.52637, best error:     12058.26660 on epoch:    17, epoch:   19/50, mu: 0.8198317885, learning_rate: 0.0000904492, weight_decay: 0.0001268777

epoch time: 1573.6619873047s, input_fired_time: 15.5212755203, output_fired_time: 6.9688711166, propagate_forward_time: 620.9414062500, propagate_backward_time: 915.4901123047, weight_update_time: 0.3372267187, other_time: 14.7403564453
epoch time: 130.4888916016s, input_fired_time: 1.1790261269, output_fired_time: 0.0000000000, propagate_forward_time: 126.6933975220, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6164627075
validation[          , genome  6313] predictions:    5866/  10000 (58.66%), best:    5860/10000 (58.60%), error:     12112.53223, best error:     12058.26660 on epoch:    17, epoch:   20/50, mu: 0.8292948008, learning_rate: 0.0000830484, weight_decay: 0.0001182962

epoch time: 1592.1591796875s, input_fired_time: 15.5377655029, output_fired_time: 7.0518960953, propagate_forward_time: 628.6677246094, propagate_backward_time: 926.3410034180, weight_update_time: 0.3377011418, other_time: 14.5608520508
epoch time: 129.9864959717s, input_fired_time: 1.1529973745, output_fired_time: 0.0000000000, propagate_forward_time: 126.2448120117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5886840820
validation[          , genome  6313] predictions:    5953/  10000 (59.53%), best:    5953/10000 (59.53%), error:     11946.86719, best error:     11946.86719 on epoch:    21, epoch:   21/50, mu: 0.8382316232, learning_rate: 0.0000762533, weight_decay: 0.0001102951

epoch time: 1588.8505859375s, input_fired_time: 15.6215677261, output_fired_time: 7.0238208771, propagate_forward_time: 626.5029907227, propagate_backward_time: 924.8993530273, weight_update_time: 0.3417910933, other_time: 14.8028564453
epoch time: 130.6071472168s, input_fired_time: 1.1640015841, output_fired_time: 0.0000000000, propagate_forward_time: 126.7460403442, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6971054077
validation[          , genome  6313] predictions:    5836/  10000 (58.36%), best:    5953/10000 (59.53%), error:     12094.34375, best error:     11946.86719 on epoch:    21, epoch:   22/50, mu: 0.8466714621, learning_rate: 0.0000700141, weight_decay: 0.0001028352

epoch time: 1598.4436035156s, input_fired_time: 15.5358114243, output_fired_time: 7.0642962456, propagate_forward_time: 631.1535034180, propagate_backward_time: 929.6432495117, weight_update_time: 0.3374339044, other_time: 15.0467529297
epoch time: 130.6277770996s, input_fired_time: 1.1470654011, output_fired_time: 0.0000000000, propagate_forward_time: 126.7873611450, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6933517456
validation[          , genome  6313] predictions:    5887/  10000 (58.87%), best:    5953/10000 (59.53%), error:     12164.90039, best error:     11946.86719 on epoch:    21, epoch:   23/50, mu: 0.8546419740, learning_rate: 0.0000642854, weight_decay: 0.0000958798

epoch time: 1597.7362060547s, input_fired_time: 15.5370950699, output_fired_time: 7.0177679062, propagate_forward_time: 630.4445190430, propagate_backward_time: 929.8926391602, weight_update_time: 0.3404779136, other_time: 14.8441162109
epoch time: 130.4630279541s, input_fired_time: 1.1673856974, output_fired_time: 0.0000000000, propagate_forward_time: 126.5719451904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7236938477
validation[          , genome  6313] predictions:    5910/  10000 (59.10%), best:    5953/10000 (59.53%), error:     11972.99414, best error:     11946.86719 on epoch:    21, epoch:   24/50, mu: 0.8621692061, learning_rate: 0.0000590254, weight_decay: 0.0000893948

epoch time: 1597.3843994141s, input_fired_time: 15.5671453476, output_fired_time: 7.0372872353, propagate_forward_time: 630.6795043945, propagate_backward_time: 929.0153198242, weight_update_time: 0.3378415704, other_time: 15.0852050781
epoch time: 130.7671203613s, input_fired_time: 1.1533741951, output_fired_time: 0.0000000000, propagate_forward_time: 127.0147933960, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5989456177
validation[          , genome  6313] predictions:    5760/  10000 (57.60%), best:    5953/10000 (59.53%), error:     12194.77148, best error:     11946.86719 on epoch:    21, epoch:   25/50, mu: 0.8692778349, learning_rate: 0.0000541958, weight_decay: 0.0000833485

epoch time: 1597.8625488281s, input_fired_time: 15.5650224686, output_fired_time: 7.0423598289, propagate_forward_time: 631.3217773438, propagate_backward_time: 929.0350952148, weight_update_time: 0.3473145068, other_time: 14.8982543945
epoch time: 130.7337036133s, input_fired_time: 1.1593908072, output_fired_time: 0.0000000000, propagate_forward_time: 126.8636322021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7106781006
validation[          , genome  6313] predictions:    5940/  10000 (59.40%), best:    5940/10000 (59.40%), error:     11868.03223, best error:     11868.03223 on epoch:    26, epoch:   26/50, mu: 0.8759911656, learning_rate: 0.0000497614, weight_decay: 0.0000777111

epoch time: 1598.9079589844s, input_fired_time: 15.5637111664, output_fired_time: 7.0663809776, propagate_forward_time: 631.2597656250, propagate_backward_time: 929.6926879883, weight_update_time: 0.3388001323, other_time: 15.3253784180
epoch time: 130.8605194092s, input_fired_time: 1.1602804661, output_fired_time: 0.0000000000, propagate_forward_time: 127.0024337769, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6978073120
validation[          , genome  6313] predictions:    5919/  10000 (59.19%), best:    5919/10000 (59.19%), error:     11855.35156, best error:     11855.35156 on epoch:    27, epoch:   27/50, mu: 0.8823311925, learning_rate: 0.0000456899, weight_decay: 0.0000724550

epoch time: 1598.5889892578s, input_fired_time: 15.5207538605, output_fired_time: 7.0388412476, propagate_forward_time: 631.1731567383, propagate_backward_time: 929.7359619141, weight_update_time: 0.3396698236, other_time: 15.1203002930
epoch time: 130.7872161865s, input_fired_time: 1.1726657152, output_fired_time: 0.0000000000, propagate_forward_time: 126.8582687378, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7562789917
validation[          , genome  6313] predictions:    5869/  10000 (58.69%), best:    5919/10000 (59.19%), error:     11972.04102, best error:     11855.35156 on epoch:    27, epoch:   28/50, mu: 0.8883186579, learning_rate: 0.0000419514, weight_decay: 0.0000675545

epoch time: 1598.1324462891s, input_fired_time: 15.5965061188, output_fired_time: 7.0576248169, propagate_forward_time: 630.3164672852, propagate_backward_time: 929.9315795898, weight_update_time: 0.3392351866, other_time: 15.2302246094
epoch time: 130.5679016113s, input_fired_time: 1.1611223221, output_fired_time: 0.0000000000, propagate_forward_time: 126.6761093140, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7306747437
validation[          , genome  6313] predictions:    5969/  10000 (59.69%), best:    5969/10000 (59.69%), error:     11814.84668, best error:     11814.84668 on epoch:    29, epoch:   29/50, mu: 0.8939731717, learning_rate: 0.0000385189, weight_decay: 0.0000629853

epoch time: 1537.4222412109s, input_fired_time: 15.3950691223, output_fired_time: 6.7402458191, propagate_forward_time: 604.0118408203, propagate_backward_time: 896.4356079102, weight_update_time: 0.3279388249, other_time: 14.8395385742
epoch time: 118.9140777588s, input_fired_time: 1.0886802673, output_fired_time: 0.0000000000, propagate_forward_time: 115.2428817749, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5825119019
validation[          , genome  6313] predictions:    6008/  10000 (60.08%), best:    6008/10000 (60.08%), error:     11726.23828, best error:     11726.23828 on epoch:    30, epoch:   30/50, mu: 0.8993132114, learning_rate: 0.0000353672, weight_decay: 0.0000587253

epoch time: 1465.5626220703s, input_fired_time: 15.0816183090, output_fired_time: 6.4330072403, propagate_forward_time: 570.9998779297, propagate_backward_time: 858.3190307617, weight_update_time: 0.3187620044, other_time: 14.7290649414
epoch time: 118.5617828369s, input_fired_time: 1.0897690058, output_fired_time: 0.0000000000, propagate_forward_time: 114.8355636597, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6364517212
validation[          , genome  6313] predictions:    6032/  10000 (60.32%), best:    6032/10000 (60.32%), error:     11708.71582, best error:     11708.71582 on epoch:    31, epoch:   31/50, mu: 0.9043563008, learning_rate: 0.0000324734, weight_decay: 0.0000547533

epoch time: 1463.8873291016s, input_fired_time: 15.0820140839, output_fired_time: 6.3986258507, propagate_forward_time: 570.4484863281, propagate_backward_time: 857.3400268555, weight_update_time: 0.3190348148, other_time: 14.6181030273
epoch time: 118.2113418579s, input_fired_time: 1.0856326818, output_fired_time: 0.0000000000, propagate_forward_time: 114.5733184814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5523910522
validation[          , genome  6313] predictions:    5999/  10000 (59.99%), best:    6032/10000 (60.32%), error:     11775.60742, best error:     11708.71582 on epoch:    31, epoch:   32/50, mu: 0.9091189504, learning_rate: 0.0000298163, weight_decay: 0.0000510500

epoch time: 1469.4071044922s, input_fired_time: 15.1072425842, output_fired_time: 6.4256601334, propagate_forward_time: 571.8408813477, propagate_backward_time: 861.0090332031, weight_update_time: 0.3171240389, other_time: 15.0242309570
epoch time: 118.1732406616s, input_fired_time: 1.0885351896, output_fired_time: 0.0000000000, propagate_forward_time: 114.4381332397, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6465759277
validation[          , genome  6313] predictions:    5930/  10000 (59.30%), best:    6032/10000 (60.32%), error:     11745.03125, best error:     11708.71582 on epoch:    31, epoch:   33/50, mu: 0.9136167169, learning_rate: 0.0000273767, weight_decay: 0.0000475972

epoch time: 1447.0297851563s, input_fired_time: 15.0503606796, output_fired_time: 6.3397221565, propagate_forward_time: 562.7185668945, propagate_backward_time: 848.2484741211, weight_update_time: 0.3182418644, other_time: 14.6726074219
epoch time: 108.4352340698s, input_fired_time: 1.0279099941, output_fired_time: 0.0000000000, propagate_forward_time: 104.7031097412, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7042160034
validation[          , genome  6313] predictions:    6051/  10000 (60.51%), best:    6051/10000 (60.51%), error:     11607.74707, best error:     11607.74707 on epoch:    34, epoch:   34/50, mu: 0.9178643823, learning_rate: 0.0000251367, weight_decay: 0.0000443779

epoch time: 1363.7043457031s, input_fired_time: 14.7774715424, output_fired_time: 6.0547652245, propagate_forward_time: 524.4816894531, propagate_backward_time: 803.3795776367, weight_update_time: 0.3059907556, other_time: 15.0108032227
epoch time: 107.4814605713s, input_fired_time: 1.0209858418, output_fired_time: 0.0000000000, propagate_forward_time: 103.8269882202, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6334838867
validation[          , genome  6313] predictions:    5869/  10000 (58.69%), best:    6051/10000 (60.51%), error:     11876.46191, best error:     11607.74707 on epoch:    34, epoch:   35/50, mu: 0.9218758345, learning_rate: 0.0000230800, weight_decay: 0.0000413763

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313_1_r1922023279_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485337
read mu: 0.925664
read mu_delta: 0.94439
read initial_learning_rate: 0.000457924
read learning_rate: 2.11915e-05
read learning_rate_delta: 0.918178
read initial_weight_decay: 0.000480024
read weight_decay: 3.85778e-05
read weight_decay_delta: 0.932364
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0683922
read input_dropout_probability: 0.0013135
read hidden_dropout_probability: 0.111598
read velocity_reset: 1258
read epoch: 36
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 34
read number_validation_images: 10000
read best_validation_predictions: 6051
read best_validation_error: 11607.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 0 0'
generator_str: '1285886611'
read generator: 1285886611
read generated_by_map:
1 enable_edge 1
reading 94 nodes.
reading 350 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1299.04284667969s, input_fired_time: 14.9723901748657, output_fired_time: 5.73713397979736, propagate_forward_time: 494.201416015625, propagate_backward_time: 764.269104003906, weight_update_time: 0.300641268491745, other_time: 19.8627319335938
epoch time: 99.9986190795898s, input_fired_time: 0.984396159648895, output_fired_time: 0, propagate_forward_time: 95.6399459838867, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.37427520751953
validation[          , genome  6313] predictions:    5989/  10000 (59.89%), best:    6051/10000 (60.51%), error:     11669.42480, best error:     11607.74707 on epoch:    34, epoch:   36/50, mu: 0.9256641865, learning_rate: 0.0000211915, weight_decay: 0.0000385778

epoch time: 1257.1291503906s, input_fired_time: 14.5453796387, output_fired_time: 5.6139373779, propagate_forward_time: 479.1380615234, propagate_backward_time: 743.8614501953, weight_update_time: 0.2916068137, other_time: 13.9703369141
epoch time: 106.0674514771s, input_fired_time: 1.0051180124, output_fired_time: 0.0000000000, propagate_forward_time: 102.5448150635, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5175170898
validation[          , genome  6313] predictions:    5948/  10000 (59.48%), best:    6051/10000 (60.51%), error:     11768.45508, best error:     11607.74707 on epoch:    34, epoch:   37/50, mu: 0.9292418957, learning_rate: 0.0000194576, weight_decay: 0.0000359685

epoch time: 1301.9031982422s, input_fired_time: 14.6739034653, output_fired_time: 5.6154403687, propagate_forward_time: 510.9153442383, propagate_backward_time: 756.7042846680, weight_update_time: 0.3026176691, other_time: 13.9941406250
epoch time: 105.6569290161s, input_fired_time: 1.0049080849, output_fired_time: 0.0000000000, propagate_forward_time: 102.0638427734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5881805420
validation[          , genome  6313] predictions:    6081/  10000 (60.81%), best:    6081/10000 (60.81%), error:     11594.67676, best error:     11594.67676 on epoch:    38, epoch:   38/50, mu: 0.9326206446, learning_rate: 0.0000178655, weight_decay: 0.0000335358

epoch time: 1305.7860107422s, input_fired_time: 14.6669368744, output_fired_time: 5.6968717575, propagate_forward_time: 510.2377624512, propagate_backward_time: 761.0975341797, weight_update_time: 0.2962789237, other_time: 14.0867919922
epoch time: 109.6130523682s, input_fired_time: 1.0448707342, output_fired_time: 0.0000000000, propagate_forward_time: 106.1841583252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3840255737
validation[          , genome  6313] predictions:    5987/  10000 (59.87%), best:    6081/10000 (60.81%), error:     11672.13477, best error:     11594.67676 on epoch:    38, epoch:   39/50, mu: 0.9358115196, learning_rate: 0.0000164037, weight_decay: 0.0000312675

epoch time: 1343.2478027344s, input_fired_time: 14.9416389465, output_fired_time: 5.9290456772, propagate_forward_time: 518.8655395508, propagate_backward_time: 789.6110839844, weight_update_time: 0.3107379675, other_time: 13.9004516602
epoch time: 125.9565887451s, input_fired_time: 1.1492605209, output_fired_time: 0.0000000000, propagate_forward_time: 122.6238250732, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1835021973
validation[          , genome  6313] predictions:    6041/  10000 (60.41%), best:    6081/10000 (60.81%), error:     11687.61328, best error:     11594.67676 on epoch:    38, epoch:   40/50, mu: 0.9388249516, learning_rate: 0.0000150615, weight_decay: 0.0000291527

epoch time: 1417.8430175781s, input_fired_time: 15.1261568069, output_fired_time: 6.2688364983, propagate_forward_time: 555.1398925781, propagate_backward_time: 827.9855346680, weight_update_time: 0.3210044503, other_time: 13.3226928711
epoch time: 99.3425979614s, input_fired_time: 0.9711660147, output_fired_time: 0.0000000000, propagate_forward_time: 95.9933853149, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3780441284
validation[          , genome  6313] predictions:    6103/  10000 (61.03%), best:    6103/10000 (61.03%), error:     11524.16895, best error:     11524.16895 on epoch:    41, epoch:   41/50, mu: 0.9416707754, learning_rate: 0.0000138292, weight_decay: 0.0000271809

epoch time: 1261.9776611328s, input_fired_time: 14.5378684998, output_fired_time: 5.6142435074, propagate_forward_time: 478.6785583496, propagate_backward_time: 749.1826171875, weight_update_time: 0.2918600738, other_time: 13.9643554688
epoch time: 99.5796051025s, input_fired_time: 0.9805591106, output_fired_time: 0.0000000000, propagate_forward_time: 96.1297988892, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4692459106
validation[          , genome  6313] predictions:    6100/  10000 (61.00%), best:    6100/10000 (61.00%), error:     11489.76563, best error:     11489.76563 on epoch:    42, epoch:   42/50, mu: 0.9443583488, learning_rate: 0.0000126976, weight_decay: 0.0000253425

epoch time: 1251.4970703125s, input_fired_time: 14.5015153885, output_fired_time: 5.5939149857, propagate_forward_time: 474.9667053223, propagate_backward_time: 742.6331176758, weight_update_time: 0.2852931321, other_time: 13.8019409180
epoch time: 99.5131759644s, input_fired_time: 0.9792515635, output_fired_time: 0.0000000000, propagate_forward_time: 96.1302032471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4037246704
validation[          , genome  6313] predictions:    6044/  10000 (60.44%), best:    6100/10000 (61.00%), error:     11541.41309, best error:     11489.76563 on epoch:    42, epoch:   43/50, mu: 0.9468964934, learning_rate: 0.0000116587, weight_decay: 0.0000236284

epoch time: 1328.6378173828s, input_fired_time: 14.8931751251, output_fired_time: 5.8431777954, propagate_forward_time: 516.8569946289, propagate_backward_time: 777.0816040039, weight_update_time: 0.3062111735, other_time: 13.9628906250
epoch time: 106.8888015747s, input_fired_time: 1.0260382891, output_fired_time: 0.0000000000, propagate_forward_time: 103.3736648560, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4890975952
validation[          , genome  6313] predictions:    6095/  10000 (60.95%), best:    6095/10000 (60.95%), error:     11478.76172, best error:     11478.76172 on epoch:    44, epoch:   44/50, mu: 0.9492934942, learning_rate: 0.0000107048, weight_decay: 0.0000220303

epoch time: 1352.8834228516s, input_fired_time: 15.0398635864, output_fired_time: 5.9678201675, propagate_forward_time: 525.4943237305, propagate_backward_time: 792.2454223633, weight_update_time: 0.3069974780, other_time: 14.1359863281
epoch time: 108.8464889526s, input_fired_time: 1.0249069929, output_fired_time: 0.0000000000, propagate_forward_time: 105.3211517334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5004272461
validation[          , genome  6313] predictions:    6086/  10000 (60.86%), best:    6086/10000 (60.86%), error:     11459.47266, best error:     11459.47266 on epoch:    45, epoch:   45/50, mu: 0.9515571594, learning_rate: 0.0000098289, weight_decay: 0.0000205403

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542628805_181_6313_1_r1922023279_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485337
read mu: 0.953695
read mu_delta: 0.94439
read initial_learning_rate: 0.000457924
read learning_rate: 9.02466e-06
read learning_rate_delta: 0.918178
read initial_weight_decay: 0.000480024
read weight_decay: 1.9151e-05
read weight_decay_delta: 0.932364
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0683922
read input_dropout_probability: 0.0013135
read hidden_dropout_probability: 0.111598
read velocity_reset: 1258
read epoch: 46
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 45
read number_validation_images: 10000
read best_validation_predictions: 6086
read best_validation_error: 11459.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 0 0'
generator_str: '1621015411'
read generator: 1621015411
read generated_by_map:
1 enable_edge 1
reading 94 nodes.
reading 350 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1344.5751953125s, input_fired_time: 14.9535551071167, output_fired_time: 5.9304461479187, propagate_forward_time: 513.540222167969, propagate_backward_time: 790.186706542969, weight_update_time: 0.308013051748276, other_time: 19.9642333984375
epoch time: 118.720039367676s, input_fired_time: 1.11851251125336, output_fired_time: 0, propagate_forward_time: 115.115699768066, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.48582458496094
validation[          , genome  6313] predictions:    6117/  10000 (61.17%), best:    6117/10000 (61.17%), error:     11444.54785, best error:     11444.54785 on epoch:    46, epoch:   46/50, mu: 0.9536949396, learning_rate: 0.0000090247, weight_decay: 0.0000191510

epoch time: 1306.9483642578s, input_fired_time: 14.7503986359, output_fired_time: 5.7866873741, propagate_forward_time: 500.5871887207, propagate_backward_time: 772.3043823242, weight_update_time: 0.3002370596, other_time: 13.5197143555
epoch time: 101.4114685059s, input_fired_time: 1.0040671825, output_fired_time: 0.0000000000, propagate_forward_time: 98.0529785156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3544235229
validation[          , genome  6313] predictions:    6100/  10000 (61.00%), best:    6117/10000 (61.17%), error:     11533.19727, best error:     11444.54785 on epoch:    46, epoch:   47/50, mu: 0.9557138681, learning_rate: 0.0000082862, weight_decay: 0.0000178557

epoch time: 1298.3930664063s, input_fired_time: 14.8046350479, output_fired_time: 5.7562942505, propagate_forward_time: 497.8455810547, propagate_backward_time: 766.4293823242, weight_update_time: 0.3055572808, other_time: 13.5570678711
epoch time: 113.9331359863s, input_fired_time: 1.0954483747, output_fired_time: 0.0000000000, propagate_forward_time: 110.6501922607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1874923706
validation[          , genome  6313] predictions:    6016/  10000 (60.16%), best:    6117/10000 (61.17%), error:     11588.18848, best error:     11444.54785 on epoch:    46, epoch:   48/50, mu: 0.9576205015, learning_rate: 0.0000076082, weight_decay: 0.0000166480

epoch time: 1267.3692626953s, input_fired_time: 14.7019987106, output_fired_time: 5.6419506073, propagate_forward_time: 484.4426269531, propagate_backward_time: 749.3775634766, weight_update_time: 0.2927727699, other_time: 13.2050781250
epoch time: 78.5721130371s, input_fired_time: 0.8541085124, output_fired_time: 0.0000000000, propagate_forward_time: 75.2829818726, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4350204468
validation[          , genome  6313] predictions:    6123/  10000 (61.23%), best:    6117/10000 (61.17%), error:     11468.47852, best error:     11444.54785 on epoch:    46, epoch:   49/50, mu: 0.9594210982, learning_rate: 0.0000069857, weight_decay: 0.0000155220

epoch time: 1067.1702880859s, input_fired_time: 14.1142845154, output_fired_time: 4.7923412323, propagate_forward_time: 399.1565856934, propagate_backward_time: 635.6449584961, weight_update_time: 0.2621616423, other_time: 13.4620971680
epoch time: 91.5106887817s, input_fired_time: 0.9460045695, output_fired_time: 0.0000000000, propagate_forward_time: 88.2824630737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2822189331
validation[          , genome  6313] predictions:    6104/  10000 (61.04%), best:    6117/10000 (61.17%), error:     11492.97168, best error:     11444.54785 on epoch:    46, epoch:   50/50, mu: 0.9611215591, learning_rate: 0.0000064141, weight_decay: 0.0000144721

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 444.7174682617s, input_fired_time: 4.5585684776, output_fired_time: 0.0000000000, propagate_forward_time: 428.4594726563, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.6994323730
best training[          , genome  6313] predictions:   32144/  50000 (64.29%), best:    6117/10000 (61.17%), error:     53904.02344, best error:     11444.54785 on epoch:    46, epoch:   51/50, mu: 0.9627274871, learning_rate: 0.0000058893, weight_decay: 0.0000134933
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 83.9554138184s, input_fired_time: 0.8723248243, output_fired_time: 0.0000000000, propagate_forward_time: 80.1260147095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9570770264
   testing[          , genome  6313] predictions:    6117/  10000 (61.17%), best:    6117/10000 (61.17%), error:     11444.54785, best error:     11444.54785 on epoch:    46, epoch:   51/50, mu: 0.9627274871, learning_rate: 0.0000058893, weight_decay: 0.0000134933
backpropagation finished successfully!
15:37:44 (9492): called boinc_finish(0)

</stderr_txt>
]]>