Nameexact_genome_1542031203_174_5567_0
Workunit4080956
Created12 Nov 2018, 17:47:12 UTC
Sent12 Nov 2018, 17:55:59 UTC
Report deadline23 Dec 2018, 21:19:55 UTC
Received15 Nov 2018, 8:51:14 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID30770
Run time13 hours 20 min 58 sec
CPU time13 hours 17 min 45 sec
Validate stateValid
Credit10,285.41
Device peak FLOPS4.85 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size521.81 MB
Peak swap size543.15 MB
Peak disk usage9.25 MB

Stderr output

<core_client_version>7.6.33</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_174_5567.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_174_5567_0_r1102848023_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542031203_174_5567.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.5
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0125
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.0005
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5567
read normal distribution: '1 0 0'
generator_str: '942647586'
read generator: 942647586
read generated_by_map:
1 enable_edge 1
reading 87 nodes.
reading 299 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1091113983
post shuffle 1: 1758402965
epoch time: 930.223449707031s, input_fired_time: 8.40646171569824, output_fired_time: 7.25107717514038, propagate_forward_time: 346.769348144531, propagate_backward_time: 556.562622070313, weight_update_time: 0.981000900268555, other_time: 11.2339477539063
epoch time: 67.382682800293s, input_fired_time: 0.935696959495544, output_fired_time: 0, propagate_forward_time: 64.56103515625, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.88594818115234
validation[          , genome  5567] predictions:    4971/  10000 (49.71%), best:    4971/10000 (49.71%), error:     18049.75781, best error:     18049.75781 on epoch:     0, epoch:    0/50, mu: 0.5000000000, learning_rate: 0.0125000002, weight_decay: 0.0005000000

epoch time: 876.0041503906s, input_fired_time: 7.9118299484, output_fired_time: 6.8330883980, propagate_forward_time: 325.7012023926, propagate_backward_time: 525.0012817383, weight_update_time: 0.9468885064, other_time: 10.5568237305
epoch time: 68.0488662720s, input_fired_time: 0.9188504815, output_fired_time: 0.0000000000, propagate_forward_time: 65.1904220581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9395904541
validation[          , genome  5567] predictions:    5037/  10000 (50.37%), best:    5037/10000 (50.37%), error:     16719.55078, best error:     16719.55078 on epoch:     1, epoch:    1/50, mu: 0.5245000124, learning_rate: 0.0118749999, weight_decay: 0.0004750000

epoch time: 874.5733032227s, input_fired_time: 7.8647351265, output_fired_time: 6.8280930519, propagate_forward_time: 324.4727478027, propagate_backward_time: 524.7998046875, weight_update_time: 0.9424182773, other_time: 10.6079101563
epoch time: 67.7962799072s, input_fired_time: 0.9255020618, output_fired_time: 0.0000000000, propagate_forward_time: 64.9636917114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9070892334
validation[          , genome  5567] predictions:    5265/  10000 (52.65%), best:    5265/10000 (52.65%), error:     15182.16406, best error:     15182.16406 on epoch:     2, epoch:    2/50, mu: 0.5477750301, learning_rate: 0.0112812500, weight_decay: 0.0004512500

epoch time: 874.5686035156s, input_fired_time: 7.8313670158, output_fired_time: 6.8071250916, propagate_forward_time: 325.4085083008, propagate_backward_time: 523.9602661133, weight_update_time: 0.9421633482, other_time: 10.5613403320
epoch time: 67.4332809448s, input_fired_time: 0.9141283631, output_fired_time: 0.0000000000, propagate_forward_time: 64.5887985229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9303512573
validation[          , genome  5567] predictions:    4790/  10000 (47.90%), best:    5265/10000 (52.65%), error:     18148.96289, best error:     15182.16406 on epoch:     2, epoch:    3/50, mu: 0.5698862672, learning_rate: 0.0107171871, weight_decay: 0.0004286875

epoch time: 1024.5279541016s, input_fired_time: 8.9571342468, output_fired_time: 8.0334138870, propagate_forward_time: 384.0946044922, propagate_backward_time: 611.0834350586, weight_update_time: 1.0515320301, other_time: 12.3593750000
epoch time: 82.8382492065s, input_fired_time: 1.1023756266, output_fired_time: 0.0000000000, propagate_forward_time: 79.4476776123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2881927490
validation[          , genome  5567] predictions:    5026/  10000 (50.26%), best:    5265/10000 (52.65%), error:     15739.38184, best error:     15182.16406 on epoch:     2, epoch:    4/50, mu: 0.5908919573, learning_rate: 0.0101813274, weight_decay: 0.0004072531

epoch time: 1004.3392333984s, input_fired_time: 8.8006429672, output_fired_time: 7.8392181396, propagate_forward_time: 376.1403198242, propagate_backward_time: 599.3834838867, weight_update_time: 1.0369783640, other_time: 12.1755371094
epoch time: 66.9316711426s, input_fired_time: 0.8894972801, output_fired_time: 0.0000000000, propagate_forward_time: 64.1573181152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8848571777
validation[          , genome  5567] predictions:    5227/  10000 (52.27%), best:    5227/10000 (52.27%), error:     15035.05176, best error:     15035.05176 on epoch:     5, epoch:    5/50, mu: 0.6108473539, learning_rate: 0.0096722608, weight_decay: 0.0003868904

epoch time: 864.6884155273s, input_fired_time: 7.7265291214, output_fired_time: 6.8286619186, propagate_forward_time: 320.4136962891, propagate_backward_time: 519.2753295898, weight_update_time: 0.9386031628, other_time: 10.4442138672
epoch time: 66.9616546631s, input_fired_time: 0.9068761468, output_fired_time: 0.0000000000, propagate_forward_time: 64.1934814453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8612976074
validation[          , genome  5567] predictions:    4892/  10000 (48.92%), best:    5227/10000 (52.27%), error:     16760.46875, best error:     15035.05176 on epoch:     5, epoch:    6/50, mu: 0.6298049688, learning_rate: 0.0091886474, weight_decay: 0.0003675459

epoch time: 868.9973754883s, input_fired_time: 7.7988595963, output_fired_time: 6.7492847443, propagate_forward_time: 323.3974304199, propagate_backward_time: 520.5855712891, weight_update_time: 0.9368937016, other_time: 10.4661865234
epoch time: 66.7551193237s, input_fired_time: 0.9055718184, output_fired_time: 0.0000000000, propagate_forward_time: 63.9741744995, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8753738403
validation[          , genome  5567] predictions:    5088/  10000 (50.88%), best:    5227/10000 (52.27%), error:     15667.83691, best error:     15035.05176 on epoch:     5, epoch:    7/50, mu: 0.6478147507, learning_rate: 0.0087292148, weight_decay: 0.0003491686

epoch time: 864.2407836914s, input_fired_time: 7.7395067215, output_fired_time: 6.7581806183, propagate_forward_time: 320.7050170898, propagate_backward_time: 518.6182250977, weight_update_time: 0.9356520176, other_time: 10.4198608398
epoch time: 67.3180847168s, input_fired_time: 0.9157496095, output_fired_time: 0.0000000000, propagate_forward_time: 64.5161895752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8861465454
validation[          , genome  5567] predictions:    4974/  10000 (49.74%), best:    5227/10000 (52.27%), error:     16805.98242, best error:     15035.05176 on epoch:     5, epoch:    8/50, mu: 0.6649240255, learning_rate: 0.0082927542, weight_decay: 0.0003317102

epoch time: 863.3015747070s, input_fired_time: 7.7501220703, output_fired_time: 6.7508673668, propagate_forward_time: 321.1083984375, propagate_backward_time: 517.2439575195, weight_update_time: 0.9369264841, other_time: 10.4482421875
epoch time: 66.6882019043s, input_fired_time: 0.9025228024, output_fired_time: 0.0000000000, propagate_forward_time: 63.9159851074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8696975708
validation[          , genome  5567] predictions:    5158/  10000 (51.58%), best:    5227/10000 (52.27%), error:     15499.62988, best error:     15035.05176 on epoch:     5, epoch:    9/50, mu: 0.6811778545, learning_rate: 0.0078781163, weight_decay: 0.0003151247

epoch time: 860.0780639648s, input_fired_time: 7.7096400261, output_fired_time: 6.7351951599, propagate_forward_time: 318.9669494629, propagate_backward_time: 516.2885742188, weight_update_time: 0.9268140197, other_time: 10.3776855469
epoch time: 65.8245162964s, input_fired_time: 0.8972358108, output_fired_time: 0.0000000000, propagate_forward_time: 63.0498580933, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8774261475
validation[          , genome  5567] predictions:    5219/  10000 (52.19%), best:    5219/10000 (52.19%), error:     14989.48926, best error:     14989.48926 on epoch:    10, epoch:   10/50, mu: 0.6966189742, learning_rate: 0.0074842102, weight_decay: 0.0002993684

epoch time: 860.2860107422s, input_fired_time: 7.7193202972, output_fired_time: 6.7348423004, propagate_forward_time: 320.3475952148, propagate_backward_time: 515.0729370117, weight_update_time: 0.9304215908, other_time: 10.4113159180
epoch time: 66.2144851685s, input_fired_time: 0.8912349939, output_fired_time: 0.0000000000, propagate_forward_time: 63.4644584656, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8587913513
validation[          , genome  5567] predictions:    5125/  10000 (51.25%), best:    5219/10000 (52.19%), error:     15392.08789, best error:     14989.48926 on epoch:    10, epoch:   11/50, mu: 0.7112880349, learning_rate: 0.0071099997, weight_decay: 0.0002844000

epoch time: 861.4569091797s, input_fired_time: 7.7263078690, output_fired_time: 6.7133946419, propagate_forward_time: 319.1591796875, propagate_backward_time: 517.4078979492, weight_update_time: 0.9289112091, other_time: 10.4501342773
epoch time: 66.0998382568s, input_fired_time: 0.9081135988, output_fired_time: 0.0000000000, propagate_forward_time: 63.3671226501, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8246040344
validation[          , genome  5567] predictions:    5350/  10000 (53.50%), best:    5350/10000 (53.50%), error:     14404.77344, best error:     14404.77344 on epoch:    12, epoch:   12/50, mu: 0.7252236605, learning_rate: 0.0067544994, weight_decay: 0.0002701800

epoch time: 854.4165649414s, input_fired_time: 7.7218041420, output_fired_time: 6.6748199463, propagate_forward_time: 316.6303100586, propagate_backward_time: 513.0132446289, weight_update_time: 0.9280652404, other_time: 10.3764038086
epoch time: 65.8250503540s, input_fired_time: 0.8956930041, output_fired_time: 0.0000000000, propagate_forward_time: 63.0336685181, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8956909180
validation[          , genome  5567] predictions:    5277/  10000 (52.77%), best:    5350/10000 (53.50%), error:     14587.80078, best error:     14404.77344 on epoch:    12, epoch:   13/50, mu: 0.7384625077, learning_rate: 0.0064167744, weight_decay: 0.0002566710

epoch time: 854.0318603516s, input_fired_time: 7.7196092606, output_fired_time: 6.6398196220, propagate_forward_time: 316.7503662109, propagate_backward_time: 512.5501098633, weight_update_time: 0.9235453606, other_time: 10.3719482422
epoch time: 66.2392654419s, input_fired_time: 0.9034189582, output_fired_time: 0.0000000000, propagate_forward_time: 63.4781761169, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8576698303
validation[          , genome  5567] predictions:    5343/  10000 (53.43%), best:    5350/10000 (53.50%), error:     14768.34375, best error:     14404.77344 on epoch:    12, epoch:   14/50, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 859.7211303711s, input_fired_time: 7.7645206451, output_fired_time: 6.7189345360, propagate_forward_time: 319.3573608398, propagate_backward_time: 515.4530639648, weight_update_time: 0.9342337847, other_time: 10.4272460938
epoch time: 67.1018447876s, input_fired_time: 0.9087443948, output_fired_time: 0.0000000000, propagate_forward_time: 64.3440856934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8490142822
validation[          , genome  5567] predictions:    5419/  10000 (54.19%), best:    5350/10000 (53.50%), error:     14589.78320, best error:     14404.77344 on epoch:    12, epoch:   15/50, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 858.0275268555s, input_fired_time: 7.7436223030, output_fired_time: 6.7097916603, propagate_forward_time: 318.0065917969, propagate_backward_time: 515.1755371094, weight_update_time: 0.9245076776, other_time: 10.3919677734
epoch time: 67.2992858887s, input_fired_time: 0.9108314514, output_fired_time: 0.0000000000, propagate_forward_time: 64.4674148560, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9210433960
validation[          , genome  5567] predictions:    5522/  10000 (55.22%), best:    5522/10000 (55.22%), error:     13426.89258, best error:     13426.89258 on epoch:    16, epoch:   16/50, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 854.3837280273s, input_fired_time: 7.7332158089, output_fired_time: 6.6614651680, propagate_forward_time: 317.5487365723, propagate_backward_time: 511.9759521484, weight_update_time: 0.9315710068, other_time: 10.4643554688
epoch time: 66.3751907349s, input_fired_time: 0.9039594531, output_fired_time: 0.0000000000, propagate_forward_time: 63.6012496948, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8699798584
validation[          , genome  5567] predictions:    5401/  10000 (54.01%), best:    5522/10000 (55.22%), error:     14043.66895, best error:     13426.89258 on epoch:    16, epoch:   17/50, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 857.0673828125s, input_fired_time: 7.7596912384, output_fired_time: 6.6868200302, propagate_forward_time: 317.2573852539, propagate_backward_time: 514.9329833984, weight_update_time: 0.9307964444, other_time: 10.4304809570
epoch time: 67.4964447021s, input_fired_time: 0.9263790846, output_fired_time: 0.0000000000, propagate_forward_time: 64.6859283447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8841400146
validation[          , genome  5567] predictions:    5610/  10000 (56.10%), best:    5610/10000 (56.10%), error:     13296.07129, best error:     13296.07129 on epoch:    18, epoch:   18/50, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 860.0650634766s, input_fired_time: 7.8077802658, output_fired_time: 6.7354907990, propagate_forward_time: 319.7147827148, propagate_backward_time: 515.3700561523, weight_update_time: 0.9331733584, other_time: 10.4369506836
epoch time: 66.0959701538s, input_fired_time: 0.9177929759, output_fired_time: 0.0000000000, propagate_forward_time: 63.2964019775, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8817749023
validation[          , genome  5567] predictions:    5298/  10000 (52.98%), best:    5610/10000 (56.10%), error:     14481.90234, best error:     13296.07129 on epoch:    18, epoch:   19/50, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

epoch time: 862.3845214844s, input_fired_time: 7.8296165466, output_fired_time: 6.7611536980, propagate_forward_time: 320.3211364746, propagate_backward_time: 517.0859375000, weight_update_time: 0.9309262633, other_time: 10.3867187500
epoch time: 66.3007736206s, input_fired_time: 0.9171631336, output_fired_time: 0.0000000000, propagate_forward_time: 63.5042495728, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8793640137
validation[          , genome  5567] predictions:    5643/  10000 (56.43%), best:    5610/10000 (56.10%), error:     13376.84180, best error:     13296.07129 on epoch:    18, epoch:   20/50, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 858.2105712891s, input_fired_time: 7.7914566994, output_fired_time: 6.7081704140, propagate_forward_time: 318.5050659180, propagate_backward_time: 514.8012084961, weight_update_time: 0.9308770299, other_time: 10.4046630859
epoch time: 66.5538101196s, input_fired_time: 0.9213383198, output_fired_time: 0.0000000000, propagate_forward_time: 63.7751770020, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8572921753
validation[          , genome  5567] predictions:    5673/  10000 (56.73%), best:    5610/10000 (56.10%), error:     13327.73535, best error:     13296.07129 on epoch:    18, epoch:   21/50, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 857.5895996094s, input_fired_time: 7.7981133461, output_fired_time: 6.7139906883, propagate_forward_time: 318.5101623535, propagate_backward_time: 514.1201171875, weight_update_time: 0.9294283986, other_time: 10.4472656250
epoch time: 66.3067169189s, input_fired_time: 0.9132964611, output_fired_time: 0.0000000000, propagate_forward_time: 63.5331611633, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8602561951
validation[          , genome  5567] predictions:    5441/  10000 (54.41%), best:    5610/10000 (56.10%), error:     13991.79980, best error:     13296.07129 on epoch:    18, epoch:   22/50, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 855.5780639648s, input_fired_time: 7.8031725883, output_fired_time: 6.6684408188, propagate_forward_time: 317.8080749512, propagate_backward_time: 512.9179077148, weight_update_time: 0.9327712655, other_time: 10.3804321289
epoch time: 66.4942932129s, input_fired_time: 0.9221447706, output_fired_time: 0.0000000000, propagate_forward_time: 63.6927833557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8793678284
validation[          , genome  5567] predictions:    5784/  10000 (57.84%), best:    5784/10000 (57.84%), error:     13049.65039, best error:     13049.65039 on epoch:    23, epoch:   23/50, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 856.0412597656s, input_fired_time: 7.8169612885, output_fired_time: 6.6698451042, propagate_forward_time: 318.1387634277, propagate_backward_time: 513.0250854492, weight_update_time: 0.9286445975, other_time: 10.3905639648
epoch time: 66.5988616943s, input_fired_time: 0.9238525629, output_fired_time: 0.0000000000, propagate_forward_time: 63.7961883545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8788223267
validation[          , genome  5567] predictions:    5737/  10000 (57.37%), best:    5784/10000 (57.84%), error:     13152.58105, best error:     13049.65039 on epoch:    23, epoch:   24/50, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 856.9971923828s, input_fired_time: 7.8174810410, output_fired_time: 6.6317329407, propagate_forward_time: 317.8842773438, propagate_backward_time: 514.2426757813, weight_update_time: 0.9388153553, other_time: 10.4210205078
epoch time: 66.7815246582s, input_fired_time: 0.9196217656, output_fired_time: 0.0000000000, propagate_forward_time: 63.9920845032, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8698158264
validation[          , genome  5567] predictions:    5609/  10000 (56.09%), best:    5784/10000 (57.84%), error:     13438.06250, best error:     13049.65039 on epoch:    23, epoch:   25/50, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 854.3352050781s, input_fired_time: 7.8055176735, output_fired_time: 6.6606588364, propagate_forward_time: 317.3767089844, propagate_backward_time: 512.1481933594, weight_update_time: 0.9315918684, other_time: 10.3441162109
epoch time: 65.2873916626s, input_fired_time: 0.9070756435, output_fired_time: 0.0000000000, propagate_forward_time: 62.5213470459, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8589706421
validation[          , genome  5567] predictions:    5534/  10000 (55.34%), best:    5784/10000 (57.84%), error:     13504.11035, best error:     13049.65039 on epoch:    23, epoch:   26/50, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 853.5402832031s, input_fired_time: 7.7958087921, output_fired_time: 6.6727728844, propagate_forward_time: 316.7984313965, propagate_backward_time: 511.9029541016, weight_update_time: 0.9297659993, other_time: 10.3702392578
epoch time: 66.4791183472s, input_fired_time: 0.9165714979, output_fired_time: 0.0000000000, propagate_forward_time: 63.6963844299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8661613464
validation[          , genome  5567] predictions:    5699/  10000 (56.99%), best:    5784/10000 (57.84%), error:     13454.15625, best error:     13049.65039 on epoch:    23, epoch:   27/50, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 855.1668090820s, input_fired_time: 7.8224859238, output_fired_time: 6.7402997017, propagate_forward_time: 317.3428344727, propagate_backward_time: 512.8575439453, weight_update_time: 0.9314986467, other_time: 10.4036254883
epoch time: 66.7155227661s, input_fired_time: 0.9324207902, output_fired_time: 0.0000000000, propagate_forward_time: 63.9240722656, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8590316772
validation[          , genome  5567] predictions:    5590/  10000 (55.90%), best:    5784/10000 (57.84%), error:     14003.98926, best error:     13049.65039 on epoch:    23, epoch:   28/50, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 855.4346923828s, input_fired_time: 7.8560237885, output_fired_time: 6.7006087303, propagate_forward_time: 318.6243896484, propagate_backward_time: 511.8620300293, weight_update_time: 0.9321479201, other_time: 10.3916320801
epoch time: 66.2915420532s, input_fired_time: 0.9304910302, output_fired_time: 0.0000000000, propagate_forward_time: 63.5040512085, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8570022583
validation[          , genome  5567] predictions:    5590/  10000 (55.90%), best:    5784/10000 (57.84%), error:     13587.95801, best error:     13049.65039 on epoch:    23, epoch:   29/50, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 852.8592529297s, input_fired_time: 7.7753024101, output_fired_time: 6.7051749229, propagate_forward_time: 315.9625549316, propagate_backward_time: 512.0635986328, weight_update_time: 0.9202650189, other_time: 10.3525390625
epoch time: 66.0321197510s, input_fired_time: 0.9198971987, output_fired_time: 0.0000000000, propagate_forward_time: 63.2877159119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8245048523
validation[          , genome  5567] predictions:    5664/  10000 (56.64%), best:    5784/10000 (57.84%), error:     13504.81641, best error:     13049.65039 on epoch:    23, epoch:   30/50, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 854.8519287109s, input_fired_time: 7.8526654243, output_fired_time: 6.6720805168, propagate_forward_time: 317.9441223145, propagate_backward_time: 511.9708251953, weight_update_time: 0.9296436310, other_time: 10.4122314453
epoch time: 66.9297866821s, input_fired_time: 0.9393376112, output_fired_time: 0.0000000000, propagate_forward_time: 64.0835876465, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9068603516
validation[          , genome  5567] predictions:    5698/  10000 (56.98%), best:    5784/10000 (57.84%), error:     13204.95605, best error:     13049.65039 on epoch:    23, epoch:   31/50, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 861.7470703125s, input_fired_time: 7.9141173363, output_fired_time: 6.7225117683, propagate_forward_time: 320.3775329590, propagate_backward_time: 516.3093872070, weight_update_time: 0.9388619065, other_time: 10.4235229492
epoch time: 66.7091827393s, input_fired_time: 0.9357667565, output_fired_time: 0.0000000000, propagate_forward_time: 63.8863143921, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8871002197
validation[          , genome  5567] predictions:    5747/  10000 (57.47%), best:    5747/10000 (57.47%), error:     12961.57422, best error:     12961.57422 on epoch:    32, epoch:   32/50, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 856.2932739258s, input_fired_time: 7.8838191032, output_fired_time: 6.7328972816, propagate_forward_time: 317.3653259277, propagate_backward_time: 513.9218750000, weight_update_time: 0.9228929877, other_time: 10.3894042969
epoch time: 67.4543228149s, input_fired_time: 0.9527779818, output_fired_time: 0.0000000000, propagate_forward_time: 64.6149139404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8866271973
validation[          , genome  5567] predictions:    5838/  10000 (58.38%), best:    5838/10000 (58.38%), error:     12556.94336, best error:     12556.94336 on epoch:    33, epoch:   33/50, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 855.3075561523s, input_fired_time: 7.8666043282, output_fired_time: 6.7025876045, propagate_forward_time: 316.7520751953, propagate_backward_time: 513.6408691406, weight_update_time: 0.9324114323, other_time: 10.3454589844
epoch time: 65.5300674438s, input_fired_time: 0.9282110929, output_fired_time: 0.0000000000, propagate_forward_time: 62.7678642273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8339958191
validation[          , genome  5567] predictions:    5760/  10000 (57.60%), best:    5838/10000 (58.38%), error:     12955.51660, best error:     12556.94336 on epoch:    33, epoch:   34/50, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 857.2041625977s, input_fired_time: 7.8982300758, output_fired_time: 6.6981010437, propagate_forward_time: 319.0467834473, propagate_backward_time: 513.2151489258, weight_update_time: 0.9223824143, other_time: 10.3458862305
epoch time: 65.3817062378s, input_fired_time: 0.9270160794, output_fired_time: 0.0000000000, propagate_forward_time: 62.5835952759, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8710937500
validation[          , genome  5567] predictions:    5750/  10000 (57.50%), best:    5838/10000 (58.38%), error:     12919.43555, best error:     12556.94336 on epoch:    33, epoch:   35/50, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 850.8621826172s, input_fired_time: 7.8721237183, output_fired_time: 6.6848635674, propagate_forward_time: 315.8683471680, propagate_backward_time: 510.0642700195, weight_update_time: 0.9292374253, other_time: 10.3725585938
epoch time: 66.3851623535s, input_fired_time: 0.9358103275, output_fired_time: 0.0000000000, propagate_forward_time: 63.5887413025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8606071472
validation[          , genome  5567] predictions:    5812/  10000 (58.12%), best:    5838/10000 (58.38%), error:     12790.37207, best error:     12556.94336 on epoch:    33, epoch:   36/50, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 854.3729248047s, input_fired_time: 7.8738722801, output_fired_time: 6.6730890274, propagate_forward_time: 317.1045532227, propagate_backward_time: 512.3650512695, weight_update_time: 0.9279546142, other_time: 10.3563232422
epoch time: 66.3022232056s, input_fired_time: 0.9358862638, output_fired_time: 0.0000000000, propagate_forward_time: 63.4840011597, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8823394775
validation[          , genome  5567] predictions:    5703/  10000 (57.03%), best:    5838/10000 (58.38%), error:     13199.92480, best error:     12556.94336 on epoch:    33, epoch:   37/50, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 851.7121582031s, input_fired_time: 7.8571009636, output_fired_time: 6.6126527786, propagate_forward_time: 316.1480407715, propagate_backward_time: 510.7646789551, weight_update_time: 0.9270028472, other_time: 10.3296813965
epoch time: 65.7943191528s, input_fired_time: 0.9331874251, output_fired_time: 0.0000000000, propagate_forward_time: 63.0067825317, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8543472290
validation[          , genome  5567] predictions:    5821/  10000 (58.21%), best:    5821/10000 (58.21%), error:     12487.85254, best error:     12487.85254 on epoch:    38, epoch:   38/50, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 853.9356079102s, input_fired_time: 7.8810381889, output_fired_time: 6.6727104187, propagate_forward_time: 316.2594909668, propagate_backward_time: 512.7243041992, weight_update_time: 0.9309014678, other_time: 10.3980102539
epoch time: 65.8099060059s, input_fired_time: 0.9456337094, output_fired_time: 0.0000000000, propagate_forward_time: 63.0122222900, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8520507813
validation[          , genome  5567] predictions:    5755/  10000 (57.55%), best:    5821/10000 (58.21%), error:     12842.80176, best error:     12487.85254 on epoch:    38, epoch:   39/50, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 851.7098999023s, input_fired_time: 7.8831453323, output_fired_time: 6.6452651024, propagate_forward_time: 316.1859130859, propagate_backward_time: 510.6272583008, weight_update_time: 0.9271666408, other_time: 10.3683471680
epoch time: 66.1791610718s, input_fired_time: 0.9381277561, output_fired_time: 0.0000000000, propagate_forward_time: 63.3928871155, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8481483459
validation[          , genome  5567] predictions:    5999/  10000 (59.99%), best:    5999/10000 (59.99%), error:     12155.98145, best error:     12155.98145 on epoch:    40, epoch:   40/50, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 852.7219848633s, input_fired_time: 7.8893246651, output_fired_time: 6.6768188477, propagate_forward_time: 316.6490478516, propagate_backward_time: 511.1785278320, weight_update_time: 0.9293610454, other_time: 10.3282470703
epoch time: 66.9590148926s, input_fired_time: 0.9490767121, output_fired_time: 0.0000000000, propagate_forward_time: 64.1329956055, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8769454956
validation[          , genome  5567] predictions:    5875/  10000 (58.75%), best:    5999/10000 (59.99%), error:     12757.87598, best error:     12155.98145 on epoch:    40, epoch:   41/50, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 856.9135131836s, input_fired_time: 7.9368023872, output_fired_time: 6.6897997856, propagate_forward_time: 318.5930480957, propagate_backward_time: 513.3171997070, weight_update_time: 0.9262216687, other_time: 10.3766479492
epoch time: 64.4961700439s, input_fired_time: 0.9356702566, output_fired_time: 0.0000000000, propagate_forward_time: 61.6901550293, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8703460693
validation[          , genome  5567] predictions:    5868/  10000 (58.68%), best:    5999/10000 (59.99%), error:     12740.69434, best error:     12155.98145 on epoch:    40, epoch:   42/50, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 861.6437377930s, input_fired_time: 7.9729752541, output_fired_time: 6.7419691086, propagate_forward_time: 319.6846618652, propagate_backward_time: 516.7525634766, weight_update_time: 0.9354791641, other_time: 10.4915771484
epoch time: 66.4007720947s, input_fired_time: 0.9485666752, output_fired_time: 0.0000000000, propagate_forward_time: 63.5824584961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8697433472
validation[          , genome  5567] predictions:    5897/  10000 (58.97%), best:    5999/10000 (59.99%), error:     12608.96289, best error:     12155.98145 on epoch:    40, epoch:   43/50, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 858.8933715820s, input_fired_time: 7.9540929794, output_fired_time: 6.7080912590, propagate_forward_time: 319.0206298828, propagate_backward_time: 514.8701171875, weight_update_time: 0.9295987487, other_time: 10.3404541016
epoch time: 66.9499130249s, input_fired_time: 0.9547864795, output_fired_time: 0.0000000000, propagate_forward_time: 64.1225814819, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8725433350
validation[          , genome  5567] predictions:    5988/  10000 (59.88%), best:    5999/10000 (59.99%), error:     12408.76270, best error:     12155.98145 on epoch:    40, epoch:   44/50, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 856.7364501953s, input_fired_time: 7.9542031288, output_fired_time: 6.7408790588, propagate_forward_time: 318.4094848633, propagate_backward_time: 513.2453613281, weight_update_time: 0.9279850721, other_time: 10.3864746094
epoch time: 66.0309600830s, input_fired_time: 0.9502956867, output_fired_time: 0.0000000000, propagate_forward_time: 63.2243118286, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8563537598
validation[          , genome  5567] predictions:    5884/  10000 (58.84%), best:    5999/10000 (59.99%), error:     12663.25000, best error:     12155.98145 on epoch:    40, epoch:   45/50, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 854.5269165039s, input_fired_time: 7.9233775139, output_fired_time: 6.6696739197, propagate_forward_time: 317.5694274902, propagate_backward_time: 511.9960937500, weight_update_time: 0.9262734056, other_time: 10.3682861328
epoch time: 67.8120956421s, input_fired_time: 0.9683901072, output_fired_time: 0.0000000000, propagate_forward_time: 64.9401931763, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9035110474
validation[          , genome  5567] predictions:    5826/  10000 (58.26%), best:    5999/10000 (59.99%), error:     12723.71289, best error:     12155.98145 on epoch:    40, epoch:   46/50, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 853.4688720703s, input_fired_time: 7.9157013893, output_fired_time: 6.6378412247, propagate_forward_time: 316.2660217285, propagate_backward_time: 512.2943725586, weight_update_time: 0.9359688759, other_time: 10.3549194336
epoch time: 66.2825622559s, input_fired_time: 0.9518932104, output_fired_time: 0.0000000000, propagate_forward_time: 63.4815940857, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8490715027
validation[          , genome  5567] predictions:    5966/  10000 (59.66%), best:    5999/10000 (59.99%), error:     12169.33594, best error:     12155.98145 on epoch:    40, epoch:   47/50, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 850.9013671875s, input_fired_time: 7.9190144539, output_fired_time: 6.6509170532, propagate_forward_time: 315.7701416016, propagate_backward_time: 510.1731872559, weight_update_time: 0.9296822548, other_time: 10.3880920410
epoch time: 66.6023864746s, input_fired_time: 0.9584515691, output_fired_time: 0.0000000000, propagate_forward_time: 63.7781791687, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8657569885
validation[          , genome  5567] predictions:    5958/  10000 (59.58%), best:    5999/10000 (59.99%), error:     12501.25000, best error:     12155.98145 on epoch:    40, epoch:   48/50, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 862.0583496094s, input_fired_time: 8.0035400391, output_fired_time: 6.7102160454, propagate_forward_time: 319.9486083984, propagate_backward_time: 516.9248046875, weight_update_time: 0.9352517128, other_time: 10.4711914063
epoch time: 66.7708282471s, input_fired_time: 0.9567993283, output_fired_time: 0.0000000000, propagate_forward_time: 63.9357032776, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8783226013
validation[          , genome  5567] predictions:    5923/  10000 (59.23%), best:    5999/10000 (59.99%), error:     12599.12402, best error:     12155.98145 on epoch:    40, epoch:   49/50, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 859.0396728516s, input_fired_time: 8.0036935806, output_fired_time: 6.6635489464, propagate_forward_time: 319.6872558594, propagate_backward_time: 514.2961425781, weight_update_time: 0.9356102943, other_time: 10.3889770508
epoch time: 67.0672073364s, input_fired_time: 0.9634506106, output_fired_time: 0.0000000000, propagate_forward_time: 64.2145309448, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8892288208
validation[          , genome  5567] predictions:    5890/  10000 (58.90%), best:    5999/10000 (59.99%), error:     12587.48047, best error:     12155.98145 on epoch:    40, epoch:   50/50, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 332.1984863281s, input_fired_time: 4.7145094872, output_fired_time: 0.0000000000, propagate_forward_time: 318.0207824707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.4631958008
best training[          , genome  5567] predictions:   34837/  50000 (69.67%), best:    5999/10000 (59.99%), error:     43681.60547, best error:     12155.98145 on epoch:    40, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 66.8681182861s, input_fired_time: 0.9442268014, output_fired_time: 0.0000000000, propagate_forward_time: 64.0682067871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8556823730
   testing[          , genome  5567] predictions:    5999/  10000 (59.99%), best:    5999/10000 (59.99%), error:     12155.98145, best error:     12155.98145 on epoch:    40, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
backpropagation finished successfully!
01:11:14 (9808): called boinc_finish(0)

</stderr_txt>
]]>