Nameexact_genome_1542605141_178_6296_0
Workunit4098481
Created19 Nov 2018, 6:41:20 UTC
Sent19 Nov 2018, 7:05:33 UTC
Report deadline30 Jan 2019, 1:21:09 UTC
Received20 Nov 2018, 20:02:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84860
Run time1 days 5 hours 55 min 13 sec
CPU time1 days 5 hours 36 min 46 sec
Validate stateValid
Credit17,940.21
Device peak FLOPS4.41 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size547.00 MB
Peak swap size549.73 MB
Peak disk usage10.75 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6296.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6296_0_r2063986718_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6296.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494775
read mu: 0.494775
read mu_delta: 0.957276
read initial_learning_rate: 0.000439578
read learning_rate: 0.000439578
read learning_rate_delta: 0.920913
read initial_weight_decay: 0.000423226
read weight_decay: 0.000423226
read weight_decay_delta: 0.923837
read batch_size: 96
read epsilon: 1e-07
read alpha: 0.0879971
read input_dropout_probability: 0.00110144
read hidden_dropout_probability: 0.110596
read velocity_reset: 2154
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6296
read normal distribution: '1 0 0'
generator_str: '1107740576'
read generator: 1107740576
read generated_by_map:
1 crossover 1
reading 113 nodes.
reading 598 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1260124989
post shuffle 1: 538252679
epoch time: 1918.75952148438s, input_fired_time: 18.4455909729004, output_fired_time: 6.59783124923706, propagate_forward_time: 745.392578125, propagate_backward_time: 1136.39709472656, weight_update_time: 0.540557026863098, other_time: 11.926513671875
epoch time: 153.565521240234s, input_fired_time: 0.927537679672241, output_fired_time: 0, propagate_forward_time: 150.630004882812, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.00798034667969
validation[          , genome  6296] predictions:    5942/  10000 (59.42%), best:    5942/10000 (59.42%), error:     12018.20996, best error:     12018.20996 on epoch:     0, epoch:    0/50, mu: 0.4947752953, learning_rate: 0.0004395784, weight_decay: 0.0004232263

epoch time: 1931.9086914062s, input_fired_time: 18.1939086914, output_fired_time: 6.6396226883, propagate_forward_time: 746.6296997070, propagate_backward_time: 1148.4995117188, weight_update_time: 0.5289947391, other_time: 11.9460449219
epoch time: 156.9048309326s, input_fired_time: 0.9789308310, output_fired_time: 0.0000000000, propagate_forward_time: 153.9373321533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9885711670
validation[          , genome  6296] predictions:    6176/  10000 (61.76%), best:    6176/10000 (61.76%), error:     11079.82812, best error:     11079.82812 on epoch:     1, epoch:    1/50, mu: 0.5159330964, learning_rate: 0.0004048132, weight_decay: 0.0003909922

epoch time: 1920.5014648438s, input_fired_time: 18.2096004486, output_fired_time: 6.6319332123, propagate_forward_time: 739.7448730469, propagate_backward_time: 1144.0151367188, weight_update_time: 0.5294479728, other_time: 11.8999023438
epoch time: 153.6091461182s, input_fired_time: 0.9164304137, output_fired_time: 0.0000000000, propagate_forward_time: 150.7473907471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9453277588
validation[          , genome  6296] predictions:    5993/  10000 (59.93%), best:    6176/10000 (61.76%), error:     11623.84473, best error:     11079.82812 on epoch:     1, epoch:    2/50, mu: 0.5361869335, learning_rate: 0.0003727975, weight_decay: 0.0003612132

epoch time: 1908.5395507812s, input_fired_time: 18.8300304413, output_fired_time: 6.5299258232, propagate_forward_time: 752.6434936523, propagate_backward_time: 1118.3516845703, weight_update_time: 0.5608985424, other_time: 12.1844482422
epoch time: 157.2247619629s, input_fired_time: 0.9502763152, output_fired_time: 0.0000000000, propagate_forward_time: 154.3059082031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9685821533
validation[          , genome  6296] predictions:    5947/  10000 (59.47%), best:    6176/10000 (61.76%), error:     11861.22656, best error:     11079.82812 on epoch:     1, epoch:    3/50, mu: 0.5555754304, learning_rate: 0.0003433139, weight_decay: 0.0003337023

epoch time: 1930.2503662109s, input_fired_time: 19.1780490875, output_fired_time: 6.6836814880, propagate_forward_time: 758.7095336914, propagate_backward_time: 1133.3094482422, weight_update_time: 0.5734753013, other_time: 12.3695068359
epoch time: 152.9048614502s, input_fired_time: 0.9666231275, output_fired_time: 0.0000000000, propagate_forward_time: 150.0190277100, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9192047119
validation[          , genome  6296] predictions:    6075/  10000 (60.75%), best:    6176/10000 (61.76%), error:     11468.53906, best error:     11079.82812 on epoch:     1, epoch:    4/50, mu: 0.5741356015, learning_rate: 0.0003161621, weight_decay: 0.0003082866

epoch time: 1883.8194580078s, input_fired_time: 18.7042675018, output_fired_time: 6.5444426537, propagate_forward_time: 741.0393066406, propagate_backward_time: 1105.6739501953, weight_update_time: 0.5606371760, other_time: 11.8575439453
epoch time: 152.8136138916s, input_fired_time: 0.9849237204, output_fired_time: 0.0000000000, propagate_forward_time: 149.7933959961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0352935791
validation[          , genome  6296] predictions:    6075/  10000 (60.75%), best:    6176/10000 (61.76%), error:     11554.66504, best error:     11079.82812 on epoch:     1, epoch:    5/50, mu: 0.5919028521, learning_rate: 0.0002911576, weight_decay: 0.0002848067

epoch time: 1897.2252197266s, input_fired_time: 18.7710952759, output_fired_time: 6.5569176674, propagate_forward_time: 749.3449707031, propagate_backward_time: 1110.5051269531, weight_update_time: 0.5643557310, other_time: 12.0471191406
epoch time: 154.5181579590s, input_fired_time: 0.9423800707, output_fired_time: 0.0000000000, propagate_forward_time: 151.6118011475, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9639739990
validation[          , genome  6296] predictions:    5979/  10000 (59.79%), best:    6176/10000 (61.76%), error:     11862.23438, best error:     11079.82812 on epoch:     1, epoch:    6/50, mu: 0.6089109778, learning_rate: 0.0002681307, weight_decay: 0.0002631151

epoch time: 1943.5692138672s, input_fired_time: 19.2913703918, output_fired_time: 6.5824460983, propagate_forward_time: 779.4862060547, propagate_backward_time: 1125.5852050781, weight_update_time: 0.5839843750, other_time: 12.6240234375
epoch time: 157.5757751465s, input_fired_time: 1.0015071630, output_fired_time: 0.0000000000, propagate_forward_time: 154.4789733887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0952911377
validation[          , genome  6296] predictions:    6172/  10000 (61.72%), best:    6176/10000 (61.76%), error:     11434.63770, best error:     11079.82812 on epoch:     1, epoch:    7/50, mu: 0.6251924634, learning_rate: 0.0002469249, weight_decay: 0.0002430755

epoch time: 1878.6081542969s, input_fired_time: 18.9512195587, output_fired_time: 6.5210967064, propagate_forward_time: 745.2236328125, propagate_backward_time: 1095.7277832031, weight_update_time: 0.5822407603, other_time: 12.1844482422
epoch time: 142.5318450928s, input_fired_time: 0.9470528364, output_fired_time: 0.0000000000, propagate_forward_time: 139.7514495850, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8333435059
validation[          , genome  6296] predictions:    6086/  10000 (60.86%), best:    6176/10000 (61.76%), error:     11484.76562, best error:     11079.82812 on epoch:     1, epoch:    8/50, mu: 0.6407783628, learning_rate: 0.0002273962, weight_decay: 0.0002245623

epoch time: 1921.3688964844s, input_fired_time: 18.5516815186, output_fired_time: 6.5674448013, propagate_forward_time: 752.9039306641, propagate_backward_time: 1131.3770751953, weight_update_time: 0.5455899239, other_time: 11.9687500000
epoch time: 153.2101135254s, input_fired_time: 0.9543737173, output_fired_time: 0.0000000000, propagate_forward_time: 150.2401733398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0155639648
validation[          , genome  6296] predictions:    6119/  10000 (61.19%), best:    6176/10000 (61.76%), error:     11352.68066, best error:     11079.82812 on epoch:     1, epoch:    9/50, mu: 0.6556983590, learning_rate: 0.0002094120, weight_decay: 0.0002074590

epoch time: 1907.5141601562s, input_fired_time: 18.5577011108, output_fired_time: 6.7334995270, propagate_forward_time: 741.6018066406, propagate_backward_time: 1128.6280517578, weight_update_time: 0.5717340112, other_time: 11.9930419922
epoch time: 160.4041137695s, input_fired_time: 0.9695249200, output_fired_time: 0.0000000000, propagate_forward_time: 157.3618011475, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0727844238
validation[          , genome  6296] predictions:    6098/  10000 (60.98%), best:    6176/10000 (61.76%), error:     11433.20215, best error:     11079.82812 on epoch:     1, epoch:   10/50, mu: 0.6699809432, learning_rate: 0.0001928502, weight_decay: 0.0001916584

epoch time: 2015.4499511719s, input_fired_time: 19.8779850006, output_fired_time: 6.7314810753, propagate_forward_time: 816.9508056641, propagate_backward_time: 1158.9808349609, weight_update_time: 0.5975353122, other_time: 12.9089355469
epoch time: 180.8494567871s, input_fired_time: 1.0405539274, output_fired_time: 0.0000000000, propagate_forward_time: 177.6654205322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1434783936
validation[          , genome  6296] predictions:    6144/  10000 (61.44%), best:    6176/10000 (61.76%), error:     11254.94531, best error:     11079.82812 on epoch:     1, epoch:   11/50, mu: 0.6836532950, learning_rate: 0.0001775981, weight_decay: 0.0001770612

epoch time: 1917.5825195312s, input_fired_time: 18.6712341309, output_fired_time: 6.5586051941, propagate_forward_time: 754.7568359375, propagate_backward_time: 1125.2094726562, weight_update_time: 0.5656327009, other_time: 12.3863525391
epoch time: 156.7275085449s, input_fired_time: 0.9748393297, output_fired_time: 0.0000000000, propagate_forward_time: 153.6738433838, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0788269043
validation[          , genome  6296] predictions:    6096/  10000 (60.96%), best:    6176/10000 (61.76%), error:     11443.51758, best error:     11079.82812 on epoch:     1, epoch:   12/50, mu: 0.6967415214, learning_rate: 0.0001635523, weight_decay: 0.0001635757

epoch time: 1999.6026611328s, input_fired_time: 19.7557582855, output_fired_time: 6.7817382812, propagate_forward_time: 801.7984008789, propagate_backward_time: 1158.3181152344, weight_update_time: 0.6093014479, other_time: 12.9487304688
epoch time: 150.3769378662s, input_fired_time: 0.9959786534, output_fired_time: 0.0000000000, propagate_forward_time: 147.3350067139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0459594727
validation[          , genome  6296] predictions:    6131/  10000 (61.31%), best:    6176/10000 (61.76%), error:     11277.40039, best error:     11079.82812 on epoch:     1, epoch:   13/50, mu: 0.7092705965, learning_rate: 0.0001506174, weight_decay: 0.0001511174

epoch time: 1975.3209228516s, input_fired_time: 19.3759689331, output_fired_time: 6.6591186523, propagate_forward_time: 793.9066772461, propagate_backward_time: 1142.5888671875, weight_update_time: 0.5904092789, other_time: 12.7902832031
epoch time: 153.3080444336s, input_fired_time: 0.9179307222, output_fired_time: 0.0000000000, propagate_forward_time: 150.3553619385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0347442627
validation[          , genome  6296] predictions:    5654/  10000 (56.54%), best:    6176/10000 (61.76%), error:     12741.87988, best error:     11079.82812 on epoch:     1, epoch:   14/50, mu: 0.7212643623, learning_rate: 0.0001387054, weight_decay: 0.0001396079

epoch time: 1938.6943359375s, input_fired_time: 19.2185535431, output_fired_time: 6.6499848366, propagate_forward_time: 772.3092651367, propagate_backward_time: 1127.9185791016, weight_update_time: 0.5900359750, other_time: 12.5980224609
epoch time: 153.6827545166s, input_fired_time: 0.9219012856, output_fired_time: 0.0000000000, propagate_forward_time: 150.7491149902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0117340088
validation[          , genome  6296] predictions:    6218/  10000 (62.18%), best:    6218/10000 (62.18%), error:     11068.54785, best error:     11068.54785 on epoch:    15, epoch:   15/50, mu: 0.7327457070, learning_rate: 0.0001277356, weight_decay: 0.0001289749

epoch time: 1943.5573730469s, input_fired_time: 18.9322414398, output_fired_time: 6.5897321701, propagate_forward_time: 771.7225341797, propagate_backward_time: 1133.6538085938, weight_update_time: 0.5824952722, other_time: 12.6590576172
epoch time: 151.9252471924s, input_fired_time: 0.9688572288, output_fired_time: 0.0000000000, propagate_forward_time: 148.9662628174, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9901275635
validation[          , genome  6296] predictions:    6286/  10000 (62.86%), best:    6286/10000 (62.86%), error:     10898.79102, best error:     10898.79102 on epoch:    16, epoch:   16/50, mu: 0.7437365651, learning_rate: 0.0001176333, weight_decay: 0.0001191519

epoch time: 1915.1024169922s, input_fired_time: 19.1723136902, output_fired_time: 6.6715664864, propagate_forward_time: 759.9205322266, propagate_backward_time: 1116.7246093750, weight_update_time: 0.5978926420, other_time: 12.6134033203
epoch time: 144.5409088135s, input_fired_time: 0.9902249575, output_fired_time: 0.0000000000, propagate_forward_time: 141.6366577148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9140319824
validation[          , genome  6296] predictions:    6318/  10000 (63.18%), best:    6286/10000 (62.86%), error:     10919.38281, best error:     10898.79102 on epoch:    16, epoch:   17/50, mu: 0.7542577982, learning_rate: 0.0001083300, weight_decay: 0.0001100770

epoch time: 1907.7705078125s, input_fired_time: 19.1554374695, output_fired_time: 6.5241589546, propagate_forward_time: 760.2415771484, propagate_backward_time: 1109.3753662109, weight_update_time: 0.5840047598, other_time: 12.4739990234
epoch time: 159.4020538330s, input_fired_time: 1.0066530704, output_fired_time: 0.0000000000, propagate_forward_time: 156.2799530029, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1154479980
validation[          , genome  6296] predictions:    6297/  10000 (62.97%), best:    6297/10000 (62.97%), error:     10816.26074, best error:     10816.26074 on epoch:    18, epoch:   18/50, mu: 0.7643295527, learning_rate: 0.0000997624, weight_decay: 0.0001016932

epoch time: 1969.2855224609s, input_fired_time: 19.6950664520, output_fired_time: 6.6486482620, propagate_forward_time: 793.9790039062, propagate_backward_time: 1135.9848632812, weight_update_time: 0.6046910882, other_time: 12.9779052734
epoch time: 169.4103698730s, input_fired_time: 0.9871118069, output_fired_time: 0.0000000000, propagate_forward_time: 166.3461303711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0771331787
validation[          , genome  6296] predictions:    6321/  10000 (63.21%), best:    6297/10000 (62.97%), error:     11004.35352, best error:     10816.26074 on epoch:    18, epoch:   19/50, mu: 0.7739710212, learning_rate: 0.0000918725, weight_decay: 0.0000939480

epoch time: 1935.8934326172s, input_fired_time: 19.1748275757, output_fired_time: 6.5788111687, propagate_forward_time: 770.2113647461, propagate_backward_time: 1127.4620361328, weight_update_time: 0.5746125579, other_time: 12.4664306641
epoch time: 160.5965728760s, input_fired_time: 0.9951212406, output_fired_time: 0.0000000000, propagate_forward_time: 157.5718231201, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0296325684
validation[          , genome  6296] predictions:    6337/  10000 (63.37%), best:    6337/10000 (63.37%), error:     10800.42383, best error:     10800.42383 on epoch:    20, epoch:   20/50, mu: 0.7832005620, learning_rate: 0.0000846065, weight_decay: 0.0000867927

epoch time: 1897.1887207031s, input_fired_time: 18.9831542969, output_fired_time: 6.5294947624, propagate_forward_time: 751.1587524414, propagate_backward_time: 1108.0128173828, weight_update_time: 0.5783068538, other_time: 12.5045166016
epoch time: 163.6976470947s, input_fired_time: 1.0019180775, output_fired_time: 0.0000000000, propagate_forward_time: 160.5947265625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1009979248
validation[          , genome  6296] predictions:    6337/  10000 (63.37%), best:    6337/10000 (63.37%), error:     10824.74805, best error:     10800.42383 on epoch:    20, epoch:   21/50, mu: 0.7920357585, learning_rate: 0.0000779152, weight_decay: 0.0000801823

epoch time: 1911.9923095703s, input_fired_time: 18.7042598724, output_fired_time: 6.5873651505, propagate_forward_time: 750.1547851562, propagate_backward_time: 1123.9506835938, weight_update_time: 0.5756053329, other_time: 12.5952148438
epoch time: 153.1593322754s, input_fired_time: 0.9316833019, output_fired_time: 0.0000000000, propagate_forward_time: 150.1625213623, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0651245117
validation[          , genome  6296] predictions:    6345/  10000 (63.45%), best:    6337/10000 (63.37%), error:     10815.89648, best error:     10800.42383 on epoch:    20, epoch:   22/50, mu: 0.8004934788, learning_rate: 0.0000717531, weight_decay: 0.0000740754

epoch time: 1924.1516113281s, input_fired_time: 19.1202011108, output_fired_time: 6.6092791557, propagate_forward_time: 767.2557373047, propagate_backward_time: 1118.4063720703, weight_update_time: 0.6026202440, other_time: 12.7600097656
epoch time: 161.6020050049s, input_fired_time: 1.0489182472, output_fired_time: 0.0000000000, propagate_forward_time: 158.3862457275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1668395996
validation[          , genome  6296] predictions:    6264/  10000 (62.64%), best:    6337/10000 (63.37%), error:     10963.21777, best error:     10800.42383 on epoch:    20, epoch:   23/50, mu: 0.8085898757, learning_rate: 0.0000660783, weight_decay: 0.0000684336

epoch time: 2012.4265136719s, input_fired_time: 20.4244155884, output_fired_time: 6.9227743149, propagate_forward_time: 813.6154785156, propagate_backward_time: 1158.0032958984, weight_update_time: 0.6513378620, other_time: 13.4605712891
epoch time: 170.9052276611s, input_fired_time: 1.0925757885, output_fired_time: 0.0000000000, propagate_forward_time: 167.5803833008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2322692871
validation[          , genome  6296] predictions:    6322/  10000 (63.22%), best:    6322/10000 (63.22%), error:     10792.41602, best error:     10792.41602 on epoch:    24, epoch:   24/50, mu: 0.8163403869, learning_rate: 0.0000608523, weight_decay: 0.0000632215

epoch time: 1992.3834228516s, input_fired_time: 20.0238361359, output_fired_time: 6.7652864456, propagate_forward_time: 802.2122192383, propagate_backward_time: 1150.2593994141, weight_update_time: 0.6227367520, other_time: 13.1226806641
epoch time: 148.6272277832s, input_fired_time: 0.9864432216, output_fired_time: 0.0000000000, propagate_forward_time: 145.6514892578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9892883301
validation[          , genome  6296] predictions:    6305/  10000 (63.05%), best:    6305/10000 (63.05%), error:     10737.97266, best error:     10737.97266 on epoch:    25, epoch:   25/50, mu: 0.8237597346, learning_rate: 0.0000560397, weight_decay: 0.0000584064

epoch time: 2030.7667236328s, input_fired_time: 20.5960769653, output_fired_time: 7.0492787361, propagate_forward_time: 812.2005004883, propagate_backward_time: 1177.6090087891, weight_update_time: 0.6090791821, other_time: 13.3118896484
epoch time: 164.7334594727s, input_fired_time: 1.0780158043, output_fired_time: 0.0000000000, propagate_forward_time: 161.5661010742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0893402100
validation[          , genome  6296] predictions:    6334/  10000 (63.34%), best:    6305/10000 (63.05%), error:     10774.86426, best error:     10737.97266 on epoch:    25, epoch:   26/50, mu: 0.8308621049, learning_rate: 0.0000516076, weight_decay: 0.0000539580

epoch time: 1964.0028076172s, input_fired_time: 19.7941570282, output_fired_time: 6.7561373711, propagate_forward_time: 783.0817871094, propagate_backward_time: 1141.6201171875, weight_update_time: 0.6209490299, other_time: 12.7506103516
epoch time: 165.1252899170s, input_fired_time: 1.0689252615, output_fired_time: 0.0000000000, propagate_forward_time: 161.8568420410, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1995239258
validation[          , genome  6296] predictions:    6457/  10000 (64.57%), best:    6457/10000 (64.57%), error:     10609.49219, best error:     10609.49219 on epoch:    27, epoch:   27/50, mu: 0.8376610279, learning_rate: 0.0000475261, weight_decay: 0.0000498484

epoch time: 1953.7307128906s, input_fired_time: 19.5780448914, output_fired_time: 6.6181797981, propagate_forward_time: 782.6459960938, propagate_backward_time: 1132.1437988281, weight_update_time: 0.5911117792, other_time: 12.7447509766
epoch time: 163.3936920166s, input_fired_time: 1.0138515234, output_fired_time: 0.0000000000, propagate_forward_time: 160.3480987549, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0317382812
validation[          , genome  6296] predictions:    6501/  10000 (65.01%), best:    6501/10000 (65.01%), error:     10375.97656, best error:     10375.97656 on epoch:    28, epoch:   28/50, mu: 0.8441694975, learning_rate: 0.0000437674, weight_decay: 0.0000460519

epoch time: 1935.2779541016s, input_fired_time: 19.4139060974, output_fired_time: 6.5632410049, propagate_forward_time: 773.5515136719, propagate_backward_time: 1123.3831787109, weight_update_time: 0.5792382360, other_time: 12.3660888672
epoch time: 159.0447235107s, input_fired_time: 0.9721211791, output_fired_time: 0.0000000000, propagate_forward_time: 156.0627593994, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0098419189
validation[          , genome  6296] predictions:    6433/  10000 (64.33%), best:    6501/10000 (65.01%), error:     10430.67676, best error:     10375.97656 on epoch:    28, epoch:   29/50, mu: 0.8503999114, learning_rate: 0.0000403059, weight_decay: 0.0000425444

epoch time: 1943.3666992188s, input_fired_time: 19.4158782959, output_fired_time: 6.6320514679, propagate_forward_time: 776.2099609375, propagate_backward_time: 1128.4797363281, weight_update_time: 0.5972166061, other_time: 12.6290283203
epoch time: 150.5548706055s, input_fired_time: 0.9894386530, output_fired_time: 0.0000000000, propagate_forward_time: 147.5478973389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0175323486
validation[          , genome  6296] predictions:    6372/  10000 (63.72%), best:    6501/10000 (65.01%), error:     10627.19238, best error:     10375.97656 on epoch:    28, epoch:   30/50, mu: 0.8563641310, learning_rate: 0.0000371182, weight_decay: 0.0000393041

epoch time: 1951.0797119141s, input_fired_time: 19.4638690948, output_fired_time: 6.6125159264, propagate_forward_time: 779.6444702148, propagate_backward_time: 1132.6425781250, weight_update_time: 0.5929379463, other_time: 12.7163085938
epoch time: 156.5122375488s, input_fired_time: 1.0418257713, output_fired_time: 0.0000000000, propagate_forward_time: 153.4291534424, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0412597656
validation[          , genome  6296] predictions:    6380/  10000 (63.80%), best:    6501/10000 (65.01%), error:     10616.60742, best error:     10375.97656 on epoch:    28, epoch:   31/50, mu: 0.8620735407, learning_rate: 0.0000341827, weight_decay: 0.0000363106

epoch time: 1929.5833740234s, input_fired_time: 19.3616313934, output_fired_time: 6.5847468376, propagate_forward_time: 765.9423828125, propagate_backward_time: 1125.3319091797, weight_update_time: 0.5789623260, other_time: 12.3627929688
epoch time: 155.5613250732s, input_fired_time: 0.9694487453, output_fired_time: 0.0000000000, propagate_forward_time: 152.7153778076, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8764953613
validation[          , genome  6296] predictions:    6463/  10000 (64.63%), best:    6501/10000 (65.01%), error:     10459.90820, best error:     10375.97656 on epoch:    28, epoch:   32/50, mu: 0.8675390482, learning_rate: 0.0000314792, weight_decay: 0.0000335451

epoch time: 1936.7391357422s, input_fired_time: 19.0649566650, output_fired_time: 6.5569038391, propagate_forward_time: 766.1162109375, propagate_backward_time: 1133.2478027344, weight_update_time: 0.5396797657, other_time: 11.7532958984
epoch time: 155.3576507568s, input_fired_time: 0.9330983758, output_fired_time: 0.0000000000, propagate_forward_time: 152.5044097900, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9201354980
validation[          , genome  6296] predictions:    6472/  10000 (64.72%), best:    6501/10000 (65.01%), error:     10413.88965, best error:     10375.97656 on epoch:    28, epoch:   33/50, mu: 0.8727710247, learning_rate: 0.0000289896, weight_decay: 0.0000309902

epoch time: 1947.9506835938s, input_fired_time: 18.7575454712, output_fired_time: 6.6308293343, propagate_forward_time: 760.1602783203, propagate_backward_time: 1150.8043212891, weight_update_time: 0.5304226279, other_time: 11.5976562500
epoch time: 158.8680725098s, input_fired_time: 0.9613658190, output_fired_time: 0.0000000000, propagate_forward_time: 156.0210113525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8856964111
validation[          , genome  6296] predictions:    6480/  10000 (64.80%), best:    6480/10000 (64.80%), error:     10325.35352, best error:     10325.35352 on epoch:    34, epoch:   34/50, mu: 0.8777794838, learning_rate: 0.0000266969, weight_decay: 0.0000286299

epoch time: 1930.6014404297s, input_fired_time: 18.5919437408, output_fired_time: 6.5722351074, propagate_forward_time: 752.9573974609, propagate_backward_time: 1141.3911132812, weight_update_time: 0.5064982772, other_time: 11.0887451172
epoch time: 153.8809204102s, input_fired_time: 0.9163614511, output_fired_time: 0.0000000000, propagate_forward_time: 151.1989135742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7656402588
validation[          , genome  6296] predictions:    6515/  10000 (65.15%), best:    6515/10000 (65.15%), error:     10277.49902, best error:     10277.49902 on epoch:    35, epoch:   35/50, mu: 0.8825739622, learning_rate: 0.0000245855, weight_decay: 0.0000264494

epoch time: 1930.1008300781s, input_fired_time: 18.4763984680, output_fired_time: 6.6406598091, propagate_forward_time: 748.4400634766, propagate_backward_time: 1145.5959472656, weight_update_time: 0.5038177371, other_time: 10.9477539062
epoch time: 153.2317352295s, input_fired_time: 0.9675081968, output_fired_time: 0.0000000000, propagate_forward_time: 150.4171600342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8470611572
validation[          , genome  6296] predictions:    6497/  10000 (64.97%), best:    6515/10000 (65.15%), error:     10332.14941, best error:     10277.49902 on epoch:    35, epoch:   36/50, mu: 0.8871635795, learning_rate: 0.0000226411, weight_decay: 0.0000244349

epoch time: 1943.6458740234s, input_fired_time: 18.4745368958, output_fired_time: 6.6065940857, propagate_forward_time: 756.8264160156, propagate_backward_time: 1150.7192382812, weight_update_time: 0.5003216267, other_time: 11.0191650391
epoch time: 152.5753784180s, input_fired_time: 0.9461426139, output_fired_time: 0.0000000000, propagate_forward_time: 149.8749694824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7542724609
validation[          , genome  6296] predictions:    6523/  10000 (65.23%), best:    6523/10000 (65.23%), error:     10273.42090, best error:     10273.42090 on epoch:    37, epoch:   37/50, mu: 0.8915570974, learning_rate: 0.0000208505, weight_decay: 0.0000225739

epoch time: 1917.2733154297s, input_fired_time: 18.9353122711, output_fired_time: 6.5518074036, propagate_forward_time: 754.3481445312, propagate_backward_time: 1125.7868652344, weight_update_time: 0.5350834131, other_time: 11.6512451172
epoch time: 156.5706024170s, input_fired_time: 1.0176618099, output_fired_time: 0.0000000000, propagate_forward_time: 153.6265716553, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9263763428
validation[          , genome  6296] predictions:    6485/  10000 (64.85%), best:    6523/10000 (65.23%), error:     10360.29199, best error:     10273.42090 on epoch:    37, epoch:   38/50, mu: 0.8957629204, learning_rate: 0.0000192015, weight_decay: 0.0000208546

epoch time: 1937.7135009766s, input_fired_time: 18.7414073944, output_fired_time: 6.6126136780, propagate_forward_time: 759.1784667969, propagate_backward_time: 1141.3819580078, weight_update_time: 0.5370831490, other_time: 11.7989501953
epoch time: 159.9371643066s, input_fired_time: 0.9828353524, output_fired_time: 0.0000000000, propagate_forward_time: 156.9697723389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9845581055
validation[          , genome  6296] predictions:    6504/  10000 (65.04%), best:    6523/10000 (65.23%), error:     10296.46582, best error:     10273.42090 on epoch:    37, epoch:   39/50, mu: 0.8997890949, learning_rate: 0.0000176829, weight_decay: 0.0000192663

epoch time: 1953.7701416016s, input_fired_time: 18.6131782532, output_fired_time: 6.6543273926, propagate_forward_time: 759.2474365234, propagate_backward_time: 1157.6120605469, weight_update_time: 0.5233930945, other_time: 11.6431884766
epoch time: 158.0404663086s, input_fired_time: 0.9755548239, output_fired_time: 0.0000000000, propagate_forward_time: 155.1203002930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9446105957
validation[          , genome  6296] predictions:    6504/  10000 (65.04%), best:    6504/10000 (65.04%), error:     10230.13672, best error:     10230.13672 on epoch:    40, epoch:   40/50, mu: 0.9036432505, learning_rate: 0.0000162844, weight_decay: 0.0000177989

epoch time: 1945.2312011719s, input_fired_time: 18.7009696960, output_fired_time: 6.6918072701, propagate_forward_time: 758.6278076172, propagate_backward_time: 1149.4248046875, weight_update_time: 0.5356542468, other_time: 11.7858886719
epoch time: 155.5200805664s, input_fired_time: 0.9753297567, output_fired_time: 0.0000000000, propagate_forward_time: 152.6505279541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8942260742
validation[          , genome  6296] predictions:    6499/  10000 (64.99%), best:    6504/10000 (65.04%), error:     10299.45605, best error:     10230.13672 on epoch:    40, epoch:   41/50, mu: 0.9073327184, learning_rate: 0.0000149965, weight_decay: 0.0000164433

epoch time: 1956.0799560547s, input_fired_time: 18.7889099121, output_fired_time: 6.6561675072, propagate_forward_time: 764.7065429688, propagate_backward_time: 1154.0803222656, weight_update_time: 0.5353795886, other_time: 11.8480224609
epoch time: 160.2454223633s, input_fired_time: 0.9901422262, output_fired_time: 0.0000000000, propagate_forward_time: 157.2448577881, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0104217529
validation[          , genome  6296] predictions:    6516/  10000 (65.16%), best:    6504/10000 (65.04%), error:     10239.41504, best error:     10230.13672 on epoch:    40, epoch:   42/50, mu: 0.9108645916, learning_rate: 0.0000138104, weight_decay: 0.0000151909

epoch time: 1931.3314208984s, input_fired_time: 18.9930019379, output_fired_time: 6.5609164238, propagate_forward_time: 763.1747436523, propagate_backward_time: 1130.9071044922, weight_update_time: 0.5321350098, other_time: 11.6956787109
epoch time: 151.9445343018s, input_fired_time: 0.9722625017, output_fired_time: 0.0000000000, propagate_forward_time: 149.0872497559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8850250244
validation[          , genome  6296] predictions:    6583/  10000 (65.83%), best:    6583/10000 (65.83%), error:     10206.49805, best error:     10206.49805 on epoch:    43, epoch:   43/50, mu: 0.9142455459, learning_rate: 0.0000127182, weight_decay: 0.0000140340

epoch time: 1928.6368408203s, input_fired_time: 18.7846641541, output_fired_time: 6.5485591888, propagate_forward_time: 755.2431030273, propagate_backward_time: 1136.4727783203, weight_update_time: 0.5275196433, other_time: 11.5877685547
epoch time: 162.7840118408s, input_fired_time: 1.0266616344, output_fired_time: 0.0000000000, propagate_forward_time: 159.7069091797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0504455566
validation[          , genome  6296] predictions:    6583/  10000 (65.83%), best:    6583/10000 (65.83%), error:     10090.79688, best error:     10090.79688 on epoch:    44, epoch:   44/50, mu: 0.9174820781, learning_rate: 0.0000117124, weight_decay: 0.0000129651

epoch time: 2016.3267822266s, input_fired_time: 20.0996704102, output_fired_time: 6.8584866524, propagate_forward_time: 802.6421508789, propagate_backward_time: 1174.2148437500, weight_update_time: 0.5842398405, other_time: 12.5114746094
epoch time: 160.7762298584s, input_fired_time: 0.9764898419, output_fired_time: 0.0000000000, propagate_forward_time: 157.8399353027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9598083496
validation[          , genome  6296] predictions:    6564/  10000 (65.64%), best:    6583/10000 (65.83%), error:     10111.13281, best error:     10090.79688 on epoch:    44, epoch:   45/50, mu: 0.9205803275, learning_rate: 0.0000107861, weight_decay: 0.0000119776

epoch time: 1933.6899414062s, input_fired_time: 18.7512054443, output_fired_time: 6.5861887932, propagate_forward_time: 756.0213012695, propagate_backward_time: 1140.7430419922, weight_update_time: 0.5296679139, other_time: 11.5882568359
epoch time: 156.0249786377s, input_fired_time: 0.9508162737, output_fired_time: 0.0000000000, propagate_forward_time: 153.1958923340, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8782653809
validation[          , genome  6296] predictions:    6591/  10000 (65.91%), best:    6591/10000 (65.91%), error:     10076.27734, best error:     10076.27734 on epoch:    46, epoch:   46/50, mu: 0.9235461950, learning_rate: 0.0000099330, weight_decay: 0.0000110654

epoch time: 1911.2510986328s, input_fired_time: 18.7428512573, output_fired_time: 6.5272588730, propagate_forward_time: 749.4780273438, propagate_backward_time: 1125.0152587891, weight_update_time: 0.5261065364, other_time: 11.4877929688
epoch time: 152.1791687012s, input_fired_time: 1.0014243126, output_fired_time: 0.0000000000, propagate_forward_time: 149.3181915283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8595581055
validation[          , genome  6296] predictions:    6585/  10000 (65.85%), best:    6591/10000 (65.91%), error:     10105.37207, best error:     10076.27734 on epoch:    46, epoch:   47/50, mu: 0.9263853431, learning_rate: 0.0000091474, weight_decay: 0.0000102226

epoch time: 1896.9655761719s, input_fired_time: 18.6983242035, output_fired_time: 6.5194535255, propagate_forward_time: 742.8756103516, propagate_backward_time: 1117.6124267578, weight_update_time: 0.5209556222, other_time: 11.2597656250
epoch time: 154.1054077148s, input_fired_time: 0.9430025816, output_fired_time: 0.0000000000, propagate_forward_time: 151.3362579346, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8261413574
validation[          , genome  6296] predictions:    6592/  10000 (65.92%), best:    6592/10000 (65.92%), error:     10064.21289, best error:     10064.21289 on epoch:    48, epoch:   48/50, mu: 0.9291031957, learning_rate: 0.0000084240, weight_decay: 0.0000094440

epoch time: 1886.1196289062s, input_fired_time: 18.8277282715, output_fired_time: 6.5004062653, propagate_forward_time: 738.1328125000, propagate_backward_time: 1111.5528564453, weight_update_time: 0.5272531509, other_time: 11.1058349609
epoch time: 143.9749603271s, input_fired_time: 0.9508703351, output_fired_time: 0.0000000000, propagate_forward_time: 141.3347473145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6893463135
validation[          , genome  6296] predictions:    6613/  10000 (66.13%), best:    6613/10000 (66.13%), error:     10031.13086, best error:     10031.13086 on epoch:    49, epoch:   49/50, mu: 0.9317049384, learning_rate: 0.0000077578, weight_decay: 0.0000087248

epoch time: 1889.4135742188s, input_fired_time: 18.7145309448, output_fired_time: 6.5205340385, propagate_forward_time: 736.4433593750, propagate_backward_time: 1116.7753906250, weight_update_time: 0.5296884775, other_time: 10.9598388672
epoch time: 151.3839569092s, input_fired_time: 0.9458881021, output_fired_time: 0.0000000000, propagate_forward_time: 148.6945953369, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7434692383
validation[          , genome  6296] predictions:    6614/  10000 (66.14%), best:    6613/10000 (66.13%), error:     10039.04395, best error:     10031.13086 on epoch:    49, epoch:   50/50, mu: 0.9341955185, learning_rate: 0.0000071442, weight_decay: 0.0000080603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 754.1580810547s, input_fired_time: 4.7082562447, output_fired_time: 0.0000000000, propagate_forward_time: 740.9094238281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5404052734
best training[          , genome  6296] predictions:   35601/  50000 (71.20%), best:    6613/10000 (66.13%), error:     44489.20703, best error:     10031.13086 on epoch:    49, epoch:   51/50, mu: 0.9365797043, learning_rate: 0.0000065792, weight_decay: 0.0000074464
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 152.6842651367s, input_fired_time: 0.9285001755, output_fired_time: 0.0000000000, propagate_forward_time: 150.0795440674, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6762237549
   testing[          , genome  6296] predictions:    6613/  10000 (66.13%), best:    6613/10000 (66.13%), error:     10031.13086, best error:     10031.13086 on epoch:    49, epoch:   51/50, mu: 0.9365797043, learning_rate: 0.0000065792, weight_decay: 0.0000074464
backpropagation finished successfully!
09:10:48 (4979): called boinc_finish(0)

</stderr_txt>
]]>