Nameexact_genome_1542189904_179_5906_3
Workunit4089001
Created5 Jan 2019, 16:50:56 UTC
Sent5 Jan 2019, 16:51:04 UTC
Report deadline26 Feb 2019, 16:07:06 UTC
Received6 Jan 2019, 22:47:36 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID78968
Run time1 days 5 hours 4 min 24 sec
CPU time1 days 4 hours 28 min 41 sec
Validate stateValid
Credit12,992.37
Device peak FLOPS2.20 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size559.95 MB
Peak swap size579.83 MB
Peak disk usage13.03 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542189904_179_5906.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542189904_179_5906_3_r1946544433_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542189904_179_5906.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.514042
read mu: 0.514042
read mu_delta: 0.958811
read initial_learning_rate: 0.000436836
read learning_rate: 0.000436836
read learning_rate_delta: 0.91526
read initial_weight_decay: 0.000405921
read weight_decay: 0.000405921
read weight_decay_delta: 0.932674
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.126734
read input_dropout_probability: 0.00132254
read hidden_dropout_probability: 0.0922898
read velocity_reset: 1940
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5906
read normal distribution: '1 0.546298682689667 0.458099991083145'
generator_str: '238499470'
read generator: 238499470
read generated_by_map:
1 split_edge 1
reading 89 nodes.
reading 441 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1256106988
post shuffle 1: 2039592733
epoch time: 1755.32421875s, input_fired_time: 28.7704067230225, output_fired_time: 11.452977180481, propagate_forward_time: 657.014465332031, propagate_backward_time: 1037.7021484375, weight_update_time: 0.580671548843384, other_time: 20.38427734375
epoch time: 129.955413818359s, input_fired_time: 1.88810300827026, output_fired_time: 0, propagate_forward_time: 124.667892456055, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.3994140625
validation[          , genome  5906] predictions:    5873/  10000 (58.73%), best:    5873/10000 (58.73%), error:     11701.97754, best error:     11701.97754 on epoch:     0, epoch:    0/50, mu: 0.5140424967, learning_rate: 0.0004368363, weight_decay: 0.0004059213

epoch time: 1776.6887207031s, input_fired_time: 28.9801425934, output_fired_time: 11.6043529510, propagate_forward_time: 673.1199340820, propagate_backward_time: 1042.5800781250, weight_update_time: 0.5871084332, other_time: 20.4042968750
epoch time: 140.1399688721s, input_fired_time: 2.0822176933, output_fired_time: 0.0000000000, propagate_forward_time: 134.5827941895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4749603271
validation[          , genome  5906] predictions:    6160/  10000 (61.60%), best:    6160/10000 (61.60%), error:     11092.43359, best error:     11092.43359 on epoch:     1, epoch:    1/50, mu: 0.5336467028, learning_rate: 0.0003998188, weight_decay: 0.0003785922

epoch time: 1792.3360595703s, input_fired_time: 28.7645835876, output_fired_time: 11.7775659561, propagate_forward_time: 674.8909301758, propagate_backward_time: 1055.3786621094, weight_update_time: 0.6218160391, other_time: 21.5244140625
epoch time: 134.5849304199s, input_fired_time: 1.9361654520, output_fired_time: 0.0000000000, propagate_forward_time: 129.2896423340, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3591156006
validation[          , genome  5906] predictions:    5706/  10000 (57.06%), best:    6160/10000 (61.60%), error:     12299.87500, best error:     11092.43359 on epoch:     1, epoch:    2/50, mu: 0.5524434447, learning_rate: 0.0003659381, weight_decay: 0.0003531030

epoch time: 1797.5810546875s, input_fired_time: 29.5221843719, output_fired_time: 11.9519376755, propagate_forward_time: 678.5053710938, propagate_backward_time: 1055.9926757813, weight_update_time: 0.6118428707, other_time: 21.6088867188
epoch time: 141.9739074707s, input_fired_time: 1.9956530333, output_fired_time: 0.0000000000, propagate_forward_time: 136.2881164551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6901397705
validation[          , genome  5906] predictions:    6114/  10000 (61.14%), best:    6160/10000 (61.60%), error:     11482.93262, best error:     11092.43359 on epoch:     1, epoch:    3/50, mu: 0.5704659224, learning_rate: 0.0003349285, weight_decay: 0.0003293299

epoch time: 1815.8582763672s, input_fired_time: 29.2033386230, output_fired_time: 11.7982034683, propagate_forward_time: 682.4319458008, propagate_backward_time: 1071.1975097656, weight_update_time: 0.6217689514, other_time: 21.2272949219
epoch time: 153.3491210938s, input_fired_time: 2.2031550407, output_fired_time: 0.0000000000, propagate_forward_time: 147.0257110596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1202545166
validation[          , genome  5906] predictions:    6106/  10000 (61.06%), best:    6160/10000 (61.60%), error:     11181.55469, best error:     11092.43359 on epoch:     1, epoch:    4/50, mu: 0.5877460837, learning_rate: 0.0003065466, weight_decay: 0.0003071574

epoch time: 1826.6497802734s, input_fired_time: 29.1950435638, output_fired_time: 11.8493509293, propagate_forward_time: 687.3091430664, propagate_backward_time: 1076.9415283203, weight_update_time: 0.6564835906, other_time: 21.3546142578
epoch time: 136.5993499756s, input_fired_time: 1.9197453260, output_fired_time: 0.0000000000, propagate_forward_time: 131.1415100098, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5381011963
validation[          , genome  5906] predictions:    6101/  10000 (61.01%), best:    6160/10000 (61.60%), error:     11434.42480, best error:     11092.43359 on epoch:     1, epoch:    5/50, mu: 0.6043145061, learning_rate: 0.0002805699, weight_decay: 0.0002864777

epoch time: 1739.3798828125s, input_fired_time: 28.2456645966, output_fired_time: 11.2460861206, propagate_forward_time: 652.8039550781, propagate_backward_time: 1026.7407226563, weight_update_time: 0.5884460211, other_time: 20.3435058594
epoch time: 141.3448181152s, input_fired_time: 2.0228135586, output_fired_time: 0.0000000000, propagate_forward_time: 135.7082672119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6137390137
validation[          , genome  5906] predictions:    5815/  10000 (58.15%), best:    6160/10000 (61.60%), error:     11980.31738, best error:     11092.43359 on epoch:     1, epoch:    6/50, mu: 0.6202005148, learning_rate: 0.0002567944, weight_decay: 0.0002671902

epoch time: 1862.7539062500s, input_fired_time: 29.7240524292, output_fired_time: 12.0225315094, propagate_forward_time: 704.4969482422, propagate_backward_time: 1094.6116943359, weight_update_time: 0.6298239827, other_time: 21.8986816406
epoch time: 145.9387359619s, input_fired_time: 2.0856790543, output_fired_time: 0.0000000000, propagate_forward_time: 140.2528533936, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6002044678
validation[          , genome  5906] predictions:    6180/  10000 (61.80%), best:    6160/10000 (61.60%), error:     11211.45117, best error:     11092.43359 on epoch:     1, epoch:    7/50, mu: 0.6354321837, learning_rate: 0.0002350336, weight_decay: 0.0002492013

epoch time: 1763.1234130859s, input_fired_time: 28.5667419434, output_fired_time: 11.5267276764, propagate_forward_time: 662.1657104492, propagate_backward_time: 1040.3151855469, weight_update_time: 0.5931582451, other_time: 20.5490722656
epoch time: 129.9641418457s, input_fired_time: 1.8463468552, output_fired_time: 0.0000000000, propagate_forward_time: 124.7215576172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3962402344
validation[          , genome  5906] predictions:    5877/  10000 (58.77%), best:    6160/10000 (61.60%), error:     11915.79883, best error:     11092.43359 on epoch:     1, epoch:    8/50, mu: 0.6500364542, learning_rate: 0.0002151168, weight_decay: 0.0002324235

epoch time: 1760.6940917969s, input_fired_time: 28.7003669739, output_fired_time: 11.8119468689, propagate_forward_time: 662.7611083984, propagate_backward_time: 1036.8917236328, weight_update_time: 0.5845362544, other_time: 20.5290527344
epoch time: 155.1921691895s, input_fired_time: 2.2396764755, output_fired_time: 0.0000000000, propagate_forward_time: 149.0458068848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9066925049
validation[          , genome  5906] predictions:    6067/  10000 (60.67%), best:    6160/10000 (61.60%), error:     11334.70605, best error:     11092.43359 on epoch:     1, epoch:    9/50, mu: 0.6640391946, learning_rate: 0.0001968878, weight_decay: 0.0002167754

epoch time: 1738.8273925781s, input_fired_time: 28.2454738617, output_fired_time: 11.4424772263, propagate_forward_time: 653.5713500977, propagate_backward_time: 1024.7980957031, weight_update_time: 0.5696027279, other_time: 20.7700195313
epoch time: 138.5944366455s, input_fired_time: 1.9556329250, output_fired_time: 0.0000000000, propagate_forward_time: 133.2373352051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4014739990
validation[          , genome  5906] predictions:    5972/  10000 (59.72%), best:    6160/10000 (61.60%), error:     11805.38574, best error:     11092.43359 on epoch:     1, epoch:   10/50, mu: 0.6774652004, learning_rate: 0.0001802036, weight_decay: 0.0002021807

epoch time: 1800.8278808594s, input_fired_time: 29.1008815765, output_fired_time: 11.7980165482, propagate_forward_time: 676.1945190430, propagate_backward_time: 1062.8320312500, weight_update_time: 0.6223762035, other_time: 20.9025878906
epoch time: 140.2367095947s, input_fired_time: 1.9770630598, output_fired_time: 0.0000000000, propagate_forward_time: 134.6385345459, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6211090088
validation[          , genome  5906] predictions:    6101/  10000 (61.01%), best:    6160/10000 (61.60%), error:     11374.44629, best error:     11092.43359 on epoch:     1, epoch:   11/50, mu: 0.6903381944, learning_rate: 0.0001649331, weight_decay: 0.0001885687

epoch time: 1750.4479980469s, input_fired_time: 28.7279891968, output_fired_time: 11.5405759811, propagate_forward_time: 661.2170410156, propagate_backward_time: 1028.3975830078, weight_update_time: 0.5776689053, other_time: 20.5648193359
epoch time: 139.8343963623s, input_fired_time: 1.9428267479, output_fired_time: 0.0000000000, propagate_forward_time: 134.3342895508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5572814941
validation[          , genome  5906] predictions:    6087/  10000 (60.87%), best:    6160/10000 (61.60%), error:     11231.13574, best error:     11092.43359 on epoch:     1, epoch:   12/50, mu: 0.7026809454, learning_rate: 0.0001509567, weight_decay: 0.0001758730

epoch time: 1793.4449462891s, input_fired_time: 29.1389827728, output_fired_time: 11.8144502640, propagate_forward_time: 676.4337768555, propagate_backward_time: 1055.5483398438, weight_update_time: 0.6148276329, other_time: 20.5092773438
epoch time: 126.6694793701s, input_fired_time: 1.8436255455, output_fired_time: 0.0000000000, propagate_forward_time: 121.6036758423, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2221755981
validation[          , genome  5906] predictions:    6244/  10000 (62.44%), best:    6244/10000 (62.44%), error:     10982.92676, best error:     10982.92676 on epoch:    13, epoch:   13/50, mu: 0.7145153284, learning_rate: 0.0001381646, weight_decay: 0.0001640322

epoch time: 1770.0100097656s, input_fired_time: 28.7084808350, output_fired_time: 11.4511308670, propagate_forward_time: 663.0265502930, propagate_backward_time: 1045.9542236328, weight_update_time: 0.6087539792, other_time: 20.8695068359
epoch time: 149.0514678955s, input_fired_time: 2.0720267296, output_fired_time: 0.0000000000, propagate_forward_time: 143.3142547607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6651916504
validation[          , genome  5906] predictions:    6123/  10000 (61.23%), best:    6244/10000 (62.44%), error:     11172.28027, best error:     10982.92676 on epoch:    13, epoch:   14/50, mu: 0.7258622646, learning_rate: 0.0001264565, weight_decay: 0.0001529885

epoch time: 1882.5010986328s, input_fired_time: 29.8037014008, output_fired_time: 12.2528238297, propagate_forward_time: 707.2486572266, propagate_backward_time: 1111.4801025391, weight_update_time: 0.6125322580, other_time: 21.7158203125
epoch time: 154.6631774902s, input_fired_time: 2.1730031967, output_fired_time: 0.0000000000, propagate_forward_time: 148.4329986572, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0571746826
validation[          , genome  5906] predictions:    6263/  10000 (62.63%), best:    6263/10000 (62.63%), error:     10938.13770, best error:     10938.13770 on epoch:    15, epoch:   15/50, mu: 0.7367418408, learning_rate: 0.0001157406, weight_decay: 0.0001426884

epoch time: 1799.0393066406s, input_fired_time: 29.1778411865, output_fired_time: 11.7055387497, propagate_forward_time: 674.4887084961, propagate_backward_time: 1062.6295166016, weight_update_time: 0.5953460336, other_time: 21.0377197266
epoch time: 130.5241241455s, input_fired_time: 1.8589504957, output_fired_time: 0.0000000000, propagate_forward_time: 125.2905654907, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3746109009
validation[          , genome  5906] predictions:    6113/  10000 (61.13%), best:    6263/10000 (62.63%), error:     11208.25391, best error:     10938.13770 on epoch:    15, epoch:   16/50, mu: 0.7471733093, learning_rate: 0.0001059327, weight_decay: 0.0001330817

epoch time: 1904.0772705078s, input_fired_time: 30.2104206085, output_fired_time: 12.5026521683, propagate_forward_time: 723.3529052734, propagate_backward_time: 1116.0572509766, weight_update_time: 0.6625270247, other_time: 21.9539794922
epoch time: 172.5925598145s, input_fired_time: 2.4252791405, output_fired_time: 0.0000000000, propagate_forward_time: 165.6715393066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4957427979
validation[          , genome  5906] predictions:    5984/  10000 (59.84%), best:    6263/10000 (62.63%), error:     11679.77051, best error:     10938.13770 on epoch:    15, epoch:   17/50, mu: 0.7571750879, learning_rate: 0.0000969560, weight_decay: 0.0001241218

epoch time: 2028.9154052734s, input_fired_time: 31.4635505676, output_fired_time: 13.3370361328, propagate_forward_time: 765.2697143555, propagate_backward_time: 1194.8443603516, weight_update_time: 0.6729457974, other_time: 24.0008544922
epoch time: 157.4417724609s, input_fired_time: 2.1729292870, output_fired_time: 0.0000000000, propagate_forward_time: 151.3874816895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8813629150
validation[          , genome  5906] predictions:    6234/  10000 (62.34%), best:    6234/10000 (62.34%), error:     10841.48633, best error:     10841.48633 on epoch:    18, epoch:   18/50, mu: 0.7667648792, learning_rate: 0.0000887399, weight_decay: 0.0001157652

epoch time: 2005.3673095703s, input_fired_time: 31.8545398712, output_fired_time: 13.1524257660, propagate_forward_time: 757.0608520508, propagate_backward_time: 1179.6903076172, weight_update_time: 0.6502390504, other_time: 23.6092529297
epoch time: 162.2646331787s, input_fired_time: 2.2682449818, output_fired_time: 0.0000000000, propagate_forward_time: 155.9603729248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0360107422
validation[          , genome  5906] predictions:    6164/  10000 (61.64%), best:    6234/10000 (62.34%), error:     11077.58301, best error:     10841.48633 on epoch:    18, epoch:   19/50, mu: 0.7759597301, learning_rate: 0.0000812201, weight_decay: 0.0001079711

epoch time: 2034.0015869141s, input_fired_time: 31.9346866608, output_fired_time: 13.4853954315, propagate_forward_time: 768.6152343750, propagate_backward_time: 1196.2551269531, weight_update_time: 0.6740806103, other_time: 23.7111816406
epoch time: 168.6464080811s, input_fired_time: 2.3272716999, output_fired_time: 0.0000000000, propagate_forward_time: 161.9403076172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3788299561
validation[          , genome  5906] predictions:    5975/  10000 (59.75%), best:    6234/10000 (62.34%), error:     11607.11523, best error:     10841.48633 on epoch:    18, epoch:   20/50, mu: 0.7847758532, learning_rate: 0.0000743375, weight_decay: 0.0001007019

epoch time: 1924.2084960938s, input_fired_time: 30.3097953796, output_fired_time: 12.5900220871, propagate_forward_time: 729.7239990234, propagate_backward_time: 1128.6677246094, weight_update_time: 0.6681466103, other_time: 22.9169921875
epoch time: 157.9222564697s, input_fired_time: 2.1504650116, output_fired_time: 0.0000000000, propagate_forward_time: 151.8197631836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9520263672
validation[          , genome  5906] predictions:    6285/  10000 (62.85%), best:    6285/10000 (62.85%), error:     10694.61816, best error:     10694.61816 on epoch:    21, epoch:   21/50, mu: 0.7932288051, learning_rate: 0.0000680381, weight_decay: 0.0000939220

epoch time: 1957.5446777344s, input_fired_time: 30.9796466827, output_fired_time: 12.7924356461, propagate_forward_time: 738.2056274414, propagate_backward_time: 1152.5931396484, weight_update_time: 0.6729629040, other_time: 22.9737548828
epoch time: 162.7735290527s, input_fired_time: 2.3269503117, output_fired_time: 0.0000000000, propagate_forward_time: 156.4875030518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9590759277
validation[          , genome  5906] predictions:    5964/  10000 (59.64%), best:    6285/10000 (62.85%), error:     11562.15820, best error:     10694.61816 on epoch:    21, epoch:   22/50, mu: 0.8013336062, learning_rate: 0.0000622726, weight_decay: 0.0000875986

epoch time: 1930.0233154297s, input_fired_time: 30.6853179932, output_fired_time: 12.8379364014, propagate_forward_time: 728.1513671875, propagate_backward_time: 1135.0625000000, weight_update_time: 0.7025545239, other_time: 23.2862548828
epoch time: 140.9366149902s, input_fired_time: 2.0389881134, output_fired_time: 0.0000000000, propagate_forward_time: 135.2678527832, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6297760010
validation[          , genome  5906] predictions:    6328/  10000 (63.28%), best:    6328/10000 (63.28%), error:     10641.41992, best error:     10641.41992 on epoch:    23, epoch:   23/50, mu: 0.8091045618, learning_rate: 0.0000569956, weight_decay: 0.0000817009

epoch time: 1874.2924804688s, input_fired_time: 30.1358489990, output_fired_time: 12.4974555969, propagate_forward_time: 705.4651489258, propagate_backward_time: 1103.7130126953, weight_update_time: 0.6289131045, other_time: 22.4810791016
epoch time: 148.0867156982s, input_fired_time: 2.1615040302, output_fired_time: 0.0000000000, propagate_forward_time: 142.3545532227, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5706634521
validation[          , genome  5906] predictions:    6323/  10000 (63.23%), best:    6328/10000 (63.28%), error:     10711.78711, best error:     10641.41992 on epoch:    23, epoch:   24/50, mu: 0.8165554404, learning_rate: 0.0000521658, weight_decay: 0.0000762003

epoch time: 1953.3322753906s, input_fired_time: 30.6102104187, output_fired_time: 13.0794658661, propagate_forward_time: 735.3536376953, propagate_backward_time: 1151.5465087891, weight_update_time: 0.6483950019, other_time: 22.7424316406
epoch time: 157.4329681396s, input_fired_time: 2.2183120251, output_fired_time: 0.0000000000, propagate_forward_time: 151.2911071777, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9235534668
validation[          , genome  5906] predictions:    6246/  10000 (62.46%), best:    6328/10000 (63.28%), error:     10745.62207, best error:     10641.41992 on epoch:    23, epoch:   25/50, mu: 0.8236994147, learning_rate: 0.0000477453, weight_decay: 0.0000710700

epoch time: 1933.0692138672s, input_fired_time: 30.6093521118, output_fired_time: 12.6552572250, propagate_forward_time: 731.8734130859, propagate_backward_time: 1135.1350097656, weight_update_time: 0.6363866925, other_time: 22.7961425781
epoch time: 149.8749084473s, input_fired_time: 2.1159477234, output_fired_time: 0.0000000000, propagate_forward_time: 143.8771514893, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8818054199
validation[          , genome  5906] predictions:    6347/  10000 (63.47%), best:    6328/10000 (63.28%), error:     10650.66699, best error:     10641.41992 on epoch:    23, epoch:   26/50, mu: 0.8305491805, learning_rate: 0.0000436993, weight_decay: 0.0000662851

epoch time: 1876.8914794922s, input_fired_time: 30.0844936371, output_fired_time: 12.4814691544, propagate_forward_time: 708.0635986328, propagate_backward_time: 1104.3187255859, weight_update_time: 0.6321745515, other_time: 21.9432373047
epoch time: 161.1330261230s, input_fired_time: 2.2143414021, output_fired_time: 0.0000000000, propagate_forward_time: 154.8400726318, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0786132813
validation[          , genome  5906] predictions:    6346/  10000 (63.46%), best:    6328/10000 (63.28%), error:     10648.94629, best error:     10641.41992 on epoch:    23, epoch:   27/50, mu: 0.8371167779, learning_rate: 0.0000399962, weight_decay: 0.0000618224

epoch time: 1974.8338623047s, input_fired_time: 31.3605003357, output_fired_time: 12.9798479080, propagate_forward_time: 750.2587280273, propagate_backward_time: 1156.5069580078, weight_update_time: 0.6708493829, other_time: 23.7279052734
epoch time: 143.4096221924s, input_fired_time: 2.0446331501, output_fired_time: 0.0000000000, propagate_forward_time: 137.7298736572, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6351165771
validation[          , genome  5906] predictions:    6215/  10000 (62.15%), best:    6328/10000 (63.28%), error:     10855.38281, best error:     10641.41992 on epoch:    23, epoch:   28/50, mu: 0.8434138894, learning_rate: 0.0000366070, weight_decay: 0.0000576601

epoch time: 1935.8205566406s, input_fired_time: 30.3068733215, output_fired_time: 12.4454517365, propagate_forward_time: 731.6825561523, propagate_backward_time: 1138.2758789063, weight_update_time: 0.6558291316, other_time: 23.1098632813
epoch time: 147.0225219727s, input_fired_time: 2.1284582615, output_fired_time: 0.0000000000, propagate_forward_time: 140.9667053223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9273529053
validation[          , genome  5906] predictions:    6341/  10000 (63.41%), best:    6341/10000 (63.41%), error:     10632.93848, best error:     10632.93848 on epoch:    29, epoch:   29/50, mu: 0.8494516015, learning_rate: 0.0000335049, weight_decay: 0.0000537781

epoch time: 1896.9382324219s, input_fired_time: 30.4234600067, output_fired_time: 12.2205972672, propagate_forward_time: 713.8840332031, propagate_backward_time: 1118.0560302734, weight_update_time: 0.6442078352, other_time: 22.3541259766
epoch time: 158.5610961914s, input_fired_time: 2.3311448097, output_fired_time: 0.0000000000, propagate_forward_time: 152.1490478516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0809020996
validation[          , genome  5906] predictions:    6276/  10000 (62.76%), best:    6341/10000 (63.41%), error:     10719.99023, best error:     10632.93848 on epoch:    29, epoch:   30/50, mu: 0.8552406430, learning_rate: 0.0000306657, weight_decay: 0.0000501574

epoch time: 1955.1862792969s, input_fired_time: 30.8004550934, output_fired_time: 12.8163270950, propagate_forward_time: 735.0744018555, propagate_backward_time: 1153.3950195313, weight_update_time: 0.6637589931, other_time: 23.1000976563
epoch time: 166.7169036865s, input_fired_time: 2.3474361897, output_fired_time: 0.0000000000, propagate_forward_time: 160.2736206055, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0958404541
validation[          , genome  5906] predictions:    6413/  10000 (64.13%), best:    6413/10000 (64.13%), error:     10399.04004, best error:     10399.04004 on epoch:    31, epoch:   31/50, mu: 0.8607912660, learning_rate: 0.0000280671, weight_decay: 0.0000467805

epoch time: 2079.8618164063s, input_fired_time: 32.2784881592, output_fired_time: 13.6349344254, propagate_forward_time: 786.5621337891, propagate_backward_time: 1223.4992675781, weight_update_time: 0.7120352983, other_time: 23.8870849609
epoch time: 175.0108795166s, input_fired_time: 2.5038893223, output_fired_time: 0.0000000000, propagate_forward_time: 168.1459808350, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3610076904
validation[          , genome  5906] predictions:    6353/  10000 (63.53%), best:    6413/10000 (64.13%), error:     10539.45801, best error:     10399.04004 on epoch:    31, epoch:   32/50, mu: 0.8661132455, learning_rate: 0.0000256887, weight_decay: 0.0000436310

epoch time: 1973.9697265625s, input_fired_time: 31.4071350098, output_fired_time: 12.8595409393, propagate_forward_time: 743.9031982422, propagate_backward_time: 1162.9544677734, weight_update_time: 0.6610605121, other_time: 22.8454589844
epoch time: 152.2171936035s, input_fired_time: 2.2098081112, output_fired_time: 0.0000000000, propagate_forward_time: 146.2195281982, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7878570557
validation[          , genome  5906] predictions:    6232/  10000 (62.32%), best:    6413/10000 (64.13%), error:     10860.02637, best error:     10399.04004 on epoch:    31, epoch:   33/50, mu: 0.8712159991, learning_rate: 0.0000235118, weight_decay: 0.0000406935

epoch time: 1838.3453369141s, input_fired_time: 29.4591140747, output_fired_time: 11.7916316986, propagate_forward_time: 690.5809936523, propagate_backward_time: 1084.8601074219, weight_update_time: 0.6341074705, other_time: 21.6535644531
epoch time: 152.2094726563s, input_fired_time: 2.1967141628, output_fired_time: 0.0000000000, propagate_forward_time: 146.2473754883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7653808594
validation[          , genome  5906] predictions:    6319/  10000 (63.19%), best:    6413/10000 (64.13%), error:     10638.92090, best error:     10399.04004 on epoch:    31, epoch:   34/50, mu: 0.8761085868, learning_rate: 0.0000215194, weight_decay: 0.0000379537

epoch time: 1872.4361572266s, input_fired_time: 29.9385204315, output_fired_time: 12.3420906067, propagate_forward_time: 706.2960205078, propagate_backward_time: 1102.3616943359, weight_update_time: 0.6068280339, other_time: 21.4979248047
epoch time: 133.7708282471s, input_fired_time: 1.9726730585, output_fired_time: 0.0000000000, propagate_forward_time: 128.2544250488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5437316895
validation[          , genome  5906] predictions:    6420/  10000 (64.20%), best:    6420/10000 (64.20%), error:     10359.32227, best error:     10359.32227 on epoch:    35, epoch:   35/50, mu: 0.8807996511, learning_rate: 0.0000196959, weight_decay: 0.0000353984

epoch time: 1761.4167480469s, input_fired_time: 28.6312294006, output_fired_time: 11.6190700531, propagate_forward_time: 659.8768920898, propagate_backward_time: 1041.1188964844, weight_update_time: 0.5670793056, other_time: 20.1706542969
epoch time: 138.6655426025s, input_fired_time: 1.9454686642, output_fired_time: 0.0000000000, propagate_forward_time: 133.1644439697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5556335449
validation[          , genome  5906] predictions:    6399/  10000 (63.99%), best:    6420/10000 (64.20%), error:     10362.54883, best error:     10359.32227 on epoch:    35, epoch:   36/50, mu: 0.8852974772, learning_rate: 0.0000180268, weight_decay: 0.0000330152

epoch time: 1752.8376464844s, input_fired_time: 28.6587409973, output_fired_time: 11.5732889175, propagate_forward_time: 659.8725585938, propagate_backward_time: 1032.4277343750, weight_update_time: 0.5858704448, other_time: 20.3054199219
epoch time: 142.4144592285s, input_fired_time: 2.0252478123, output_fired_time: 0.0000000000, propagate_forward_time: 136.8493499756, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5398559570
validation[          , genome  5906] predictions:    6377/  10000 (63.77%), best:    6420/10000 (64.20%), error:     10414.04590, best error:     10359.32227 on epoch:    35, epoch:   37/50, mu: 0.8896100521, learning_rate: 0.0000164992, weight_decay: 0.0000307924

epoch time: 1840.1538085938s, input_fired_time: 29.8679637909, output_fired_time: 12.1764268875, propagate_forward_time: 692.4259033203, propagate_backward_time: 1084.1440429688, weight_update_time: 0.5920746922, other_time: 21.5395507813
epoch time: 168.2823333740s, input_fired_time: 2.3718640804, output_fired_time: 0.0000000000, propagate_forward_time: 161.6082916260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3021850586
validation[          , genome  5906] predictions:    6429/  10000 (64.29%), best:    6420/10000 (64.20%), error:     10392.61816, best error:     10359.32227 on epoch:    35, epoch:   38/50, mu: 0.8937450051, learning_rate: 0.0000151011, weight_decay: 0.0000287193

epoch time: 2069.7070312500s, input_fired_time: 32.2295112610, output_fired_time: 13.4085626602, propagate_forward_time: 785.2494506836, propagate_backward_time: 1214.9001464844, weight_update_time: 0.6674519181, other_time: 23.9194335938
epoch time: 163.4801788330s, input_fired_time: 2.3023004532, output_fired_time: 0.0000000000, propagate_forward_time: 156.9408416748, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2370300293
validation[          , genome  5906] predictions:    6441/  10000 (64.41%), best:    6420/10000 (64.20%), error:     10372.74121, best error:     10359.32227 on epoch:    35, epoch:   39/50, mu: 0.8977096677, learning_rate: 0.0000138214, weight_decay: 0.0000267857

epoch time: 1967.7508544922s, input_fired_time: 31.0137691498, output_fired_time: 12.7665519714, propagate_forward_time: 742.0522460938, propagate_backward_time: 1159.1270751953, weight_update_time: 0.6456653476, other_time: 22.7911376953
epoch time: 144.4350128174s, input_fired_time: 2.0905089378, output_fired_time: 0.0000000000, propagate_forward_time: 138.4797515869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8647460938
validation[          , genome  5906] predictions:    6378/  10000 (63.78%), best:    6420/10000 (64.20%), error:     10496.04980, best error:     10359.32227 on epoch:    35, epoch:   40/50, mu: 0.9015110135, learning_rate: 0.0000126502, weight_decay: 0.0000249823

epoch time: 1847.9744873047s, input_fired_time: 29.5293731689, output_fired_time: 12.0034751892, propagate_forward_time: 695.2163085938, propagate_backward_time: 1089.6677246094, weight_update_time: 0.6040433645, other_time: 21.5576171875
epoch time: 144.1767730713s, input_fired_time: 2.1172575951, output_fired_time: 0.0000000000, propagate_forward_time: 138.5333251953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5261840820
validation[          , genome  5906] predictions:    6403/  10000 (64.03%), best:    6403/10000 (64.03%), error:     10332.04199, best error:     10332.04199 on epoch:    41, epoch:   41/50, mu: 0.9051557779, learning_rate: 0.0000115782, weight_decay: 0.0000233004

epoch time: 1856.7226562500s, input_fired_time: 30.0714111328, output_fired_time: 12.0832033157, propagate_forward_time: 699.6665649414, propagate_backward_time: 1093.5133056641, weight_update_time: 0.6221742630, other_time: 21.3880615234
epoch time: 145.3739013672s, input_fired_time: 2.1090395451, output_fired_time: 0.0000000000, propagate_forward_time: 139.7709197998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4939422607
validation[          , genome  5906] predictions:    6451/  10000 (64.51%), best:    6451/10000 (64.51%), error:     10282.59180, best error:     10282.59180 on epoch:    42, epoch:   42/50, mu: 0.9086503983, learning_rate: 0.0000105971, weight_decay: 0.0000217316

epoch time: 1820.4318847656s, input_fired_time: 29.2410793304, output_fired_time: 11.8362550735, propagate_forward_time: 683.6724243164, propagate_backward_time: 1074.5703125000, weight_update_time: 0.6313875318, other_time: 21.1118164063
epoch time: 142.3127746582s, input_fired_time: 2.0599653721, output_fired_time: 0.0000000000, propagate_forward_time: 136.6615447998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5912628174
validation[          , genome  5906] predictions:    6493/  10000 (64.93%), best:    6493/10000 (64.93%), error:     10176.83984, best error:     10176.83984 on epoch:    43, epoch:   43/50, mu: 0.9120011330, learning_rate: 0.0000096991, weight_decay: 0.0000202685

epoch time: 1773.4483642578s, input_fired_time: 28.8323974609, output_fired_time: 11.6383876801, propagate_forward_time: 668.7355957031, propagate_backward_time: 1043.1459960938, weight_update_time: 0.6170038581, other_time: 21.0959472656
epoch time: 130.6403503418s, input_fired_time: 1.9284615517, output_fired_time: 0.0000000000, propagate_forward_time: 125.3996887207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3121948242
validation[          , genome  5906] predictions:    6447/  10000 (64.47%), best:    6493/10000 (64.93%), error:     10242.57520, best error:     10176.83984 on epoch:    43, epoch:   44/50, mu: 0.9152138233, learning_rate: 0.0000088772, weight_decay: 0.0000189039

epoch time: 1763.5377197266s, input_fired_time: 28.8046970367, output_fired_time: 11.2877883911, propagate_forward_time: 661.1025390625, propagate_backward_time: 1041.8105468750, weight_update_time: 0.5747215152, other_time: 20.5321044922
epoch time: 139.2960815430s, input_fired_time: 2.0796883106, output_fired_time: 0.0000000000, propagate_forward_time: 133.6865386963, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5298614502
validation[          , genome  5906] predictions:    6505/  10000 (65.05%), best:    6493/10000 (64.93%), error:     10181.36523, best error:     10176.83984 on epoch:    43, epoch:   45/50, mu: 0.9182941914, learning_rate: 0.0000081249, weight_decay: 0.0000176312

epoch time: 1952.0588378906s, input_fired_time: 31.4753379822, output_fired_time: 12.9619512558, propagate_forward_time: 737.6395874023, propagate_backward_time: 1147.2091064453, weight_update_time: 0.6362347603, other_time: 22.7728271484
epoch time: 164.2037048340s, input_fired_time: 2.3763237000, output_fired_time: 0.0000000000, propagate_forward_time: 157.5426025391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2847747803
validation[          , genome  5906] predictions:    6491/  10000 (64.91%), best:    6491/10000 (64.91%), error:     10130.05078, best error:     10130.05078 on epoch:    46, epoch:   46/50, mu: 0.9212476611, learning_rate: 0.0000074364, weight_decay: 0.0000164442

epoch time: 1977.6616210938s, input_fired_time: 31.4844436646, output_fired_time: 13.0771627426, propagate_forward_time: 750.0371704102, propagate_backward_time: 1159.8048095703, weight_update_time: 0.7058156729, other_time: 23.2579345703
epoch time: 148.0121917725s, input_fired_time: 2.1400306225, output_fired_time: 0.0000000000, propagate_forward_time: 142.2482604980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6239013672
validation[          , genome  5906] predictions:    6494/  10000 (64.94%), best:    6494/10000 (64.94%), error:     10120.68066, best error:     10120.68066 on epoch:    47, epoch:   47/50, mu: 0.9240794778, learning_rate: 0.0000068063, weight_decay: 0.0000153370

epoch time: 2015.2176513672s, input_fired_time: 31.2659931183, output_fired_time: 13.0388765335, propagate_forward_time: 762.0692749023, propagate_backward_time: 1185.4832763672, weight_update_time: 0.6616532207, other_time: 23.3602294922
epoch time: 139.7020874023s, input_fired_time: 2.0406827927, output_fired_time: 0.0000000000, propagate_forward_time: 134.0671539307, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5942535400
validation[          , genome  5906] predictions:    6478/  10000 (64.78%), best:    6494/10000 (64.94%), error:     10189.85449, best error:     10120.68066 on epoch:    47, epoch:   48/50, mu: 0.9267946482, learning_rate: 0.0000062295, weight_decay: 0.0000143045

epoch time: 2114.5954589844s, input_fired_time: 32.8771247864, output_fired_time: 13.7036066055, propagate_forward_time: 800.9642944336, propagate_backward_time: 1242.4261474609, weight_update_time: 0.6979523301, other_time: 24.6241455078
epoch time: 168.8204803467s, input_fired_time: 2.4371216297, output_fired_time: 0.0000000000, propagate_forward_time: 161.8807983398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5025634766
validation[          , genome  5906] predictions:    6463/  10000 (64.63%), best:    6494/10000 (64.94%), error:     10276.57422, best error:     10120.68066 on epoch:    47, epoch:   49/50, mu: 0.9293980002, learning_rate: 0.0000057016, weight_decay: 0.0000133414

epoch time: 2122.0749511719s, input_fired_time: 33.3311843872, output_fired_time: 13.9240999222, propagate_forward_time: 806.2302856445, propagate_backward_time: 1244.2121582031, weight_update_time: 0.6854988337, other_time: 24.3771972656
epoch time: 163.2565917969s, input_fired_time: 2.2815978527, output_fired_time: 0.0000000000, propagate_forward_time: 156.9226074219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0523834229
validation[          , genome  5906] predictions:    6531/  10000 (65.31%), best:    6531/10000 (65.31%), error:     10102.97949, best error:     10102.97949 on epoch:    50, epoch:   50/50, mu: 0.9318941236, learning_rate: 0.0000052185, weight_decay: 0.0000124432

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 833.6926879883s, input_fired_time: 11.7470378876, output_fired_time: 0.0000000000, propagate_forward_time: 801.0975952148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 20.8480834961
best training[          , genome  5906] predictions:   34527/  50000 (69.05%), best:    6531/10000 (65.31%), error:     46400.18750, best error:     10102.97949 on epoch:    50, epoch:   51/50, mu: 0.9342874289, learning_rate: 0.0000047762, weight_decay: 0.0000116054
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 172.5018768311s, input_fired_time: 2.4185736179, output_fired_time: 0.0000000000, propagate_forward_time: 165.9883422852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0949554443
   testing[          , genome  5906] predictions:    6531/  10000 (65.31%), best:    6531/10000 (65.31%), error:     10102.97949, best error:     10102.97949 on epoch:    50, epoch:   51/50, mu: 0.9342874289, learning_rate: 0.0000047762, weight_decay: 0.0000116054
backpropagation finished successfully!
13:55:30 (7368): called boinc_finish(0)

</stderr_txt>
]]>