Nameexact_genome_1542692703_180_6389_1
Workunit4100887
Created20 Nov 2018, 10:42:01 UTC
Sent20 Nov 2018, 11:38:57 UTC
Report deadline15 Jan 2019, 4:44:29 UTC
Received23 Nov 2018, 12:57:11 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time1 days 1 hours 0 min 20 sec
CPU time1 days 1 hours 0 min 20 sec
Validate stateValid
Credit13,928.05
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size561.63 MB
Peak swap size560.32 MB
Peak disk usage10.98 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_180_6389.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_180_6389_1_r921384131_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542692703_180_6389.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.490434
read mu: 0.490434
read mu_delta: 0.933216
read initial_learning_rate: 0.000282395
read learning_rate: 0.000282395
read learning_rate_delta: 0.916828
read initial_weight_decay: 0.000498799
read weight_decay: 0.000498799
read weight_decay_delta: 0.940338
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.103812
read input_dropout_probability: 0.00138131
read hidden_dropout_probability: 0.0906803
read velocity_reset: 1599
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6389
read normal distribution: '1 -0.628138184547424 0.175326183438301'
generator_str: '980710976'
read generator: 980710976
read generated_by_map:
1 add_edge 1
reading 88 nodes.
reading 334 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 872382907
post shuffle 1: 1552158732
epoch time: 1528.29992675781s, input_fired_time: 24.3576068878174, output_fired_time: 11.9102172851563, propagate_forward_time: 604.886901855469, propagate_backward_time: 870.513793945313, weight_update_time: 0.480508148670197, other_time: 16.6312866210938
epoch time: 92.5541610717773s, input_fired_time: 1.5184839963913, output_fired_time: 0, propagate_forward_time: 88.3319931030273, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.70368194580078
validation[          , genome  6389] predictions:    6239/  10000 (62.39%), best:    6239/10000 (62.39%), error:     11050.86133, best error:     11050.86133 on epoch:     0, epoch:    0/50, mu: 0.4904336035, learning_rate: 0.0002823946, weight_decay: 0.0004987985

epoch time: 1588.8366699219s, input_fired_time: 24.6555290222, output_fired_time: 12.3599576950, propagate_forward_time: 629.1484375000, propagate_backward_time: 905.8612670898, weight_update_time: 0.4953378439, other_time: 16.8114624023
epoch time: 92.8536834717s, input_fired_time: 1.5151268244, output_fired_time: 0.0000000000, propagate_forward_time: 88.6200790405, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7184753418
validation[          , genome  6389] predictions:    5992/  10000 (59.92%), best:    6239/10000 (62.39%), error:     11866.09961, best error:     11050.86133 on epoch:     0, epoch:    1/50, mu: 0.5237965584, learning_rate: 0.0002589072, weight_decay: 0.0004690391

epoch time: 1516.0041503906s, input_fired_time: 24.4343166351, output_fired_time: 11.9041242599, propagate_forward_time: 598.7819213867, propagate_backward_time: 863.6994628906, weight_update_time: 0.4936829507, other_time: 17.1842651367
epoch time: 117.3265533447s, input_fired_time: 1.8027588129, output_fired_time: 0.0000000000, propagate_forward_time: 112.5515518188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9722442627
validation[          , genome  6389] predictions:    6009/  10000 (60.09%), best:    6239/10000 (62.39%), error:     11865.72852, best error:     11050.86133 on epoch:     0, epoch:    2/50, mu: 0.5549314022, learning_rate: 0.0002373733, weight_decay: 0.0004410552

epoch time: 1681.9232177734s, input_fired_time: 25.1751728058, output_fired_time: 13.0391292572, propagate_forward_time: 671.2213134766, propagate_backward_time: 955.1585693359, weight_update_time: 0.5213990211, other_time: 17.3289794922
epoch time: 111.9143981934s, input_fired_time: 1.7387338877, output_fired_time: 0.0000000000, propagate_forward_time: 107.1281280518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0475387573
validation[          , genome  6389] predictions:    5937/  10000 (59.37%), best:    6239/10000 (62.39%), error:     11974.22168, best error:     11050.86133 on epoch:     0, epoch:    3/50, mu: 0.5839869380, learning_rate: 0.0002176304, weight_decay: 0.0004147408

epoch time: 1681.8852539063s, input_fired_time: 25.4132366180, output_fired_time: 13.2016830444, propagate_forward_time: 669.6730346680, propagate_backward_time: 955.5342407227, weight_update_time: 0.5368065834, other_time: 18.0631103516
epoch time: 93.0221710205s, input_fired_time: 1.5319192410, output_fired_time: 0.0000000000, propagate_forward_time: 88.5744781494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9157714844
validation[          , genome  6389] predictions:    5676/  10000 (56.76%), best:    6239/10000 (62.39%), error:     12757.16309, best error:     11050.86133 on epoch:     0, epoch:    4/50, mu: 0.6111020446, learning_rate: 0.0001995296, weight_decay: 0.0003899965

epoch time: 1557.7501220703s, input_fired_time: 24.5948295593, output_fired_time: 12.2897319794, propagate_forward_time: 614.0717163086, propagate_backward_time: 889.1997070313, weight_update_time: 0.5045568943, other_time: 17.5941772461
epoch time: 123.8854370117s, input_fired_time: 1.8514996767, output_fired_time: 0.0000000000, propagate_forward_time: 118.9477233887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0862121582
validation[          , genome  6389] predictions:    5876/  10000 (58.76%), best:    6239/10000 (62.39%), error:     12203.77246, best error:     11050.86133 on epoch:     0, epoch:    5/50, mu: 0.6364063025, learning_rate: 0.0001829342, weight_decay: 0.0003667284

epoch time: 1593.0792236328s, input_fired_time: 24.4047164917, output_fired_time: 12.3771800995, propagate_forward_time: 631.3361816406, propagate_backward_time: 908.1388549805, weight_update_time: 0.4981144369, other_time: 16.8223266602
epoch time: 160.1892242432s, input_fired_time: 2.2016980648, output_fired_time: 0.0000000000, propagate_forward_time: 154.6411743164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3463592529
validation[          , genome  6389] predictions:    5861/  10000 (58.61%), best:    6239/10000 (62.39%), error:     12179.41992, best error:     11050.86133 on epoch:     0, epoch:    6/50, mu: 0.6600206494, learning_rate: 0.0001677192, weight_decay: 0.0003448486

epoch time: 1491.3531494141s, input_fired_time: 24.1776351929, output_fired_time: 11.7507524490, propagate_forward_time: 588.5430297852, propagate_backward_time: 849.4328002930, weight_update_time: 0.4898720086, other_time: 17.4489746094
epoch time: 92.6778335571s, input_fired_time: 1.5348668098, output_fired_time: 0.0000000000, propagate_forward_time: 88.2810058594, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8619613647
validation[          , genome  6389] predictions:    5887/  10000 (58.87%), best:    6239/10000 (62.39%), error:     12227.98535, best error:     11050.86133 on epoch:     0, epoch:    7/50, mu: 0.6820579171, learning_rate: 0.0001537696, weight_decay: 0.0003242742

epoch time: 1559.2130126953s, input_fired_time: 24.2881774902, output_fired_time: 12.1617221832, propagate_forward_time: 617.4235839844, propagate_backward_time: 888.3297119141, weight_update_time: 0.4948460162, other_time: 17.0097656250
epoch time: 142.6604309082s, input_fired_time: 1.9581947327, output_fired_time: 0.0000000000, propagate_forward_time: 137.6724700928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0297698975
validation[          , genome  6389] predictions:    5685/  10000 (56.85%), best:    6239/10000 (62.39%), error:     12544.38086, best error:     11050.86133 on epoch:     0, epoch:    8/50, mu: 0.7026234269, learning_rate: 0.0001409802, weight_decay: 0.0003049272

epoch time: 1509.2270507813s, input_fired_time: 23.9141273499, output_fired_time: 11.7327003479, propagate_forward_time: 595.7211914063, propagate_backward_time: 861.2791137695, weight_update_time: 0.4789689779, other_time: 16.5798950195
epoch time: 151.6696472168s, input_fired_time: 2.1156358719, output_fired_time: 0.0000000000, propagate_forward_time: 146.2742309570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2797851563
validation[          , genome  6389] predictions:    5852/  10000 (58.52%), best:    6239/10000 (62.39%), error:     12218.00293, best error:     11050.86133 on epoch:     0, epoch:    9/50, mu: 0.7218155265, learning_rate: 0.0001292545, weight_decay: 0.0002867346

epoch time: 1868.7373046875s, input_fired_time: 26.1300220490, output_fired_time: 14.5079936981, propagate_forward_time: 750.1981201172, propagate_backward_time: 1059.7086181641, weight_update_time: 0.5752110481, other_time: 18.1926269531
epoch time: 143.6040039063s, input_fired_time: 2.0229275227, output_fired_time: 0.0000000000, propagate_forward_time: 138.3601074219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2209625244
validation[          , genome  6389] predictions:    5818/  10000 (58.18%), best:    6239/10000 (62.39%), error:     12301.15234, best error:     11050.86133 on epoch:     0, epoch:   10/50, mu: 0.7397258878, learning_rate: 0.0001185041, weight_decay: 0.0002696274

epoch time: 1722.7541503906s, input_fired_time: 25.4504337311, output_fired_time: 13.4984788895, propagate_forward_time: 687.0275268555, propagate_backward_time: 978.5908813477, weight_update_time: 0.5452399850, other_time: 18.1867675781
epoch time: 97.6811676025s, input_fired_time: 1.5670418739, output_fired_time: 0.0000000000, propagate_forward_time: 93.2002410889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9138870239
validation[          , genome  6389] predictions:    5592/  10000 (55.92%), best:    6239/10000 (62.39%), error:     12554.28418, best error:     11050.86133 on epoch:     0, epoch:   11/50, mu: 0.7564401627, learning_rate: 0.0001086479, weight_decay: 0.0002535408

epoch time: 1559.7509765625s, input_fired_time: 24.5113868713, output_fired_time: 12.2278118134, propagate_forward_time: 616.7481079102, propagate_backward_time: 888.6295166016, weight_update_time: 0.5054826140, other_time: 17.6342163086
epoch time: 166.0323028564s, input_fired_time: 2.2436838150, output_fired_time: 0.0000000000, propagate_forward_time: 160.4152984619, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3733215332
validation[          , genome  6389] predictions:    5886/  10000 (58.86%), best:    6239/10000 (62.39%), error:     12062.38281, best error:     11050.86133 on epoch:     0, epoch:   12/50, mu: 0.7720381618, learning_rate: 0.0000996114, weight_decay: 0.0002384140

epoch time: 1510.0946044922s, input_fired_time: 24.0387935638, output_fired_time: 11.8402624130, propagate_forward_time: 596.3571777344, propagate_backward_time: 860.9347534180, weight_update_time: 0.4824936688, other_time: 16.9236450195
epoch time: 164.1176757813s, input_fired_time: 2.2424659729, output_fired_time: 0.0000000000, propagate_forward_time: 158.5180358887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3571777344
validation[          , genome  6389] predictions:    5882/  10000 (58.82%), best:    6239/10000 (62.39%), error:     12143.04395, best error:     11050.86133 on epoch:     0, epoch:   13/50, mu: 0.7865944505, learning_rate: 0.0000913265, weight_decay: 0.0002241897

epoch time: 1670.4948730469s, input_fired_time: 24.8319740295, output_fired_time: 12.9688892365, propagate_forward_time: 665.1524047852, propagate_backward_time: 950.3598022461, weight_update_time: 0.5174022317, other_time: 17.1817626953
epoch time: 175.2968902588s, input_fired_time: 2.2868349552, output_fired_time: 0.0000000000, propagate_forward_time: 169.7225341797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2875213623
validation[          , genome  6389] predictions:    5756/  10000 (57.56%), best:    6239/10000 (62.39%), error:     12251.68457, best error:     11050.86133 on epoch:     0, epoch:   14/50, mu: 0.8001786470, learning_rate: 0.0000837306, weight_decay: 0.0002108140

epoch time: 1799.9011230469s, input_fired_time: 25.5522785187, output_fired_time: 13.9294395447, propagate_forward_time: 720.8029174805, propagate_backward_time: 1022.0808715820, weight_update_time: 0.5456835628, other_time: 17.5355834961
epoch time: 175.9360809326s, input_fired_time: 2.3046002388, output_fired_time: 0.0000000000, propagate_forward_time: 170.3359069824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2955780029
validation[          , genome  6389] predictions:    5980/  10000 (59.80%), best:    6239/10000 (62.39%), error:     11806.05566, best error:     11050.86133 on epoch:     0, epoch:   15/50, mu: 0.8128556609, learning_rate: 0.0000767666, weight_decay: 0.0001982364

epoch time: 1604.2401123047s, input_fired_time: 24.4220790863, output_fired_time: 12.4151754379, propagate_forward_time: 637.0178222656, propagate_backward_time: 913.4855957031, weight_update_time: 0.4999438524, other_time: 16.8994140625
epoch time: 92.4621429443s, input_fired_time: 1.4708538055, output_fired_time: 0.0000000000, propagate_forward_time: 88.2802429199, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7110443115
validation[          , genome  6389] predictions:    5831/  10000 (58.31%), best:    6239/10000 (62.39%), error:     12008.21582, best error:     11050.86133 on epoch:     0, epoch:   16/50, mu: 0.8246860504, learning_rate: 0.0000703817, weight_decay: 0.0001864092

epoch time: 1759.4190673828s, input_fired_time: 25.5841331482, output_fired_time: 13.7037048340, propagate_forward_time: 703.4301757813, propagate_backward_time: 998.7070922852, weight_update_time: 0.5524523258, other_time: 17.9939575195
epoch time: 107.3217849731s, input_fired_time: 1.6555185318, output_fired_time: 0.0000000000, propagate_forward_time: 102.7643203735, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9019470215
validation[          , genome  6389] predictions:    5913/  10000 (59.13%), best:    6239/10000 (62.39%), error:     11987.10059, best error:     11050.86133 on epoch:     0, epoch:   17/50, mu: 0.8357263207, learning_rate: 0.0000645279, weight_decay: 0.0001752876

epoch time: 1667.7211914063s, input_fired_time: 24.7668609619, output_fired_time: 12.9096660614, propagate_forward_time: 662.1077270508, propagate_backward_time: 950.9129638672, weight_update_time: 0.5135296583, other_time: 17.0239868164
epoch time: 125.3177642822s, input_fired_time: 1.7989366055, output_fired_time: 0.0000000000, propagate_forward_time: 120.5784454346, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9403839111
validation[          , genome  6389] predictions:    5995/  10000 (59.95%), best:    6239/10000 (62.39%), error:     11751.67188, best error:     11050.86133 on epoch:     0, epoch:   18/50, mu: 0.8460292816, learning_rate: 0.0000591610, weight_decay: 0.0001648295

epoch time: 1531.5958251953s, input_fired_time: 24.1216335297, output_fired_time: 11.9566822052, propagate_forward_time: 605.6478881836, propagate_backward_time: 873.0166625977, weight_update_time: 0.4883342385, other_time: 16.8530273438
epoch time: 147.7761840820s, input_fired_time: 2.0628044605, output_fired_time: 0.0000000000, propagate_forward_time: 142.4860076904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2273712158
validation[          , genome  6389] predictions:    6014/  10000 (60.14%), best:    6239/10000 (62.39%), error:     11574.77344, best error:     11050.86133 on epoch:     0, epoch:   19/50, mu: 0.8556441665, learning_rate: 0.0000542404, weight_decay: 0.0001549954

epoch time: 1567.6612548828s, input_fired_time: 24.2717456818, output_fired_time: 12.1951398849, propagate_forward_time: 622.4287109375, propagate_backward_time: 891.9778442383, weight_update_time: 0.4892371297, other_time: 16.7877807617
epoch time: 122.0752563477s, input_fired_time: 1.7525606155, output_fired_time: 0.0000000000, propagate_forward_time: 117.4070739746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9156188965
validation[          , genome  6389] predictions:    5903/  10000 (59.03%), best:    6239/10000 (62.39%), error:     11959.81934, best error:     11050.86133 on epoch:     0, epoch:   20/50, mu: 0.8646169305, learning_rate: 0.0000497291, weight_decay: 0.0001457481

epoch time: 1630.2132568359s, input_fired_time: 24.5940933228, output_fired_time: 12.5730638504, propagate_forward_time: 648.6604614258, propagate_backward_time: 927.5332031250, weight_update_time: 0.5059993863, other_time: 16.8523559570
epoch time: 92.2683334351s, input_fired_time: 1.4534503222, output_fired_time: 0.0000000000, propagate_forward_time: 88.1851654053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6297149658
validation[          , genome  6389] predictions:    6069/  10000 (60.69%), best:    6239/10000 (62.39%), error:     11716.41016, best error:     11050.86133 on epoch:     0, epoch:   21/50, mu: 0.8729904890, learning_rate: 0.0000455930, weight_decay: 0.0001370524

epoch time: 1553.5677490234s, input_fired_time: 24.1375579834, output_fired_time: 12.0338144302, propagate_forward_time: 616.8040161133, propagate_backward_time: 884.1058959961, weight_update_time: 0.4852803349, other_time: 16.4864501953
epoch time: 117.4635620117s, input_fired_time: 1.7093323469, output_fired_time: 0.0000000000, propagate_forward_time: 112.9343490601, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8198776245
validation[          , genome  6389] predictions:    5944/  10000 (59.44%), best:    6239/10000 (62.39%), error:     11939.47363, best error:     11050.86133 on epoch:     0, epoch:   22/50, mu: 0.8808048368, learning_rate: 0.0000418009, weight_decay: 0.0001288756

epoch time: 1572.8637695313s, input_fired_time: 24.2773418427, output_fired_time: 12.2238769531, propagate_forward_time: 623.4237670898, propagate_backward_time: 896.2819213867, weight_update_time: 0.4914542437, other_time: 16.6568603516
epoch time: 130.8114013672s, input_fired_time: 1.9260334969, output_fired_time: 0.0000000000, propagate_forward_time: 125.7378845215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1474761963
validation[          , genome  6389] predictions:    5812/  10000 (58.12%), best:    6239/10000 (62.39%), error:     11975.21191, best error:     11050.86133 on epoch:     0, epoch:   23/50, mu: 0.8880972862, learning_rate: 0.0000383243, weight_decay: 0.0001211866

epoch time: 1562.7344970703s, input_fired_time: 24.6051235199, output_fired_time: 12.2372064590, propagate_forward_time: 618.9503173828, propagate_backward_time: 889.2955932617, weight_update_time: 0.5086935163, other_time: 17.6463012695
epoch time: 161.3703460693s, input_fired_time: 2.1559243202, output_fired_time: 0.0000000000, propagate_forward_time: 156.0413818359, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1730346680
validation[          , genome  6389] predictions:    5812/  10000 (58.12%), best:    6239/10000 (62.39%), error:     12252.52832, best error:     11050.86133 on epoch:     0, epoch:   24/50, mu: 0.8949027061, learning_rate: 0.0000351367, weight_decay: 0.0001139563

epoch time: 1667.9295654297s, input_fired_time: 24.9610061646, output_fired_time: 12.9903116226, propagate_forward_time: 666.0068359375, propagate_backward_time: 946.7048950195, weight_update_time: 0.5214836001, other_time: 17.2664184570
epoch time: 173.7257843018s, input_fired_time: 2.3389313221, output_fired_time: 0.0000000000, propagate_forward_time: 167.9584503174, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4284057617
validation[          , genome  6389] predictions:    6079/  10000 (60.79%), best:    6239/10000 (62.39%), error:     11442.31543, best error:     11050.86133 on epoch:     0, epoch:   25/50, mu: 0.9012536407, learning_rate: 0.0000322143, weight_decay: 0.0001071574

epoch time: 1522.0433349609s, input_fired_time: 24.1406097412, output_fired_time: 11.8471632004, propagate_forward_time: 600.8108520508, propagate_backward_time: 868.2997436523, weight_update_time: 0.4898328185, other_time: 16.9449462891
epoch time: 175.0225677490s, input_fired_time: 2.3039162159, output_fired_time: 0.0000000000, propagate_forward_time: 169.4385375977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2801208496
validation[          , genome  6389] predictions:    5971/  10000 (59.71%), best:    6239/10000 (62.39%), error:     11698.87695, best error:     11050.86133 on epoch:     0, epoch:   26/50, mu: 0.9071804881, learning_rate: 0.0000295350, weight_decay: 0.0001007642

epoch time: 1738.8282470703s, input_fired_time: 25.6777954102, output_fired_time: 13.6115093231, propagate_forward_time: 694.8105468750, propagate_backward_time: 986.4972534180, weight_update_time: 0.5498193502, other_time: 18.2311401367
epoch time: 126.8997497559s, input_fired_time: 1.8809354305, output_fired_time: 0.0000000000, propagate_forward_time: 121.8988876343, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1199264526
validation[          , genome  6389] predictions:    6107/  10000 (61.07%), best:    6239/10000 (62.39%), error:     11370.24707, best error:     11050.86133 on epoch:     0, epoch:   27/50, mu: 0.9127115011, learning_rate: 0.0000270785, weight_decay: 0.0000947524

epoch time: 1723.5498046875s, input_fired_time: 25.5963764191, output_fired_time: 13.4929094315, propagate_forward_time: 687.0270385742, propagate_backward_time: 979.2313842773, weight_update_time: 0.5488188863, other_time: 18.2020263672
epoch time: 176.4262695313s, input_fired_time: 2.3763477802, output_fired_time: 0.0000000000, propagate_forward_time: 170.6037902832, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4461364746
validation[          , genome  6389] predictions:    5981/  10000 (59.81%), best:    6239/10000 (62.39%), error:     11589.83105, best error:     11050.86133 on epoch:     0, epoch:   28/50, mu: 0.9178731441, learning_rate: 0.0000248263, weight_decay: 0.0000890992

epoch time: 1765.3643798828s, input_fired_time: 25.8179340363, output_fired_time: 13.8185901642, propagate_forward_time: 704.4447631836, propagate_backward_time: 1002.9428710938, weight_update_time: 0.5576207042, other_time: 18.3401489258
epoch time: 177.0220642090s, input_fired_time: 2.3794789314, output_fired_time: 0.0000000000, propagate_forward_time: 171.1951141357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4474639893
validation[          , genome  6389] predictions:    6126/  10000 (61.26%), best:    6239/10000 (62.39%), error:     11407.47266, best error:     11050.86133 on epoch:     0, epoch:   29/50, mu: 0.9226900339, learning_rate: 0.0000227615, weight_decay: 0.0000837834

epoch time: 1625.6391601563s, input_fired_time: 24.5994644165, output_fired_time: 12.5646104813, propagate_forward_time: 646.0583496094, propagate_backward_time: 925.6452026367, weight_update_time: 0.5049358010, other_time: 16.7715454102
epoch time: 132.5110473633s, input_fired_time: 1.8814821243, output_fired_time: 0.0000000000, propagate_forward_time: 127.6728973389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9566650391
validation[          , genome  6389] predictions:    6097/  10000 (60.97%), best:    6239/10000 (62.39%), error:     11350.40723, best error:     11050.86133 on epoch:     0, epoch:   30/50, mu: 0.9271852374, learning_rate: 0.0000208683, weight_decay: 0.0000787847

epoch time: 1677.5795898438s, input_fired_time: 25.2223110199, output_fired_time: 13.0674791336, propagate_forward_time: 668.3595581055, propagate_backward_time: 953.0813598633, weight_update_time: 0.5307922959, other_time: 17.8488769531
epoch time: 155.7232055664s, input_fired_time: 2.1207993031, output_fired_time: 0.0000000000, propagate_forward_time: 150.4786682129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1237335205
validation[          , genome  6389] predictions:    6116/  10000 (61.16%), best:    6239/10000 (62.39%), error:     11336.68750, best error:     11050.86133 on epoch:     0, epoch:   31/50, mu: 0.9313802719, learning_rate: 0.0000191327, weight_decay: 0.0000740842

epoch time: 1642.3149414063s, input_fired_time: 24.7178478241, output_fired_time: 12.6836557388, propagate_forward_time: 654.5575561523, propagate_backward_time: 933.5529785156, weight_update_time: 0.5070393682, other_time: 16.8027954102
epoch time: 140.2784271240s, input_fired_time: 1.9740526676, output_fired_time: 0.0000000000, propagate_forward_time: 135.2530364990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0513305664
validation[          , genome  6389] predictions:    6201/  10000 (62.01%), best:    6239/10000 (62.39%), error:     11157.64844, best error:     11050.86133 on epoch:     0, epoch:   32/50, mu: 0.9352951050, learning_rate: 0.0000175414, weight_decay: 0.0000696642

epoch time: 1730.5189208984s, input_fired_time: 25.2274284363, output_fired_time: 13.3903884888, propagate_forward_time: 690.2548217773, propagate_backward_time: 984.4785156250, weight_update_time: 0.5306102633, other_time: 17.1677856445
epoch time: 105.3031997681s, input_fired_time: 1.5995825529, output_fired_time: 0.0000000000, propagate_forward_time: 100.9671401978, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7364807129
validation[          , genome  6389] predictions:    6099/  10000 (60.99%), best:    6239/10000 (62.39%), error:     11452.94336, best error:     11050.86133 on epoch:     0, epoch:   33/50, mu: 0.9389485121, learning_rate: 0.0000160824, weight_decay: 0.0000655078

epoch time: 1595.0292968750s, input_fired_time: 24.4280338287, output_fired_time: 12.3729419708, propagate_forward_time: 632.3918457031, propagate_backward_time: 909.1169433594, weight_update_time: 0.4946803451, other_time: 16.7196044922
epoch time: 160.6880950928s, input_fired_time: 2.1803164482, output_fired_time: 0.0000000000, propagate_forward_time: 155.3063354492, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2014465332
validation[          , genome  6389] predictions:    6119/  10000 (61.19%), best:    6239/10000 (62.39%), error:     11331.31543, best error:     11050.86133 on epoch:     0, epoch:   34/50, mu: 0.9423578978, learning_rate: 0.0000147448, weight_decay: 0.0000615995

epoch time: 1557.8120117188s, input_fired_time: 24.4895248413, output_fired_time: 12.1922807693, propagate_forward_time: 615.3553466797, propagate_backward_time: 888.5328979492, weight_update_time: 0.5021787882, other_time: 17.2420043945
epoch time: 174.4960021973s, input_fired_time: 2.3282990456, output_fired_time: 0.0000000000, propagate_forward_time: 168.8677978516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2999114990
validation[          , genome  6389] predictions:    6129/  10000 (61.29%), best:    6239/10000 (62.39%), error:     11403.16895, best error:     11050.86133 on epoch:     0, epoch:   35/50, mu: 0.9455395937, learning_rate: 0.0000135184, weight_decay: 0.0000579243

epoch time: 1735.4196777344s, input_fired_time: 25.7673854828, output_fired_time: 13.5942411423, propagate_forward_time: 691.0660400391, propagate_backward_time: 986.1760253906, weight_update_time: 0.5457263589, other_time: 18.8160400391
epoch time: 104.1902618408s, input_fired_time: 1.6505594254, output_fired_time: 0.0000000000, propagate_forward_time: 99.6578903198, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8818130493
validation[          , genome  6389] predictions:    6137/  10000 (61.37%), best:    6239/10000 (62.39%), error:     11336.42676, best error:     11050.86133 on epoch:     0, epoch:   36/50, mu: 0.9485087991, learning_rate: 0.0000123941, weight_decay: 0.0000544684

epoch time: 1715.8352050781s, input_fired_time: 25.4431686401, output_fired_time: 13.3291854858, propagate_forward_time: 684.6218872070, propagate_backward_time: 974.6758422852, weight_update_time: 0.5391824841, other_time: 17.7651367188
epoch time: 140.9524078369s, input_fired_time: 2.1377253532, output_fired_time: 0.0000000000, propagate_forward_time: 134.5511932373, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2634887695
validation[          , genome  6389] predictions:    6132/  10000 (61.32%), best:    6239/10000 (62.39%), error:     11337.86230, best error:     11050.86133 on epoch:     0, epoch:   37/50, mu: 0.9512797594, learning_rate: 0.0000113632, weight_decay: 0.0000512187

epoch time: 1549.8261718750s, input_fired_time: 24.5493736267, output_fired_time: 12.1936302185, propagate_forward_time: 610.4473266602, propagate_backward_time: 884.6693115234, weight_update_time: 0.5000553727, other_time: 17.9666137695
epoch time: 161.5536041260s, input_fired_time: 2.1936702728, output_fired_time: 0.0000000000, propagate_forward_time: 156.0632476807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2966918945
validation[          , genome  6389] predictions:    6225/  10000 (62.25%), best:    6239/10000 (62.39%), error:     11164.45313, best error:     11050.86133 on epoch:     0, epoch:   38/50, mu: 0.9538656473, learning_rate: 0.0000104181, weight_decay: 0.0000481629

epoch time: 1665.2902832031s, input_fired_time: 25.0417671204, output_fired_time: 12.9689550400, propagate_forward_time: 660.4685058594, propagate_backward_time: 949.2288818359, weight_update_time: 0.5201311111, other_time: 17.5821533203
epoch time: 149.0078277588s, input_fired_time: 2.0657973289, output_fired_time: 0.0000000000, propagate_forward_time: 143.8155059814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1265258789
validation[          , genome  6389] predictions:    6169/  10000 (61.69%), best:    6239/10000 (62.39%), error:     11227.44727, best error:     11050.86133 on epoch:     0, epoch:   39/50, mu: 0.9562788606, learning_rate: 0.0000095516, weight_decay: 0.0000452894

epoch time: 1597.2181396484s, input_fired_time: 24.7979316711, output_fired_time: 12.5097055435, propagate_forward_time: 630.2653808594, propagate_backward_time: 911.7128906250, weight_update_time: 0.5086569190, other_time: 17.9321289063
epoch time: 147.1846160889s, input_fired_time: 2.1196243763, output_fired_time: 0.0000000000, propagate_forward_time: 141.3664245605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6985626221
validation[          , genome  6389] predictions:    6216/  10000 (62.16%), best:    6239/10000 (62.39%), error:     11140.19531, best error:     11050.86133 on epoch:     0, epoch:   40/50, mu: 0.9585309029, learning_rate: 0.0000087572, weight_decay: 0.0000425873

epoch time: 1534.5561523438s, input_fired_time: 24.4791469574, output_fired_time: 12.0229034424, propagate_forward_time: 605.8236083984, propagate_backward_time: 874.5659179688, weight_update_time: 0.4997737706, other_time: 17.6645507813
epoch time: 104.8559799194s, input_fired_time: 1.6587458849, output_fired_time: 0.0000000000, propagate_forward_time: 100.2562561035, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9409790039
validation[          , genome  6389] predictions:    6156/  10000 (61.56%), best:    6239/10000 (62.39%), error:     11414.36816, best error:     11050.86133 on epoch:     0, epoch:   41/50, mu: 0.9606325030, learning_rate: 0.0000080288, weight_decay: 0.0000400465

epoch time: 1627.1264648438s, input_fired_time: 25.0561180115, output_fired_time: 12.7424345016, propagate_forward_time: 645.0993041992, propagate_backward_time: 926.0889282227, weight_update_time: 0.5269463062, other_time: 18.1396484375
epoch time: 114.0333938599s, input_fired_time: 1.7203108072, output_fired_time: 0.0000000000, propagate_forward_time: 109.2584228516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0546569824
validation[          , genome  6389] predictions:    6189/  10000 (61.89%), best:    6239/10000 (62.39%), error:     11186.20117, best error:     11050.86133 on epoch:     0, epoch:   42/50, mu: 0.9625937939, learning_rate: 0.0000073611, weight_decay: 0.0000376572

epoch time: 1609.6499023438s, input_fired_time: 25.2362022400, output_fired_time: 12.7580471039, propagate_forward_time: 636.1369628906, propagate_backward_time: 916.4133911133, weight_update_time: 0.5259371996, other_time: 19.1052856445
epoch time: 135.5517120361s, input_fired_time: 1.9796427488, output_fired_time: 0.0000000000, propagate_forward_time: 130.4197387695, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1523284912
validation[          , genome  6389] predictions:    6264/  10000 (62.64%), best:    6264/10000 (62.64%), error:     11016.12695, best error:     11016.12695 on epoch:    43, epoch:   43/50, mu: 0.9644240737, learning_rate: 0.0000067488, weight_decay: 0.0000354105

epoch time: 1599.8006591797s, input_fired_time: 24.9920043945, output_fired_time: 12.6063537598, propagate_forward_time: 630.9754638672, propagate_backward_time: 912.4476928711, weight_update_time: 0.5214772224, other_time: 18.7791137695
epoch time: 109.6018524170s, input_fired_time: 1.7487829924, output_fired_time: 0.0000000000, propagate_forward_time: 104.2770843506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5759887695
validation[          , genome  6389] predictions:    6207/  10000 (62.07%), best:    6264/10000 (62.64%), error:     11113.05078, best error:     11016.12695 on epoch:    43, epoch:   44/50, mu: 0.9661321044, learning_rate: 0.0000061875, weight_decay: 0.0000332978

epoch time: 1474.9997558594s, input_fired_time: 24.2146282196, output_fired_time: 11.5986671448, propagate_forward_time: 577.1338500977, propagate_backward_time: 844.0983886719, weight_update_time: 0.4847852290, other_time: 17.9542846680
epoch time: 139.7620544434s, input_fired_time: 1.9820169210, output_fired_time: 0.0000000000, propagate_forward_time: 134.3169250488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4631195068
validation[          , genome  6389] predictions:    6125/  10000 (61.25%), best:    6264/10000 (62.64%), error:     11204.85449, best error:     11016.12695 on epoch:    43, epoch:   45/50, mu: 0.9677261114, learning_rate: 0.0000056729, weight_decay: 0.0000313112

epoch time: 1451.1279296875s, input_fired_time: 23.8159370422, output_fired_time: 11.4491424561, propagate_forward_time: 567.6102294922, propagate_backward_time: 830.6558227539, weight_update_time: 0.4719912112, other_time: 17.5968627930
epoch time: 125.1883010864s, input_fired_time: 1.8393021822, output_fired_time: 0.0000000000, propagate_forward_time: 119.8534088135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4955902100
validation[          , genome  6389] predictions:    6172/  10000 (61.72%), best:    6264/10000 (62.64%), error:     11263.56641, best error:     11016.12695 on epoch:    43, epoch:   46/50, mu: 0.9692136645, learning_rate: 0.0000052011, weight_decay: 0.0000294431

epoch time: 1478.1705322266s, input_fired_time: 23.9898910522, output_fired_time: 11.5897855759, propagate_forward_time: 580.9364013672, propagate_backward_time: 844.4647216797, weight_update_time: 0.4760231972, other_time: 17.1896972656
epoch time: 92.1111373901s, input_fired_time: 1.4858753681, output_fired_time: 0.0000000000, propagate_forward_time: 87.9866943359, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6385650635
validation[          , genome  6389] predictions:    6191/  10000 (61.91%), best:    6264/10000 (62.64%), error:     11248.92578, best error:     11016.12695 on epoch:    43, epoch:   47/50, mu: 0.9706018567, learning_rate: 0.0000047685, weight_decay: 0.0000276865

epoch time: 1522.8171386719s, input_fired_time: 24.2565727234, output_fired_time: 11.8776292801, propagate_forward_time: 599.7844238281, propagate_backward_time: 869.6768188477, weight_update_time: 0.4837903976, other_time: 17.2216186523
epoch time: 157.6538391113s, input_fired_time: 2.2307813168, output_fired_time: 0.0000000000, propagate_forward_time: 151.1360626221, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2870025635
validation[          , genome  6389] predictions:    6144/  10000 (61.44%), best:    6264/10000 (62.64%), error:     11447.97070, best error:     11016.12695 on epoch:    43, epoch:   48/50, mu: 0.9718973637, learning_rate: 0.0000043719, weight_decay: 0.0000260346

epoch time: 1550.8469238281s, input_fired_time: 24.4970703125, output_fired_time: 12.3297615051, propagate_forward_time: 605.7904052734, propagate_backward_time: 889.6359863281, weight_update_time: 0.5024467111, other_time: 18.5937500000
epoch time: 104.8894119263s, input_fired_time: 1.6252732277, output_fired_time: 0.0000000000, propagate_forward_time: 100.1606216431, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1035156250
validation[          , genome  6389] predictions:    6216/  10000 (62.16%), best:    6264/10000 (62.64%), error:     11201.80273, best error:     11016.12695 on epoch:    43, epoch:   49/50, mu: 0.9731063247, learning_rate: 0.0000040082, weight_decay: 0.0000244814

epoch time: 1711.8632812500s, input_fired_time: 25.9260921478, output_fired_time: 13.6529254913, propagate_forward_time: 672.6192626953, propagate_backward_time: 977.8669433594, weight_update_time: 0.5620344877, other_time: 21.7979736328
epoch time: 108.0320510864s, input_fired_time: 1.7471667528, output_fired_time: 0.0000000000, propagate_forward_time: 102.7158660889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5690155029
validation[          , genome  6389] predictions:    6210/  10000 (62.10%), best:    6264/10000 (62.64%), error:     11127.25684, best error:     11016.12695 on epoch:    43, epoch:   50/50, mu: 0.9742345214, learning_rate: 0.0000036749, weight_decay: 0.0000230207

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 686.3211059570s, input_fired_time: 9.7819805145, output_fired_time: 0.0000000000, propagate_forward_time: 660.3015136719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 16.2376098633
best training[          , genome  6389] predictions:   32619/  50000 (65.24%), best:    6264/10000 (62.64%), error:     52679.08594, best error:     11016.12695 on epoch:    43, epoch:   51/50, mu: 0.9752873778, learning_rate: 0.0000033692, weight_decay: 0.0000216473
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 158.2158660889s, input_fired_time: 2.1746664047, output_fired_time: 0.0000000000, propagate_forward_time: 152.8096160889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2315826416
   testing[          , genome  6389] predictions:    6264/  10000 (62.64%), best:    6264/10000 (62.64%), error:     11016.12695, best error:     11016.12695 on epoch:    43, epoch:   51/50, mu: 0.9752873778, learning_rate: 0.0000033692, weight_decay: 0.0000216473
backpropagation finished successfully!
06:54:40 (4068): called boinc_finish(0)

</stderr_txt>
]]>