Nameexact_genome_1542069603_179_5774_1
Workunit4085833
Created13 Nov 2018, 11:25:40 UTC
Sent13 Nov 2018, 11:39:06 UTC
Report deadline4 Jan 2019, 0:56:07 UTC
Received18 Nov 2018, 3:40:35 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID83973
Run time1 days 20 hours 45 min 17 sec
CPU time1 days 20 hours 34 min 59 sec
Validate stateValid
Credit12,888.37
Device peak FLOPS2.50 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size572.48 MB
Peak swap size592.79 MB
Peak disk usage13.44 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_179_5774.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_179_5774_1_r892699360_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542069603_179_5774.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.524869
read mu: 0.524869
read mu_delta: 0.957838
read initial_learning_rate: 0.00179599
read learning_rate: 0.00179599
read learning_rate_delta: 0.909111
read initial_weight_decay: 0.000431419
read weight_decay: 0.000431419
read weight_decay_delta: 0.936692
read batch_size: 111
read epsilon: 1e-07
read alpha: 0.114969
read input_dropout_probability: 0.00130862
read hidden_dropout_probability: 0.0991712
read velocity_reset: 1718
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5774
read normal distribution: '1 0 0'
generator_str: '458052682'
read generator: 458052682
read generated_by_map:
1 enable_edge 1
reading 96 nodes.
reading 462 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1910035526
post shuffle 1: 1469976734
epoch time: 3182.03833007813s, input_fired_time: 40.4614143371582, output_fired_time: 20.7982807159424, propagate_forward_time: 1230.36853027344, propagate_backward_time: 1856.24548339844, weight_update_time: 0.765736520290375, other_time: 34.16455078125
epoch time: 257.992126464844s, input_fired_time: 2.9330141544342, output_fired_time: 0, propagate_forward_time: 249.348403930664, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.71070861816406
validation[          , genome  5774] predictions:    5401/  10000 (54.01%), best:    5401/10000 (54.01%), error:     13335.15234, best error:     13335.15234 on epoch:     0, epoch:    0/50, mu: 0.5248690844, learning_rate: 0.0017959862, weight_decay: 0.0004314194

epoch time: 3184.7622070313s, input_fired_time: 39.8283309937, output_fired_time: 23.6269664764, propagate_forward_time: 1234.1096191406, propagate_backward_time: 1849.7166748047, weight_update_time: 0.7835870981, other_time: 37.4805908203
epoch time: 258.8279113770s, input_fired_time: 2.9359896183, output_fired_time: 0.0000000000, propagate_forward_time: 249.1756439209, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7162780762
validation[          , genome  5774] predictions:    5478/  10000 (54.78%), best:    5478/10000 (54.78%), error:     13035.30371, best error:     13035.30371 on epoch:     1, epoch:    1/50, mu: 0.5444801450, learning_rate: 0.0016327506, weight_decay: 0.0004041071

epoch time: 3187.1132812500s, input_fired_time: 41.3092765808, output_fired_time: 27.4778919220, propagate_forward_time: 1229.6092529297, propagate_backward_time: 1851.2585449219, weight_update_time: 0.7673267722, other_time: 37.4583740234
epoch time: 257.9146423340s, input_fired_time: 2.9514460564, output_fired_time: 0.0000000000, propagate_forward_time: 249.2024841309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7607116699
validation[          , genome  5774] predictions:    5626/  10000 (56.26%), best:    5626/10000 (56.26%), error:     12500.26953, best error:     12500.26953 on epoch:     2, epoch:    2/50, mu: 0.5632643700, learning_rate: 0.0014843513, weight_decay: 0.0003785239

epoch time: 3196.3637695313s, input_fired_time: 42.3192672729, output_fired_time: 23.7718181610, propagate_forward_time: 1229.2653808594, propagate_backward_time: 1864.1597900391, weight_update_time: 0.7899786830, other_time: 36.8475341797
epoch time: 256.6376953125s, input_fired_time: 2.9721965790, output_fired_time: 0.0000000000, propagate_forward_time: 247.8901977539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7752990723
validation[          , genome  5774] predictions:    6184/  10000 (61.84%), best:    6184/10000 (61.84%), error:     11042.04785, best error:     11042.04785 on epoch:     3, epoch:    3/50, mu: 0.5812566280, learning_rate: 0.0013494400, weight_decay: 0.0003545603

epoch time: 3191.2097167969s, input_fired_time: 40.1540336609, output_fired_time: 22.7566947937, propagate_forward_time: 1235.8133544922, propagate_backward_time: 1857.8629150391, weight_update_time: 0.8007141948, other_time: 34.6228027344
epoch time: 256.2051086426s, input_fired_time: 3.8995740414, output_fired_time: 0.0000000000, propagate_forward_time: 245.5453796387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7601623535
validation[          , genome  5774] predictions:    5774/  10000 (57.74%), best:    6184/10000 (61.84%), error:     12004.19336, best error:     11042.04785 on epoch:     3, epoch:    4/50, mu: 0.5984902382, learning_rate: 0.0012267906, weight_decay: 0.0003321139

epoch time: 3187.5720214844s, input_fired_time: 42.8819351196, output_fired_time: 23.5733833313, propagate_forward_time: 1219.1999511719, propagate_backward_time: 1866.0052490234, weight_update_time: 0.7866550684, other_time: 35.9114990234
epoch time: 256.1159973145s, input_fired_time: 2.9664256573, output_fired_time: 0.0000000000, propagate_forward_time: 246.4079895020, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7415771484
validation[          , genome  5774] predictions:    6052/  10000 (60.52%), best:    6184/10000 (61.84%), error:     11375.07324, best error:     11042.04785 on epoch:     3, epoch:    5/50, mu: 0.6149972677, learning_rate: 0.0011152887, weight_decay: 0.0003110884

epoch time: 3182.1557617188s, input_fired_time: 40.0984535217, output_fired_time: 20.5564136505, propagate_forward_time: 1229.2403564453, propagate_backward_time: 1857.6647949219, weight_update_time: 1.8532114029, other_time: 34.5958251953
epoch time: 258.4523620605s, input_fired_time: 2.9916863441, output_fired_time: 0.0000000000, propagate_forward_time: 248.6160583496, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8446197510
validation[          , genome  5774] predictions:    5932/  10000 (59.32%), best:    6184/10000 (61.84%), error:     11854.20117, best error:     11042.04785 on epoch:     3, epoch:    6/50, mu: 0.6308082938, learning_rate: 0.0010139211, weight_decay: 0.0002913940

epoch time: 3180.5913085938s, input_fired_time: 41.1496772766, output_fired_time: 23.7754669189, propagate_forward_time: 1230.0782470703, propagate_backward_time: 1852.7622070313, weight_update_time: 0.8065118790, other_time: 32.8258056641
epoch time: 258.5321350098s, input_fired_time: 2.9907751083, output_fired_time: 0.0000000000, propagate_forward_time: 249.7501068115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7912597656
validation[          , genome  5774] predictions:    6048/  10000 (60.48%), best:    6184/10000 (61.84%), error:     11334.18945, best error:     11042.04785 on epoch:     3, epoch:    7/50, mu: 0.6459527016, learning_rate: 0.0009217667, weight_decay: 0.0002729465

epoch time: 3186.4721679688s, input_fired_time: 39.2142372131, output_fired_time: 21.7650604248, propagate_forward_time: 1233.0018310547, propagate_backward_time: 1857.9238281250, weight_update_time: 1.8477045298, other_time: 34.5670166016
epoch time: 256.4103393555s, input_fired_time: 2.9863994122, output_fired_time: 0.0000000000, propagate_forward_time: 247.7909088135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6330261230
validation[          , genome  5774] predictions:    6351/  10000 (63.51%), best:    6351/10000 (63.51%), error:     10591.10449, best error:     10591.10449 on epoch:     8, epoch:    8/50, mu: 0.6604585648, learning_rate: 0.0008379882, weight_decay: 0.0002556668

epoch time: 3198.2519531250s, input_fired_time: 40.3552284241, output_fired_time: 23.6675014496, propagate_forward_time: 1236.8626708984, propagate_backward_time: 1855.5264892578, weight_update_time: 0.8068723083, other_time: 41.8400878906
epoch time: 258.4135742188s, input_fired_time: 4.0846986771, output_fired_time: 0.0000000000, propagate_forward_time: 248.4828491211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8460235596
validation[          , genome  5774] predictions:    6348/  10000 (63.48%), best:    6351/10000 (63.51%), error:     10688.05957, best error:     10591.10449 on epoch:     8, epoch:    9/50, mu: 0.6743528247, learning_rate: 0.0007618243, weight_decay: 0.0002394811

epoch time: 3186.4177246094s, input_fired_time: 41.3313598633, output_fired_time: 21.5914535522, propagate_forward_time: 1228.7020263672, propagate_backward_time: 1860.1955566406, weight_update_time: 0.7928357720, other_time: 34.5972900391
epoch time: 258.2446289063s, input_fired_time: 5.1587839127, output_fired_time: 0.0000000000, propagate_forward_time: 247.3429107666, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7429351807
validation[          , genome  5774] predictions:    6364/  10000 (63.64%), best:    6364/10000 (63.64%), error:     10563.55859, best error:     10563.55859 on epoch:    10, epoch:   10/50, mu: 0.6876612902, learning_rate: 0.0006925827, weight_decay: 0.0002243200

epoch time: 3192.1806640625s, input_fired_time: 40.2814102173, output_fired_time: 23.5557708740, propagate_forward_time: 1229.4910888672, propagate_backward_time: 1860.6915283203, weight_update_time: 0.8033434749, other_time: 38.1608886719
epoch time: 258.7214660645s, input_fired_time: 2.9950153828, output_fired_time: 0.0000000000, propagate_forward_time: 249.9142913818, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8121643066
validation[          , genome  5774] predictions:    6303/  10000 (63.03%), best:    6364/10000 (63.64%), error:     10753.58594, best error:     10563.55859 on epoch:    10, epoch:   11/50, mu: 0.7004086375, learning_rate: 0.0006296345, weight_decay: 0.0002101188

epoch time: 3184.0769042969s, input_fired_time: 43.2816047668, output_fired_time: 24.6051216125, propagate_forward_time: 1221.6578369141, propagate_backward_time: 1859.3182373047, weight_update_time: 0.7903041840, other_time: 35.2141113281
epoch time: 257.4843139648s, input_fired_time: 3.9782607555, output_fired_time: 0.0000000000, propagate_forward_time: 247.7069549561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7991027832
validation[          , genome  5774] predictions:    6318/  10000 (63.18%), best:    6364/10000 (63.64%), error:     10716.43945, best error:     10563.55859 on epoch:    10, epoch:   12/50, mu: 0.7126185298, learning_rate: 0.0005724076, weight_decay: 0.0001968166

epoch time: 3226.3234863281s, input_fired_time: 40.5397605896, output_fired_time: 24.9061489105, propagate_forward_time: 1253.7813720703, propagate_backward_time: 1872.1925048828, weight_update_time: 0.7990086079, other_time: 34.9035644531
epoch time: 259.5892639160s, input_fired_time: 3.0448153019, output_fired_time: 0.0000000000, propagate_forward_time: 249.6124114990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9320220947
validation[          , genome  5774] predictions:    6425/  10000 (64.25%), best:    6425/10000 (64.25%), error:     10387.08496, best error:     10387.08496 on epoch:    13, epoch:   13/50, mu: 0.7243136168, learning_rate: 0.0005203820, weight_decay: 0.0001843565

epoch time: 3196.3090820313s, input_fired_time: 39.5868148804, output_fired_time: 24.7272644043, propagate_forward_time: 1240.8604736328, propagate_backward_time: 1859.3374023438, weight_update_time: 0.8000761867, other_time: 31.7969970703
epoch time: 256.7132263184s, input_fired_time: 3.9906864166, output_fired_time: 0.0000000000, propagate_forward_time: 246.9568786621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7656555176
validation[          , genome  5774] predictions:    6413/  10000 (64.13%), best:    6413/10000 (64.13%), error:     10331.94043, best error:     10331.94043 on epoch:    14, epoch:   14/50, mu: 0.7355155945, learning_rate: 0.0004730850, weight_decay: 0.0001726853

epoch time: 3184.9587402344s, input_fired_time: 40.3380203247, output_fired_time: 20.7545719147, propagate_forward_time: 1226.9237060547, propagate_backward_time: 1863.0706787109, weight_update_time: 0.8038684726, other_time: 33.8715820313
epoch time: 258.0480346680s, input_fired_time: 3.0425648689, output_fired_time: 0.0000000000, propagate_forward_time: 245.2405395508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.7649230957
validation[          , genome  5774] predictions:    6442/  10000 (64.42%), best:    6413/10000 (64.13%), error:     10372.83496, best error:     10331.94043 on epoch:    14, epoch:   15/50, mu: 0.7462452650, learning_rate: 0.0004300867, weight_decay: 0.0001617530

epoch time: 3190.7114257813s, input_fired_time: 45.4520606995, output_fired_time: 22.5825405121, propagate_forward_time: 1229.2919921875, propagate_backward_time: 1860.8706054688, weight_update_time: 0.7949416637, other_time: 32.5141601563
epoch time: 260.8077392578s, input_fired_time: 4.1259074211, output_fired_time: 0.0000000000, propagate_forward_time: 249.9400634766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7417602539
validation[          , genome  5774] predictions:    6374/  10000 (63.74%), best:    6413/10000 (64.13%), error:     10580.90723, best error:     10331.94043 on epoch:    14, epoch:   16/50, mu: 0.7565225959, learning_rate: 0.0003909965, weight_decay: 0.0001515127

epoch time: 3191.0209960938s, input_fired_time: 41.3451194763, output_fired_time: 21.7286567688, propagate_forward_time: 1228.8238525391, propagate_backward_time: 1863.2419433594, weight_update_time: 0.8112840652, other_time: 35.8812255859
epoch time: 257.9564819336s, input_fired_time: 3.0165171623, output_fired_time: 0.0000000000, propagate_forward_time: 249.2210540771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7189178467
validation[          , genome  5774] predictions:    6391/  10000 (63.91%), best:    6413/10000 (64.13%), error:     10580.26465, best error:     10331.94043 on epoch:    14, epoch:   17/50, mu: 0.7663666010, learning_rate: 0.0003554592, weight_decay: 0.0001419208

epoch time: 3191.8151855469s, input_fired_time: 41.3240203857, output_fired_time: 22.7595310211, propagate_forward_time: 1234.0799560547, propagate_backward_time: 1858.0716552734, weight_update_time: 0.8125853539, other_time: 35.5800781250
epoch time: 258.1279602051s, input_fired_time: 4.1192564964, output_fired_time: 0.0000000000, propagate_forward_time: 248.1460113525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8626861572
validation[          , genome  5774] predictions:    6532/  10000 (65.32%), best:    6532/10000 (65.32%), error:     10154.40039, best error:     10154.40039 on epoch:    18, epoch:   18/50, mu: 0.7757955194, learning_rate: 0.0003231518, weight_decay: 0.0001329361

epoch time: 3184.4946289063s, input_fired_time: 39.3667449951, output_fired_time: 22.7552280426, propagate_forward_time: 1227.1748046875, propagate_backward_time: 1858.2548828125, weight_update_time: 0.8195676208, other_time: 36.9431152344
epoch time: 258.5218505859s, input_fired_time: 3.9902744293, output_fired_time: 0.0000000000, propagate_forward_time: 247.9240264893, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6075439453
validation[          , genome  5774] predictions:    6357/  10000 (63.57%), best:    6532/10000 (65.32%), error:     10615.69727, best error:     10154.40039 on epoch:    18, epoch:   19/50, mu: 0.7848269343, learning_rate: 0.0002937808, weight_decay: 0.0001245201

epoch time: 3196.7458496094s, input_fired_time: 40.8434829712, output_fired_time: 20.7241592407, propagate_forward_time: 1239.2214355469, propagate_backward_time: 1860.0997314453, weight_update_time: 0.8082252741, other_time: 35.8570556641
epoch time: 258.1251525879s, input_fired_time: 4.1023068428, output_fired_time: 0.0000000000, propagate_forward_time: 246.1797637939, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8430786133
validation[          , genome  5774] predictions:    6470/  10000 (64.70%), best:    6532/10000 (65.32%), error:     10254.54590, best error:     10154.40039 on epoch:    18, epoch:   20/50, mu: 0.7934775352, learning_rate: 0.0002670794, weight_decay: 0.0001166370

epoch time: 3191.4321289063s, input_fired_time: 41.4439277649, output_fired_time: 23.6142101288, propagate_forward_time: 1235.2873535156, propagate_backward_time: 1855.5664062500, weight_update_time: 0.8065565228, other_time: 35.5202636719
epoch time: 258.7068481445s, input_fired_time: 3.0290408134, output_fired_time: 0.0000000000, propagate_forward_time: 249.9922637939, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6855468750
validation[          , genome  5774] predictions:    6445/  10000 (64.45%), best:    6532/10000 (65.32%), error:     10346.81934, best error:     10154.40039 on epoch:    18, epoch:   21/50, mu: 0.8017634153, learning_rate: 0.0002428048, weight_decay: 0.0001092530

epoch time: 3202.0380859375s, input_fired_time: 44.6028594971, output_fired_time: 24.4762325287, propagate_forward_time: 1232.3792724609, propagate_backward_time: 1866.9666748047, weight_update_time: 0.8085426688, other_time: 33.6130371094
epoch time: 257.3541564941s, input_fired_time: 4.1086821556, output_fired_time: 0.0000000000, propagate_forward_time: 246.5600585938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6854095459
validation[          , genome  5774] predictions:    6255/  10000 (62.55%), best:    6532/10000 (65.32%), error:     10789.22070, best error:     10154.40039 on epoch:    18, epoch:   22/50, mu: 0.8096999526, learning_rate: 0.0002207365, weight_decay: 0.0001023364

epoch time: 3191.7719726563s, input_fired_time: 42.3814544678, output_fired_time: 21.7925167084, propagate_forward_time: 1226.5559082031, propagate_backward_time: 1866.1528320313, weight_update_time: 0.8248681426, other_time: 34.8894042969
epoch time: 260.2555236816s, input_fired_time: 3.0431034565, output_fired_time: 0.0000000000, propagate_forward_time: 251.3843994141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8280334473
validation[          , genome  5774] predictions:    6552/  10000 (65.52%), best:    6552/10000 (65.52%), error:     10136.63672, best error:     10136.63672 on epoch:    23, epoch:   23/50, mu: 0.8173018694, learning_rate: 0.0002006739, weight_decay: 0.0000958577

epoch time: 3414.8444824219s, input_fired_time: 45.7442245483, output_fired_time: 25.5062561035, propagate_forward_time: 1297.4078369141, propagate_backward_time: 1989.8862304688, weight_update_time: 10.9448347092, other_time: 56.2999267578
epoch time: 270.3594970703s, input_fired_time: 3.1026613712, output_fired_time: 0.0000000000, propagate_forward_time: 259.1549377441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1018981934
validation[          , genome  5774] predictions:    6523/  10000 (65.23%), best:    6523/10000 (65.23%), error:     10054.18750, best error:     10054.18750 on epoch:    24, epoch:   24/50, mu: 0.8245832324, learning_rate: 0.0001824349, weight_decay: 0.0000897891

epoch time: 3368.4182128906s, input_fired_time: 39.6304931641, output_fired_time: 26.3445224762, propagate_forward_time: 1269.8062744141, propagate_backward_time: 1983.8254394531, weight_update_time: 0.8485090137, other_time: 48.8114013672
epoch time: 258.7624511719s, input_fired_time: 4.0399975777, output_fired_time: 0.0000000000, propagate_forward_time: 247.9836120605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7388458252
validation[          , genome  5774] predictions:    6309/  10000 (63.09%), best:    6523/10000 (65.23%), error:     10534.65625, best error:     10054.18750 on epoch:    24, epoch:   25/50, mu: 0.8315576315, learning_rate: 0.0001658535, weight_decay: 0.0000841048

epoch time: 3347.5263671875s, input_fired_time: 43.8445739746, output_fired_time: 24.0857715607, propagate_forward_time: 1280.2763671875, propagate_backward_time: 1955.1306152344, weight_update_time: 0.8480896950, other_time: 44.1892089844
epoch time: 259.5009765625s, input_fired_time: 5.0622506142, output_fired_time: 0.0000000000, propagate_forward_time: 248.6426239014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7960968018
validation[          , genome  5774] predictions:    6416/  10000 (64.16%), best:    6523/10000 (65.23%), error:     10283.38574, best error:     10054.18750 on epoch:    24, epoch:   26/50, mu: 0.8382379413, learning_rate: 0.0001507792, weight_decay: 0.0000787803

epoch time: 3422.4055175781s, input_fired_time: 43.9617156982, output_fired_time: 20.9817504883, propagate_forward_time: 1317.6820068359, propagate_backward_time: 2001.9460449219, weight_update_time: 0.8560762405, other_time: 37.8341064453
epoch time: 272.4634094238s, input_fired_time: 3.1599926949, output_fired_time: 0.0000000000, propagate_forward_time: 262.5167236328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7866821289
validation[          , genome  5774] predictions:    6493/  10000 (64.93%), best:    6523/10000 (65.23%), error:     10241.45020, best error:     10054.18750 on epoch:    24, epoch:   27/50, mu: 0.8446366191, learning_rate: 0.0001370751, weight_decay: 0.0000737928

epoch time: 3250.2404785156s, input_fired_time: 42.8090133667, output_fired_time: 24.9547290802, propagate_forward_time: 1240.6406250000, propagate_backward_time: 1908.0899658203, weight_update_time: 0.8617345095, other_time: 33.7459716797
epoch time: 257.7101135254s, input_fired_time: 3.0591104031, output_fired_time: 0.0000000000, propagate_forward_time: 248.9222869873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7287139893
validation[          , genome  5774] predictions:    6563/  10000 (65.63%), best:    6563/10000 (65.63%), error:     10021.76465, best error:     10021.76465 on epoch:    28, epoch:   28/50, mu: 0.8507655263, learning_rate: 0.0001246164, weight_decay: 0.0000691212

epoch time: 3220.7365722656s, input_fired_time: 40.5935173035, output_fired_time: 22.7059078217, propagate_forward_time: 1237.8139648438, propagate_backward_time: 1882.9991455078, weight_update_time: 0.8604800701, other_time: 36.6241455078
epoch time: 260.3464050293s, input_fired_time: 3.0593595505, output_fired_time: 0.0000000000, propagate_forward_time: 249.3155670166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9714813232
validation[          , genome  5774] predictions:    6584/  10000 (65.84%), best:    6584/10000 (65.84%), error:      9952.88672, best error:      9952.88672 on epoch:    29, epoch:   29/50, mu: 0.8566359878, learning_rate: 0.0001132902, weight_decay: 0.0000647453

epoch time: 3208.6269531250s, input_fired_time: 42.6452140808, output_fired_time: 25.9576568604, propagate_forward_time: 1230.3767089844, propagate_backward_time: 1875.2348632813, weight_update_time: 0.8745297790, other_time: 34.4123535156
epoch time: 258.9059448242s, input_fired_time: 4.0392522812, output_fired_time: 0.0000000000, propagate_forward_time: 248.4294281006, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4372711182
validation[          , genome  5774] predictions:    6564/  10000 (65.64%), best:    6584/10000 (65.84%), error:     10033.53125, best error:      9952.88672 on epoch:    29, epoch:   30/50, mu: 0.8622589707, learning_rate: 0.0001029933, weight_decay: 0.0000606464

epoch time: 3214.9541015625s, input_fired_time: 42.6917419434, output_fired_time: 24.9032363892, propagate_forward_time: 1236.5858154297, propagate_backward_time: 1872.8808593750, weight_update_time: 0.8581580520, other_time: 37.8924560547
epoch time: 259.2794189453s, input_fired_time: 4.0336399078, output_fired_time: 0.0000000000, propagate_forward_time: 247.6080017090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.6377716064
validation[          , genome  5774] predictions:    6531/  10000 (65.31%), best:    6584/10000 (65.84%), error:     10057.84473, best error:      9952.88672 on epoch:    29, epoch:   31/50, mu: 0.8676448464, learning_rate: 0.0000936323, weight_decay: 0.0000568070

epoch time: 3312.5522460938s, input_fired_time: 48.4114112854, output_fired_time: 22.4699611664, propagate_forward_time: 1272.1689453125, propagate_backward_time: 1932.4067382813, weight_update_time: 0.8901135325, other_time: 37.0952148438
epoch time: 264.6335449219s, input_fired_time: 4.0994863510, output_fired_time: 0.0000000000, propagate_forward_time: 253.3852691650, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1487884521
validation[          , genome  5774] predictions:    6658/  10000 (66.58%), best:    6658/10000 (66.58%), error:      9793.87500, best error:      9793.87500 on epoch:    32, epoch:   32/50, mu: 0.8728036284, learning_rate: 0.0000851222, weight_decay: 0.0000532106

epoch time: 3241.2075195313s, input_fired_time: 40.8805999756, output_fired_time: 22.0958557129, propagate_forward_time: 1256.6658935547, propagate_backward_time: 1886.2401123047, weight_update_time: 0.8862434030, other_time: 35.3249511719
epoch time: 260.9418334961s, input_fired_time: 3.0884306431, output_fired_time: 0.0000000000, propagate_forward_time: 251.2397613525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6136322021
validation[          , genome  5774] predictions:    6621/  10000 (66.21%), best:    6658/10000 (66.58%), error:      9839.68652, best error:      9793.87500 on epoch:    32, epoch:   33/50, mu: 0.8777449131, learning_rate: 0.0000773855, weight_decay: 0.0000498420

epoch time: 3237.9104003906s, input_fired_time: 43.0525932312, output_fired_time: 20.8965473175, propagate_forward_time: 1246.8609619141, propagate_backward_time: 1891.8011474609, weight_update_time: 0.8910039663, other_time: 35.2993164063
epoch time: 259.8633117676s, input_fired_time: 3.0693747997, output_fired_time: 0.0000000000, propagate_forward_time: 250.1013946533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6925506592
validation[          , genome  5774] predictions:    6620/  10000 (66.20%), best:    6658/10000 (66.58%), error:      9872.20996, best error:      9793.87500 on epoch:    32, epoch:   34/50, mu: 0.8824778795, learning_rate: 0.0000703520, weight_decay: 0.0000466866

epoch time: 3227.6118164063s, input_fired_time: 41.5811042786, output_fired_time: 23.8105335236, propagate_forward_time: 1238.8377685547, propagate_backward_time: 1885.9918212891, weight_update_time: 0.8974058628, other_time: 37.3906250000
epoch time: 260.8213806152s, input_fired_time: 3.0379948616, output_fired_time: 0.0000000000, propagate_forward_time: 252.2923126221, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4910736084
validation[          , genome  5774] predictions:    6680/  10000 (66.80%), best:    6680/10000 (66.80%), error:      9761.87402, best error:      9761.87402 on epoch:    35, epoch:   35/50, mu: 0.8870112896, learning_rate: 0.0000639578, weight_decay: 0.0000437310

epoch time: 3231.7124023438s, input_fired_time: 43.5135078430, output_fired_time: 24.8835086823, propagate_forward_time: 1237.7390136719, propagate_backward_time: 1893.3745117188, weight_update_time: 0.9237427711, other_time: 32.2019042969
epoch time: 259.4696044922s, input_fired_time: 3.0805466175, output_fired_time: 0.0000000000, propagate_forward_time: 249.8147735596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5742950439
validation[          , genome  5774] predictions:    6698/  10000 (66.98%), best:    6698/10000 (66.98%), error:      9731.26758, best error:      9731.26758 on epoch:    36, epoch:   36/50, mu: 0.8913535476, learning_rate: 0.0000581447, weight_decay: 0.0000409624

epoch time: 3225.6320800781s, input_fired_time: 40.6851577759, output_fired_time: 22.8976058960, propagate_forward_time: 1238.8300781250, propagate_backward_time: 1884.4697265625, weight_update_time: 0.9173947573, other_time: 38.7495117188
epoch time: 259.6759948730s, input_fired_time: 3.0941045284, output_fired_time: 0.0000000000, propagate_forward_time: 250.8101806641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7716979980
validation[          , genome  5774] predictions:    6680/  10000 (66.80%), best:    6698/10000 (66.98%), error:      9754.92773, best error:      9731.26758 on epoch:    36, epoch:   37/50, mu: 0.8955127001, learning_rate: 0.0000528600, weight_decay: 0.0000383692

epoch time: 3209.3796386719s, input_fired_time: 44.9012985229, output_fired_time: 22.8658008575, propagate_forward_time: 1229.9476318359, propagate_backward_time: 1874.4921875000, weight_update_time: 2.0252096653, other_time: 37.1727294922
epoch time: 262.2079467773s, input_fired_time: 3.1308624744, output_fired_time: 0.0000000000, propagate_forward_time: 253.0613555908, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0157318115
validation[          , genome  5774] predictions:    6654/  10000 (66.54%), best:    6698/10000 (66.98%), error:      9755.00391, best error:      9731.26758 on epoch:    36, epoch:   38/50, mu: 0.8994964957, learning_rate: 0.0000480556, weight_decay: 0.0000359401

epoch time: 3222.3989257813s, input_fired_time: 41.7649497986, output_fired_time: 21.9467964172, propagate_forward_time: 1245.7421875000, propagate_backward_time: 1878.6359863281, weight_update_time: 0.9634115696, other_time: 34.3090820313
epoch time: 259.7246704102s, input_fired_time: 3.0672893524, output_fired_time: 0.0000000000, propagate_forward_time: 251.0813446045, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5760345459
validation[          , genome  5774] predictions:    6695/  10000 (66.95%), best:    6695/10000 (66.95%), error:      9693.44336, best error:      9693.44336 on epoch:    39, epoch:   39/50, mu: 0.9033123255, learning_rate: 0.0000436879, weight_decay: 0.0000336648

epoch time: 3235.2089843750s, input_fired_time: 39.9653205872, output_fired_time: 25.2090969086, propagate_forward_time: 1253.9748535156, propagate_backward_time: 1879.9741210938, weight_update_time: 0.9721324444, other_time: 36.0856933594
epoch time: 260.1844177246s, input_fired_time: 3.9825487137, output_fired_time: 0.0000000000, propagate_forward_time: 249.4928283691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7090454102
validation[          , genome  5774] predictions:    6709/  10000 (67.09%), best:    6709/10000 (67.09%), error:      9652.77441, best error:      9652.77441 on epoch:    40, epoch:   40/50, mu: 0.9069672823, learning_rate: 0.0000397171, weight_decay: 0.0000315336

epoch time: 3229.2131347656s, input_fired_time: 41.7190856934, output_fired_time: 22.0970001221, propagate_forward_time: 1245.2257080078, propagate_backward_time: 1882.8858642578, weight_update_time: 0.9701546431, other_time: 37.2856445313
epoch time: 260.9641723633s, input_fired_time: 3.0829565525, output_fired_time: 0.0000000000, propagate_forward_time: 251.2833404541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5978851318
validation[          , genome  5774] predictions:    6667/  10000 (66.67%), best:    6709/10000 (67.09%), error:      9724.94727, best error:      9652.77441 on epoch:    40, epoch:   41/50, mu: 0.9104681611, learning_rate: 0.0000361073, weight_decay: 0.0000295372

epoch time: 3243.2192382813s, input_fired_time: 45.9818191528, output_fired_time: 21.8735237122, propagate_forward_time: 1246.1256103516, propagate_backward_time: 1892.9843750000, weight_update_time: 1.9740351439, other_time: 36.2537841797
epoch time: 260.3313903809s, input_fired_time: 3.1155068874, output_fired_time: 0.0000000000, propagate_forward_time: 250.4632873535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7525939941
validation[          , genome  5774] predictions:    6708/  10000 (67.08%), best:    6709/10000 (67.09%), error:      9723.19824, best error:      9652.77441 on epoch:    40, epoch:   42/50, mu: 0.9138213992, learning_rate: 0.0000328255, weight_decay: 0.0000276673

epoch time: 3233.5476074219s, input_fired_time: 41.0635795593, output_fired_time: 22.1502361298, propagate_forward_time: 1248.6757812500, propagate_backward_time: 1888.4899902344, weight_update_time: 1.0017039776, other_time: 33.1682128906
epoch time: 259.8133544922s, input_fired_time: 3.1274790764, output_fired_time: 0.0000000000, propagate_forward_time: 250.9128417969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7730407715
validation[          , genome  5774] predictions:    6718/  10000 (67.18%), best:    6709/10000 (67.09%), error:      9717.26855, best error:      9652.77441 on epoch:    40, epoch:   43/50, mu: 0.9170332551, learning_rate: 0.0000298420, weight_decay: 0.0000259157

epoch time: 3234.9001464844s, input_fired_time: 41.0810394287, output_fired_time: 22.8450260162, propagate_forward_time: 1248.7819824219, propagate_backward_time: 1886.9118652344, weight_update_time: 1.0244613886, other_time: 35.2802734375
epoch time: 260.0671081543s, input_fired_time: 3.0631408691, output_fired_time: 0.0000000000, propagate_forward_time: 249.1289215088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.8750457764
validation[          , genome  5774] predictions:    6693/  10000 (66.93%), best:    6709/10000 (67.09%), error:      9710.08789, best error:      9652.77441 on epoch:    40, epoch:   44/50, mu: 0.9201096892, learning_rate: 0.0000271297, weight_decay: 0.0000242751

epoch time: 3235.8740234375s, input_fired_time: 39.9644699097, output_fired_time: 22.0499439240, propagate_forward_time: 1252.4091796875, propagate_backward_time: 1886.1090087891, weight_update_time: 1.0352084637, other_time: 35.3414306641
epoch time: 259.5282592773s, input_fired_time: 4.0842747688, output_fired_time: 0.0000000000, propagate_forward_time: 248.6460876465, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7978973389
validation[          , genome  5774] predictions:    6666/  10000 (66.66%), best:    6709/10000 (67.09%), error:      9702.70996, best error:      9652.77441 on epoch:    40, epoch:   45/50, mu: 0.9230564237, learning_rate: 0.0000246639, weight_decay: 0.0000227383

epoch time: 3235.9929199219s, input_fired_time: 42.5818710327, output_fired_time: 23.1191043854, propagate_forward_time: 1245.0335693359, propagate_backward_time: 1889.7401123047, weight_update_time: 1.0438784361, other_time: 35.5183105469
epoch time: 260.6480407715s, input_fired_time: 3.1007480621, output_fired_time: 0.0000000000, propagate_forward_time: 250.5606689453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9866333008
validation[          , genome  5774] predictions:    6729/  10000 (67.29%), best:    6729/10000 (67.29%), error:      9609.44824, best error:      9609.44824 on epoch:    46, epoch:   46/50, mu: 0.9258789420, learning_rate: 0.0000224222, weight_decay: 0.0000212988

epoch time: 3228.2351074219s, input_fired_time: 40.7799797058, output_fired_time: 22.9889678955, propagate_forward_time: 1246.6704101563, propagate_backward_time: 1881.0046386719, weight_update_time: 1.0543359518, other_time: 36.7910156250
epoch time: 261.8358459473s, input_fired_time: 3.1240248680, output_fired_time: 0.0000000000, propagate_forward_time: 251.8840637207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8277587891
validation[          , genome  5774] predictions:    6699/  10000 (66.99%), best:    6729/10000 (67.29%), error:      9675.31836, best error:      9609.44824 on epoch:    46, epoch:   47/50, mu: 0.9285824299, learning_rate: 0.0000203843, weight_decay: 0.0000199504

epoch time: 3236.5576171875s, input_fired_time: 44.0205307007, output_fired_time: 24.9185104370, propagate_forward_time: 1243.7441406250, propagate_backward_time: 1886.8397216797, weight_update_time: 1.0704342127, other_time: 37.0347900391
epoch time: 260.1931457520s, input_fired_time: 4.0773644447, output_fired_time: 0.0000000000, propagate_forward_time: 249.9668273926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1489562988
validation[          , genome  5774] predictions:    6741/  10000 (67.41%), best:    6741/10000 (67.41%), error:      9552.95020, best error:      9552.95020 on epoch:    48, epoch:   48/50, mu: 0.9311719537, learning_rate: 0.0000185316, weight_decay: 0.0000186874

epoch time: 3228.8476562500s, input_fired_time: 40.8986968994, output_fired_time: 27.1265296936, propagate_forward_time: 1240.1386718750, propagate_backward_time: 1885.6824951172, weight_update_time: 1.1164059639, other_time: 35.0013427734
epoch time: 260.7013244629s, input_fired_time: 3.0909557343, output_fired_time: 0.0000000000, propagate_forward_time: 250.6897277832, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9206542969
validation[          , genome  5774] predictions:    6708/  10000 (67.08%), best:    6741/10000 (67.41%), error:      9650.04102, best error:      9552.95020 on epoch:    48, epoch:   49/50, mu: 0.9336522818, learning_rate: 0.0000168473, weight_decay: 0.0000175043

epoch time: 3229.6049804688s, input_fired_time: 43.1770133972, output_fired_time: 24.9697837830, propagate_forward_time: 1238.5122070313, propagate_backward_time: 1887.2723388672, weight_update_time: 1.1522371769, other_time: 35.6737060547
epoch time: 257.9832458496s, input_fired_time: 3.0989940166, output_fired_time: 0.0000000000, propagate_forward_time: 249.0765075684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8077392578
validation[          , genome  5774] predictions:    6754/  10000 (67.54%), best:    6741/10000 (67.41%), error:      9585.60254, best error:      9552.95020 on epoch:    48, epoch:   50/50, mu: 0.9360280633, learning_rate: 0.0000153160, weight_decay: 0.0000163961

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1292.2813720703s, input_fired_time: 20.3167896271, output_fired_time: 0.0000000000, propagate_forward_time: 1239.3215332031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 32.6430664063
best training[          , genome  5774] predictions:   36319/  50000 (72.64%), best:    6741/10000 (67.41%), error:     41416.17578, best error:      9552.95020 on epoch:    48, epoch:   51/50, mu: 0.9383036494, learning_rate: 0.0000139240, weight_decay: 0.0000153581
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 261.6756286621s, input_fired_time: 4.2201151848, output_fired_time: 0.0000000000, propagate_forward_time: 250.6316528320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.8238525391
   testing[          , genome  5774] predictions:    6741/  10000 (67.41%), best:    6741/10000 (67.41%), error:      9552.95020, best error:      9552.95020 on epoch:    48, epoch:   51/50, mu: 0.9383036494, learning_rate: 0.0000139240, weight_decay: 0.0000153581
backpropagation finished successfully!
10:21:00 (9636): called boinc_finish(0)

</stderr_txt>
]]>