Nameexact_genome_1540784103_179_5378_2
Workunit4076329
Created22 Dec 2018, 2:01:51 UTC
Sent22 Dec 2018, 2:01:52 UTC
Report deadline13 Feb 2019, 19:43:00 UTC
Received23 Dec 2018, 13:56:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64351
Run time1 days 8 hours 9 min 39 sec
CPU time1 days 8 hours 2 min 5 sec
Validate stateValid
Credit13,434.23
Device peak FLOPS2.57 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size422.57 MB
Peak swap size420.95 MB
Peak disk usage13.98 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_intelx86.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540784103_179_5378.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540784103_179_5378_2_r1159592936_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1540784103_179_5378.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.516314
read mu: 0.516314
read mu_delta: 0.955668
read initial_learning_rate: 0.000426097
read learning_rate: 0.000426097
read learning_rate_delta: 0.91423
read initial_weight_decay: 0.000446766
read weight_decay: 0.000446766
read weight_decay_delta: 0.933444
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.115935
read input_dropout_probability: 0.00125365
read hidden_dropout_probability: 0.0948368
read velocity_reset: 1803
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5378
read normal distribution: '0 -0.15197879076004 0.968594551086426'
generator_str: '590065086'
read generator: 590065086
read generated_by_map:
1 crossover 1
reading 95 nodes.
reading 480 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 144418556
post shuffle 1: 1033494740
epoch time: 2332.7314453125s, input_fired_time: 91.9851455688477, output_fired_time: 14.5932426452637, propagate_forward_time: 748.038330078125, propagate_backward_time: 1425.76379394531, weight_update_time: 0.766706764698029, other_time: 52.3509521484375
epoch time: 161.890777587891s, input_fired_time: 1.93381643295288, output_fired_time: 0, propagate_forward_time: 150.637313842773, propagate_backward_time: 0, weight_update_time: 0, other_time: 9.31964111328125
validation[          , genome  5378] predictions:    5816/  10000 (58.16%), best:    5816/10000 (58.16%), error:     12176.52734, best error:     12176.52734 on epoch:     0, epoch:    0/50, mu: 0.5163138509, learning_rate: 0.0004260966, weight_decay: 0.0004467659

epoch time: 2333.2126464844s, input_fired_time: 90.8358230591, output_fired_time: 16.5999469757, propagate_forward_time: 759.6044311523, propagate_backward_time: 1414.8083496094, weight_update_time: 0.7648746967, other_time: 51.3640136719
epoch time: 162.9291687012s, input_fired_time: 1.9303764105, output_fired_time: 0.0000000000, propagate_forward_time: 152.7292633057, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2695312500
validation[          , genome  5378] predictions:    6060/  10000 (60.60%), best:    6060/10000 (60.60%), error:     11220.65234, best error:     11220.65234 on epoch:     1, epoch:    1/50, mu: 0.5373132229, learning_rate: 0.0003895502, weight_decay: 0.0004170307

epoch time: 2330.8537597656s, input_fired_time: 90.8564376831, output_fired_time: 13.5577850342, propagate_forward_time: 750.2834472656, propagate_backward_time: 1424.8037109375, weight_update_time: 0.7668857574, other_time: 51.3522949219
epoch time: 162.1406097412s, input_fired_time: 1.9340950251, output_fired_time: 0.0000000000, propagate_forward_time: 151.8934020996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3131103516
validation[          , genome  5378] predictions:    6240/  10000 (62.40%), best:    6240/10000 (62.40%), error:     11000.46387, best error:     11000.46387 on epoch:     2, epoch:    2/50, mu: 0.5573816299, learning_rate: 0.0003561384, weight_decay: 0.0003892747

epoch time: 2331.2399902344s, input_fired_time: 90.8249588013, output_fired_time: 16.5844440460, propagate_forward_time: 743.2464599609, propagate_backward_time: 1427.1705322266, weight_update_time: 0.7703165412, other_time: 53.4135742188
epoch time: 161.9860687256s, input_fired_time: 3.9240589142, output_fired_time: 0.0000000000, propagate_forward_time: 150.7832031250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2788085938
validation[          , genome  5378] predictions:    5983/  10000 (59.83%), best:    6240/10000 (62.40%), error:     11665.13086, best error:     11000.46387 on epoch:     2, epoch:    3/50, mu: 0.5765603781, learning_rate: 0.0003255923, weight_decay: 0.0003633660

epoch time: 2332.5769042969s, input_fired_time: 90.8466491699, output_fired_time: 16.5997962952, propagate_forward_time: 759.3463134766, propagate_backward_time: 1414.4005126953, weight_update_time: 0.7664311528, other_time: 51.3835449219
epoch time: 161.9823303223s, input_fired_time: 1.9170980453, output_fired_time: 0.0000000000, propagate_forward_time: 152.8102569580, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2549743652
validation[          , genome  5378] predictions:    6109/  10000 (61.09%), best:    6240/10000 (62.40%), error:     11314.94824, best error:     11000.46387 on epoch:     2, epoch:    4/50, mu: 0.5948889256, learning_rate: 0.0002976661, weight_decay: 0.0003391816

epoch time: 2333.6564941406s, input_fired_time: 88.8278427124, output_fired_time: 15.5852546692, propagate_forward_time: 755.6082763672, propagate_backward_time: 1422.2851562500, weight_update_time: 1.7712048292, other_time: 51.3499755859
epoch time: 161.8347930908s, input_fired_time: 1.9053444862, output_fired_time: 0.0000000000, propagate_forward_time: 151.6810455322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2483978271
validation[          , genome  5378] predictions:    6189/  10000 (61.89%), best:    6240/10000 (62.40%), error:     11206.86719, best error:     11000.46387 on epoch:     2, epoch:    5/50, mu: 0.6124048829, learning_rate: 0.0002721353, weight_decay: 0.0003166069

epoch time: 2335.6625976563s, input_fired_time: 93.7739410400, output_fired_time: 14.5930480957, propagate_forward_time: 747.6294555664, propagate_backward_time: 1428.2227783203, weight_update_time: 0.7641199827, other_time: 51.4434814453
epoch time: 162.1155395508s, input_fired_time: 1.9001786709, output_fired_time: 0.0000000000, propagate_forward_time: 151.9150085449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3003540039
validation[          , genome  5378] predictions:    6019/  10000 (60.19%), best:    6240/10000 (62.40%), error:     11559.12012, best error:     11000.46387 on epoch:     2, epoch:    6/50, mu: 0.6291443706, learning_rate: 0.0002487942, weight_decay: 0.0002955347

epoch time: 2336.3046875000s, input_fired_time: 89.8535079956, output_fired_time: 15.6326847076, propagate_forward_time: 757.0607299805, propagate_backward_time: 1421.3233642578, weight_update_time: 0.7630684972, other_time: 52.4344482422
epoch time: 161.6698455811s, input_fired_time: 1.8905748129, output_fired_time: 0.0000000000, propagate_forward_time: 152.4968566895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2824096680
validation[          , genome  5378] predictions:    5817/  10000 (58.17%), best:    6240/10000 (62.40%), error:     12072.82031, best error:     11000.46387 on epoch:     2, epoch:    7/50, mu: 0.6451417208, learning_rate: 0.0002274550, weight_decay: 0.0002758650

epoch time: 2330.2141113281s, input_fired_time: 85.8167572021, output_fired_time: 17.7042350769, propagate_forward_time: 751.6372680664, propagate_backward_time: 1422.6881103516, weight_update_time: 0.7654631734, other_time: 52.3677978516
epoch time: 163.7101745605s, input_fired_time: 1.9043514729, output_fired_time: 0.0000000000, propagate_forward_time: 153.4347381592, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3710784912
validation[          , genome  5378] predictions:    6139/  10000 (61.39%), best:    6240/10000 (62.40%), error:     11370.17676, best error:     11000.46387 on epoch:     2, epoch:    8/50, mu: 0.6604298949, learning_rate: 0.0002079461, weight_decay: 0.0002575044

epoch time: 2331.2795410156s, input_fired_time: 91.7202758789, output_fired_time: 14.5627098083, propagate_forward_time: 755.9132690430, propagate_backward_time: 1416.6877441406, weight_update_time: 0.7678666115, other_time: 52.3955078125
epoch time: 161.7260284424s, input_fired_time: 1.9005924463, output_fired_time: 0.0000000000, propagate_forward_time: 151.5484924316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2769470215
validation[          , genome  5378] predictions:    6176/  10000 (61.76%), best:    6240/10000 (62.40%), error:     11139.11816, best error:     11000.46387 on epoch:     2, epoch:    9/50, mu: 0.6750403047, learning_rate: 0.0001901105, weight_decay: 0.0002403658

epoch time: 2331.4528808594s, input_fired_time: 89.6727828979, output_fired_time: 15.5621767044, propagate_forward_time: 756.7001342773, propagate_backward_time: 1418.1805419922, weight_update_time: 0.7720245123, other_time: 51.3370361328
epoch time: 161.8333282471s, input_fired_time: 2.8962275982, output_fired_time: 0.0000000000, propagate_forward_time: 151.6775207520, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2595825195
validation[          , genome  5378] predictions:    6228/  10000 (62.28%), best:    6240/10000 (62.40%), error:     11076.57520, best error:     11000.46387 on epoch:     2, epoch:   10/50, mu: 0.6890030503, learning_rate: 0.0001738047, weight_decay: 0.0002243679

epoch time: 2331.3234863281s, input_fired_time: 90.7411422729, output_fired_time: 14.5723857880, propagate_forward_time: 754.8307495117, propagate_backward_time: 1419.3448486328, weight_update_time: 0.7638447881, other_time: 51.8343505859
epoch time: 161.0781250000s, input_fired_time: 2.8852119446, output_fired_time: 0.0000000000, propagate_forward_time: 149.8774261475, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3154907227
validation[          , genome  5378] predictions:    6091/  10000 (60.91%), best:    6240/10000 (62.40%), error:     11584.87109, best error:     11000.46387 on epoch:     2, epoch:   11/50, mu: 0.7023468018, learning_rate: 0.0001588974, weight_decay: 0.0002094348

epoch time: 2330.8884277344s, input_fired_time: 90.6787643433, output_fired_time: 15.5807113647, propagate_forward_time: 753.5529785156, propagate_backward_time: 1419.8175048828, weight_update_time: 0.7671258450, other_time: 51.2584228516
epoch time: 161.5945587158s, input_fired_time: 2.8982048035, output_fired_time: 0.0000000000, propagate_forward_time: 149.4217987061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2745513916
validation[          , genome  5378] predictions:    5989/  10000 (59.89%), best:    6240/10000 (62.40%), error:     11684.66406, best error:     11000.46387 on epoch:     2, epoch:   12/50, mu: 0.7150989771, learning_rate: 0.0001452688, weight_decay: 0.0001954956

epoch time: 2332.6752929688s, input_fired_time: 88.7503738403, output_fired_time: 14.5536851883, propagate_forward_time: 763.8536376953, propagate_backward_time: 1414.1293945313, weight_update_time: 0.7657604218, other_time: 51.3880615234
epoch time: 160.2292327881s, input_fired_time: 2.9026603699, output_fired_time: 0.0000000000, propagate_forward_time: 148.0370788574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2894897461
validation[          , genome  5378] predictions:    6235/  10000 (62.35%), best:    6240/10000 (62.40%), error:     11142.22070, best error:     11000.46387 on epoch:     2, epoch:   13/50, mu: 0.7272858024, learning_rate: 0.0001328090, weight_decay: 0.0001824841

epoch time: 2332.0815429688s, input_fired_time: 91.7180709839, output_fired_time: 16.5589141846, propagate_forward_time: 753.7884521484, propagate_backward_time: 1418.9957275391, weight_update_time: 0.7685704827, other_time: 51.0205078125
epoch time: 161.6738891602s, input_fired_time: 1.8820846081, output_fired_time: 0.0000000000, propagate_forward_time: 152.5306091309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2611999512
validation[          , genome  5378] predictions:    6084/  10000 (60.84%), best:    6240/10000 (62.40%), error:     11311.70996, best error:     11000.46387 on epoch:     2, epoch:   14/50, mu: 0.7389323711, learning_rate: 0.0001214180, weight_decay: 0.0001703386

epoch time: 2331.6975097656s, input_fired_time: 87.7007751465, output_fired_time: 15.5904350281, propagate_forward_time: 747.4039916992, propagate_backward_time: 1430.3382568359, weight_update_time: 0.7655336261, other_time: 50.6641845703
epoch time: 162.9539794922s, input_fired_time: 1.8965951204, output_fired_time: 0.0000000000, propagate_forward_time: 151.7839202881, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2734680176
validation[          , genome  5378] predictions:    5999/  10000 (59.99%), best:    6240/10000 (62.40%), error:     11626.63184, best error:     11000.46387 on epoch:     2, epoch:   15/50, mu: 0.7500626445, learning_rate: 0.0001110039, weight_decay: 0.0001590015

epoch time: 2334.2016601563s, input_fired_time: 84.8055801392, output_fired_time: 15.6248979568, propagate_forward_time: 761.9229125977, propagate_backward_time: 1419.4681396484, weight_update_time: 0.7659425735, other_time: 52.3800048828
epoch time: 163.2044525146s, input_fired_time: 1.9015315771, output_fired_time: 0.0000000000, propagate_forward_time: 153.0156860352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2872314453
validation[          , genome  5378] predictions:    6201/  10000 (62.01%), best:    6240/10000 (62.40%), error:     11057.48340, best error:     11000.46387 on epoch:     2, epoch:   16/50, mu: 0.7606995106, learning_rate: 0.0001014831, weight_decay: 0.0001484189

epoch time: 2338.2785644531s, input_fired_time: 91.9639587402, output_fired_time: 13.5795078278, propagate_forward_time: 750.4049682617, propagate_backward_time: 1431.8591308594, weight_update_time: 0.7791964412, other_time: 50.4709472656
epoch time: 162.0969238281s, input_fired_time: 1.9040765762, output_fired_time: 0.0000000000, propagate_forward_time: 152.9418945313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2509460449
validation[          , genome  5378] predictions:    5969/  10000 (59.69%), best:    6240/10000 (62.40%), error:     11675.69629, best error:     11000.46387 on epoch:     2, epoch:   17/50, mu: 0.7708647847, learning_rate: 0.0000927789, weight_decay: 0.0001385407

epoch time: 2337.0898437500s, input_fired_time: 90.9439392090, output_fired_time: 14.6001844406, propagate_forward_time: 765.7506713867, propagate_backward_time: 1415.3721923828, weight_update_time: 0.7658268809, other_time: 50.4229736328
epoch time: 162.4204406738s, input_fired_time: 1.9107122421, output_fired_time: 0.0000000000, propagate_forward_time: 152.2013702393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3083648682
validation[          , genome  5378] predictions:    6273/  10000 (62.73%), best:    6273/10000 (62.73%), error:     10868.58496, best error:     10868.58496 on epoch:    18, epoch:   18/50, mu: 0.7805794477, learning_rate: 0.0000848212, weight_decay: 0.0001293199

epoch time: 2335.1694335938s, input_fired_time: 94.0596389771, output_fired_time: 14.6454257965, propagate_forward_time: 753.1200561523, propagate_backward_time: 1421.0047607422, weight_update_time: 0.7646881938, other_time: 52.3394775391
epoch time: 161.9274902344s, input_fired_time: 2.9017302990, output_fired_time: 0.0000000000, propagate_forward_time: 151.7362060547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2895507813
validation[          , genome  5378] predictions:    6128/  10000 (61.28%), best:    6273/10000 (62.73%), error:     11081.47461, best error:     10868.58496 on epoch:    18, epoch:   19/50, mu: 0.7898634076, learning_rate: 0.0000775461, weight_decay: 0.0001207129

epoch time: 2335.5314941406s, input_fired_time: 89.8135681152, output_fired_time: 14.5877437592, propagate_forward_time: 751.6417846680, propagate_backward_time: 1428.1142578125, weight_update_time: 0.7661542296, other_time: 51.3742675781
epoch time: 162.2502136230s, input_fired_time: 1.9174389839, output_fired_time: 0.0000000000, propagate_forward_time: 153.0730743408, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2597045898
validation[          , genome  5378] predictions:    6249/  10000 (62.49%), best:    6273/10000 (62.73%), error:     10965.82910, best error:     10868.58496 on epoch:    18, epoch:   20/50, mu: 0.7987357974, learning_rate: 0.0000708949, weight_decay: 0.0001126787

epoch time: 2338.2370605469s, input_fired_time: 92.1288681030, output_fired_time: 13.6108579636, propagate_forward_time: 769.1387329102, propagate_backward_time: 1411.8576660156, weight_update_time: 0.7736608982, other_time: 51.5009765625
epoch time: 162.4669647217s, input_fired_time: 1.9048037529, output_fired_time: 0.0000000000, propagate_forward_time: 152.2659301758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2962341309
validation[          , genome  5378] predictions:    6236/  10000 (62.36%), best:    6273/10000 (62.73%), error:     10909.35645, best error:     10868.58496 on epoch:    18, epoch:   21/50, mu: 0.8072148561, learning_rate: 0.0000648142, weight_decay: 0.0001051792

epoch time: 2338.5864257813s, input_fired_time: 90.0213317871, output_fired_time: 14.6402559280, propagate_forward_time: 745.2775268555, propagate_backward_time: 1438.1812744141, weight_update_time: 0.7656657696, other_time: 50.4661865234
epoch time: 161.3665466309s, input_fired_time: 2.9076409340, output_fired_time: 0.0000000000, propagate_forward_time: 149.1097564697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3491516113
validation[          , genome  5378] predictions:    6253/  10000 (62.53%), best:    6273/10000 (62.73%), error:     10870.70508, best error:     10868.58496 on epoch:    18, epoch:   22/50, mu: 0.8153180480, learning_rate: 0.0000592551, weight_decay: 0.0000981788

epoch time: 2342.6672363281s, input_fired_time: 87.0899505615, output_fired_time: 16.6349506378, propagate_forward_time: 769.3044433594, propagate_backward_time: 1418.0350341797, weight_update_time: 0.7756514549, other_time: 51.6029052734
epoch time: 162.3244781494s, input_fired_time: 1.9079296589, output_fired_time: 0.0000000000, propagate_forward_time: 153.1531982422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2633514404
validation[          , genome  5378] predictions:    6356/  10000 (63.56%), best:    6356/10000 (63.56%), error:     10668.46387, best error:     10668.46387 on epoch:    23, epoch:   23/50, mu: 0.8230620027, learning_rate: 0.0000541728, weight_decay: 0.0000916444

epoch time: 2341.4482421875s, input_fired_time: 88.1269531250, output_fired_time: 15.6080427170, propagate_forward_time: 771.7695922852, propagate_backward_time: 1413.2396240234, weight_update_time: 0.7761130929, other_time: 52.7039794922
epoch time: 163.5730590820s, input_fired_time: 1.9205156565, output_fired_time: 0.0000000000, propagate_forward_time: 154.4087066650, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2438354492
validation[          , genome  5378] predictions:    6314/  10000 (63.14%), best:    6356/10000 (63.56%), error:     10830.72559, best error:     10668.46387 on epoch:    23, epoch:   24/50, mu: 0.8304626346, learning_rate: 0.0000495264, weight_decay: 0.0000855449

epoch time: 2339.0773925781s, input_fired_time: 86.0431747437, output_fired_time: 15.6699934006, propagate_forward_time: 759.2341918945, propagate_backward_time: 1428.6909179688, weight_update_time: 0.7730329633, other_time: 49.4392089844
epoch time: 162.5126647949s, input_fired_time: 2.9201183319, output_fired_time: 0.0000000000, propagate_forward_time: 151.2631835938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3293609619
validation[          , genome  5378] predictions:    6316/  10000 (63.16%), best:    6356/10000 (63.56%), error:     10747.49023, best error:     10668.46387 on epoch:    23, epoch:   25/50, mu: 0.8375352025, learning_rate: 0.0000452785, weight_decay: 0.0000798513

epoch time: 2339.1291503906s, input_fired_time: 90.0699310303, output_fired_time: 15.6156721115, propagate_forward_time: 745.1572265625, propagate_backward_time: 1436.7331542969, weight_update_time: 0.7696952224, other_time: 51.5532226563
epoch time: 162.4312286377s, input_fired_time: 1.9145857096, output_fired_time: 0.0000000000, propagate_forward_time: 153.2581787109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2584686279
validation[          , genome  5378] predictions:    6332/  10000 (63.32%), best:    6356/10000 (63.56%), error:     10728.88965, best error:     10668.46387 on epoch:    23, epoch:   26/50, mu: 0.8442942500, learning_rate: 0.0000413949, weight_decay: 0.0000745367

epoch time: 2362.5270996094s, input_fired_time: 88.0391540527, output_fired_time: 14.6374483109, propagate_forward_time: 735.8493652344, propagate_backward_time: 1455.5895996094, weight_update_time: 0.7724741697, other_time: 68.4116210938
epoch time: 162.7737884521s, input_fired_time: 2.9229938984, output_fired_time: 0.0000000000, propagate_forward_time: 151.4684448242, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3823547363
validation[          , genome  5378] predictions:    6310/  10000 (63.10%), best:    6356/10000 (63.56%), error:     10797.08398, best error:     10668.46387 on epoch:    23, epoch:   27/50, mu: 0.8507536650, learning_rate: 0.0000378445, weight_decay: 0.0000695758

epoch time: 2335.3044433594s, input_fired_time: 86.0571975708, output_fired_time: 16.7085037231, propagate_forward_time: 757.9215698242, propagate_backward_time: 1423.0966796875, weight_update_time: 0.7705903053, other_time: 51.5205078125
epoch time: 162.0509948730s, input_fired_time: 1.9111379385, output_fired_time: 0.0000000000, propagate_forward_time: 151.8420104980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2978515625
validation[          , genome  5378] predictions:    6366/  10000 (63.66%), best:    6356/10000 (63.56%), error:     10693.84082, best error:     10668.46387 on epoch:    23, epoch:   28/50, mu: 0.8569266796, learning_rate: 0.0000345985, weight_decay: 0.0000649451

epoch time: 2338.7590332031s, input_fired_time: 88.0421295166, output_fired_time: 16.6117591858, propagate_forward_time: 761.4358520508, propagate_backward_time: 1423.2148437500, weight_update_time: 0.7656238675, other_time: 49.4543457031
epoch time: 162.1877441406s, input_fired_time: 2.9134607315, output_fired_time: 0.0000000000, propagate_forward_time: 151.9858551025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2884216309
validation[          , genome  5378] predictions:    6406/  10000 (64.06%), best:    6406/10000 (64.06%), error:     10542.25000, best error:     10542.25000 on epoch:    29, epoch:   29/50, mu: 0.8628260493, learning_rate: 0.0000316310, weight_decay: 0.0000606226

epoch time: 2341.6809082031s, input_fired_time: 89.0745086670, output_fired_time: 15.6520586014, propagate_forward_time: 765.9146118164, propagate_backward_time: 1420.4002685547, weight_update_time: 0.7782472372, other_time: 50.6395263672
epoch time: 162.2625122070s, input_fired_time: 1.9088048935, output_fired_time: 0.0000000000, propagate_forward_time: 152.0592651367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2944488525
validation[          , genome  5378] predictions:    6343/  10000 (63.43%), best:    6406/10000 (64.06%), error:     10626.99023, best error:     10542.25000 on epoch:    29, epoch:   30/50, mu: 0.8684638739, learning_rate: 0.0000289180, weight_decay: 0.0000565878

epoch time: 2345.5190429688s, input_fired_time: 88.1974716187, output_fired_time: 15.6718320847, propagate_forward_time: 756.1383666992, propagate_backward_time: 1433.9285888672, weight_update_time: 0.7678472996, other_time: 51.5826416016
epoch time: 162.6591033936s, input_fired_time: 1.9128233194, output_fired_time: 0.0000000000, propagate_forward_time: 153.4233093262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3229675293
validation[          , genome  5378] predictions:    6411/  10000 (64.11%), best:    6411/10000 (64.11%), error:     10463.31738, best error:     10463.31738 on epoch:    31, epoch:   31/50, mu: 0.8738517761, learning_rate: 0.0000264377, weight_decay: 0.0000528215

epoch time: 2343.7143554688s, input_fired_time: 88.1511154175, output_fired_time: 14.6605281830, propagate_forward_time: 747.8289184570, propagate_backward_time: 1441.5445556641, weight_update_time: 0.7655584812, other_time: 51.5291748047
epoch time: 162.6166534424s, input_fired_time: 1.9222851992, output_fired_time: 0.0000000000, propagate_forward_time: 152.3216705322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3726959229
validation[          , genome  5378] predictions:    6422/  10000 (64.22%), best:    6411/10000 (64.11%), error:     10611.89453, best error:     10463.31738 on epoch:    31, epoch:   32/50, mu: 0.8790008426, learning_rate: 0.0000241701, weight_decay: 0.0000493059

epoch time: 2343.5061035156s, input_fired_time: 89.2032089233, output_fired_time: 15.6674289703, propagate_forward_time: 755.9424438477, propagate_backward_time: 1430.0772705078, weight_update_time: 0.7688165307, other_time: 52.6158447266
epoch time: 162.5768280029s, input_fired_time: 2.9215593338, output_fired_time: 0.0000000000, propagate_forward_time: 151.3811340332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2741394043
validation[          , genome  5378] predictions:    6375/  10000 (63.75%), best:    6411/10000 (64.11%), error:     10561.05957, best error:     10463.31738 on epoch:    31, epoch:   33/50, mu: 0.8839216232, learning_rate: 0.0000220971, weight_decay: 0.0000460243

epoch time: 2343.6437988281s, input_fired_time: 92.2651214600, output_fired_time: 16.6566371918, propagate_forward_time: 767.7190551758, propagate_backward_time: 1415.4479980469, weight_update_time: 2.7721381187, other_time: 51.5549316406
epoch time: 163.6816101074s, input_fired_time: 1.9268141985, output_fired_time: 0.0000000000, propagate_forward_time: 153.3613739014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3934173584
validation[          , genome  5378] predictions:    6431/  10000 (64.31%), best:    6411/10000 (64.11%), error:     10477.93262, best error:     10463.31738 on epoch:    31, epoch:   34/50, mu: 0.8886242509, learning_rate: 0.0000202018, weight_decay: 0.0000429611

epoch time: 2344.7211914063s, input_fired_time: 92.2646408081, output_fired_time: 17.7136726379, propagate_forward_time: 757.3162841797, propagate_backward_time: 1423.8706054688, weight_update_time: 0.7637034655, other_time: 53.5560302734
epoch time: 163.6563262939s, input_fired_time: 1.9183267355, output_fired_time: 0.0000000000, propagate_forward_time: 152.3906707764, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3473358154
validation[          , genome  5378] predictions:    6374/  10000 (63.74%), best:    6411/10000 (64.11%), error:     10476.70020, best error:     10463.31738 on epoch:    31, epoch:   35/50, mu: 0.8931184411, learning_rate: 0.0000184691, weight_decay: 0.0000401017

epoch time: 2341.0888671875s, input_fired_time: 89.3510665894, output_fired_time: 16.6987209320, propagate_forward_time: 750.6906127930, propagate_backward_time: 1432.7512207031, weight_update_time: 0.7771378160, other_time: 51.5971679688
epoch time: 162.5521240234s, input_fired_time: 1.9223357439, output_fired_time: 0.0000000000, propagate_forward_time: 151.3401641846, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.2896270752
validation[          , genome  5378] predictions:    6443/  10000 (64.43%), best:    6443/10000 (64.43%), error:     10461.85156, best error:     10461.85156 on epoch:    36, epoch:   36/50, mu: 0.8974133730, learning_rate: 0.0000168850, weight_decay: 0.0000374327

epoch time: 2344.7292480469s, input_fired_time: 87.2649993896, output_fired_time: 17.6935272217, propagate_forward_time: 765.0404663086, propagate_backward_time: 1424.1560058594, weight_update_time: 0.7750193477, other_time: 50.5742187500
epoch time: 162.8521728516s, input_fired_time: 1.9232660532, output_fired_time: 0.0000000000, propagate_forward_time: 151.5572052002, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3717041016
validation[          , genome  5378] predictions:    6453/  10000 (64.53%), best:    6453/10000 (64.53%), error:     10323.58008, best error:     10323.58008 on epoch:    37, epoch:   37/50, mu: 0.9015179276, learning_rate: 0.0000154368, weight_decay: 0.0000349413

epoch time: 2341.6391601563s, input_fired_time: 95.2177886963, output_fired_time: 13.6631584167, propagate_forward_time: 751.3089599609, propagate_backward_time: 1427.7415771484, weight_update_time: 0.7679026127, other_time: 53.7077636719
epoch time: 162.1414489746s, input_fired_time: 1.9193925858, output_fired_time: 0.0000000000, propagate_forward_time: 151.9094390869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3126220703
validation[          , genome  5378] predictions:    6488/  10000 (64.88%), best:    6488/10000 (64.88%), error:     10308.53906, best error:     10308.53906 on epoch:    38, epoch:   38/50, mu: 0.9054405093, learning_rate: 0.0000141127, weight_decay: 0.0000326157

epoch time: 2342.3305664063s, input_fired_time: 87.1437683105, output_fired_time: 15.6425533295, propagate_forward_time: 748.8603515625, propagate_backward_time: 1442.2171630859, weight_update_time: 0.7690106630, other_time: 48.4666748047
epoch time: 161.9885101318s, input_fired_time: 1.9197078943, output_fired_time: 0.0000000000, propagate_forward_time: 151.7966918945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2721099854
validation[          , genome  5378] predictions:    6414/  10000 (64.14%), best:    6488/10000 (64.88%), error:     10343.72363, best error:     10308.53906 on epoch:    38, epoch:   39/50, mu: 0.9091891646, learning_rate: 0.0000129023, weight_decay: 0.0000304450

epoch time: 2338.1120605469s, input_fired_time: 89.0401992798, output_fired_time: 13.6184473038, propagate_forward_time: 764.9501342773, propagate_backward_time: 1419.0252685547, weight_update_time: 0.7641566396, other_time: 51.4779052734
epoch time: 162.3066406250s, input_fired_time: 1.9252389669, output_fired_time: 0.0000000000, propagate_forward_time: 152.0544281006, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3269805908
validation[          , genome  5378] predictions:    6514/  10000 (65.14%), best:    6514/10000 (65.14%), error:     10250.15137, best error:     10250.15137 on epoch:    40, epoch:   40/50, mu: 0.9127716422, learning_rate: 0.0000117957, weight_decay: 0.0000284186

epoch time: 2340.4936523438s, input_fired_time: 91.1189346313, output_fired_time: 14.6000118256, propagate_forward_time: 761.3720092773, propagate_backward_time: 1420.8948974609, weight_update_time: 0.7737406492, other_time: 52.5076904297
epoch time: 163.3410491943s, input_fired_time: 1.9342083931, output_fired_time: 0.0000000000, propagate_forward_time: 153.0676116943, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3392333984
validation[          , genome  5378] predictions:    6447/  10000 (64.47%), best:    6514/10000 (65.14%), error:     10422.37109, best error:     10250.15137 on epoch:    40, epoch:   41/50, mu: 0.9161953330, learning_rate: 0.0000107839, weight_decay: 0.0000265272

epoch time: 2338.8862304688s, input_fired_time: 92.0750885010, output_fired_time: 14.6048212051, propagate_forward_time: 749.6003417969, propagate_backward_time: 1432.2034912109, weight_update_time: 0.7656000853, other_time: 50.4024658203
epoch time: 162.0596923828s, input_fired_time: 1.9310280085, output_fired_time: 0.0000000000, propagate_forward_time: 151.8276214600, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3010406494
validation[          , genome  5378] predictions:    6464/  10000 (64.64%), best:    6514/10000 (65.14%), error:     10331.33887, best error:     10250.15137 on epoch:    40, epoch:   42/50, mu: 0.9194672108, learning_rate: 0.0000098590, weight_decay: 0.0000247617

epoch time: 2338.1674804688s, input_fired_time: 90.1009750366, output_fired_time: 14.6149377823, propagate_forward_time: 766.5186767578, propagate_backward_time: 1414.3861083984, weight_update_time: 0.7696959972, other_time: 52.5466308594
epoch time: 162.3752593994s, input_fired_time: 1.9306935072, output_fired_time: 0.0000000000, propagate_forward_time: 153.1596069336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2849578857
validation[          , genome  5378] predictions:    6457/  10000 (64.57%), best:    6514/10000 (65.14%), error:     10379.65625, best error:     10250.15137 on epoch:    40, epoch:   43/50, mu: 0.9225940704, learning_rate: 0.0000090134, weight_decay: 0.0000231136

epoch time: 2340.0803222656s, input_fired_time: 91.0577850342, output_fired_time: 14.6322345734, propagate_forward_time: 758.4412841797, propagate_backward_time: 1422.3814697266, weight_update_time: 0.7636429667, other_time: 53.5673828125
epoch time: 163.3828735352s, input_fired_time: 1.9345973730, output_fired_time: 0.0000000000, propagate_forward_time: 153.1441192627, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3041534424
validation[          , genome  5378] predictions:    6484/  10000 (64.84%), best:    6484/10000 (64.84%), error:     10224.51172, best error:     10224.51172 on epoch:    44, epoch:   44/50, mu: 0.9255822897, learning_rate: 0.0000082403, weight_decay: 0.0000215752

epoch time: 2338.5212402344s, input_fired_time: 88.0323333740, output_fired_time: 14.6150770187, propagate_forward_time: 752.6848144531, propagate_backward_time: 1431.7296142578, weight_update_time: 0.7631009817, other_time: 51.4595947266
epoch time: 162.2652587891s, input_fired_time: 1.9273889065, output_fired_time: 0.0000000000, propagate_forward_time: 151.0184020996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3194732666
validation[          , genome  5378] predictions:    6483/  10000 (64.83%), best:    6484/10000 (64.84%), error:     10264.28809, best error:     10224.51172 on epoch:    44, epoch:   45/50, mu: 0.9284380674, learning_rate: 0.0000075335, weight_decay: 0.0000201393

epoch time: 2337.2980957031s, input_fired_time: 89.0330352783, output_fired_time: 13.6042566299, propagate_forward_time: 767.6036376953, propagate_backward_time: 1416.4761962891, weight_update_time: 0.7685706615, other_time: 50.5810546875
epoch time: 160.8099975586s, input_fired_time: 1.9267867804, output_fired_time: 0.0000000000, propagate_forward_time: 150.5707702637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3124389648
validation[          , genome  5378] predictions:    6507/  10000 (65.07%), best:    6484/10000 (64.84%), error:     10332.97852, best error:     10224.51172 on epoch:    44, epoch:   46/50, mu: 0.9311672449, learning_rate: 0.0000068874, weight_decay: 0.0000187989

epoch time: 2333.4118652344s, input_fired_time: 88.9686431885, output_fired_time: 14.6008167267, propagate_forward_time: 756.7693481445, propagate_backward_time: 1420.6218261719, weight_update_time: 0.7682273984, other_time: 52.4511718750
epoch time: 163.3193664551s, input_fired_time: 1.9362426996, output_fired_time: 0.0000000000, propagate_forward_time: 153.0798187256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3032989502
validation[          , genome  5378] predictions:    6471/  10000 (64.71%), best:    6484/10000 (64.84%), error:     10318.90332, best error:     10224.51172 on epoch:    44, epoch:   47/50, mu: 0.9337754250, learning_rate: 0.0000062966, weight_decay: 0.0000175477

epoch time: 2339.5666503906s, input_fired_time: 95.0947799683, output_fired_time: 14.7100515366, propagate_forward_time: 755.1417846680, propagate_backward_time: 1422.1036376953, weight_update_time: 0.7694623470, other_time: 52.5164794922
epoch time: 162.6864471436s, input_fired_time: 1.9412657022, output_fired_time: 0.0000000000, propagate_forward_time: 152.4010925293, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3440856934
validation[          , genome  5378] predictions:    6526/  10000 (65.26%), best:    6526/10000 (65.26%), error:     10184.00879, best error:     10184.00879 on epoch:    48, epoch:   48/50, mu: 0.9362679720, learning_rate: 0.0000057566, weight_decay: 0.0000163798

epoch time: 2344.1748046875s, input_fired_time: 93.2550811768, output_fired_time: 13.6363210678, propagate_forward_time: 755.9401855469, propagate_backward_time: 1430.7519531250, weight_update_time: 0.7670552731, other_time: 50.5913085938
epoch time: 162.4422149658s, input_fired_time: 1.9323546886, output_fired_time: 0.0000000000, propagate_forward_time: 150.2131958008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.2966613770
validation[          , genome  5378] predictions:    6534/  10000 (65.34%), best:    6526/10000 (65.26%), error:     10188.87305, best error:     10184.00879 on epoch:    48, epoch:   49/50, mu: 0.9386500120, learning_rate: 0.0000052628, weight_decay: 0.0000152896

epoch time: 2347.9909667969s, input_fired_time: 95.3585891724, output_fired_time: 14.6285371780, propagate_forward_time: 771.1557617188, propagate_backward_time: 1416.3352050781, weight_update_time: 0.7650167346, other_time: 50.5129394531
epoch time: 162.4895782471s, input_fired_time: 1.9402433634, output_fired_time: 0.0000000000, propagate_forward_time: 153.2158203125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3335113525
validation[          , genome  5378] predictions:    6532/  10000 (65.32%), best:    6532/10000 (65.32%), error:     10178.47754, best error:     10178.47754 on epoch:    50, epoch:   50/50, mu: 0.9409264326, learning_rate: 0.0000048114, weight_decay: 0.0000142720

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 810.1755981445s, input_fired_time: 9.6533117294, output_fired_time: 0.0000000000, propagate_forward_time: 757.9860839844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 42.5361938477
best training[          , genome  5378] predictions:   34445/  50000 (68.89%), best:    6532/10000 (65.32%), error:     46822.42188, best error:     10178.47754 on epoch:    50, epoch:   51/50, mu: 0.9431019425, learning_rate: 0.0000043988, weight_decay: 0.0000133221
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 163.0081787109s, input_fired_time: 2.9348890781, output_fired_time: 0.0000000000, propagate_forward_time: 151.4353027344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.6379852295
   testing[          , genome  5378] predictions:    6532/  10000 (65.32%), best:    6532/10000 (65.32%), error:     10178.47754, best error:     10178.47754 on epoch:    50, epoch:   51/50, mu: 0.9431019425, learning_rate: 0.0000043988, weight_decay: 0.0000133221
backpropagation finished successfully!
17:46:05 (5948): called boinc_finish(0)

</stderr_txt>
]]>