Nameexact_genome_1542383404_179_6104_2
Workunit4093799
Created20 Nov 2018, 1:25:34 UTC
Sent20 Nov 2018, 1:27:50 UTC
Report deadline23 Jan 2019, 2:34:20 UTC
Received23 Nov 2018, 0:59:02 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID46074
Run time1 days 1 hours 19 min 43 sec
CPU time1 days 1 hours 18 min 28 sec
Validate stateValid
Credit16,011.55
Device peak FLOPS3.50 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size568.79 MB
Peak swap size567.29 MB
Peak disk usage15.66 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104_2_r818788353_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.512452
read mu: 0.512452
read mu_delta: 0.954665
read initial_learning_rate: 0.00186827
read learning_rate: 0.00186827
read learning_rate_delta: 0.913556
read initial_weight_decay: 0.000439731
read weight_decay: 0.000439731
read weight_decay_delta: 0.936346
read batch_size: 107
read epsilon: 1e-07
read alpha: 0.118664
read input_dropout_probability: 0.00133047
read hidden_dropout_probability: 0.0964702
read velocity_reset: 1748
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6104
read normal distribution: '0 -0.355088144540787 0.925766110420227'
generator_str: '662050602'
read generator: 662050602
read generated_by_map:
1 change_size 1
reading 107 nodes.
reading 565 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 971692707
post shuffle 1: 189589552
epoch time: 1587.54553222656s, input_fired_time: 21.9814929962158, output_fired_time: 9.39474487304688, propagate_forward_time: 640.388732910156, propagate_backward_time: 901.693481445313, weight_update_time: 0.583287119865417, other_time: 14.0870971679688
epoch time: 185.976303100586s, input_fired_time: 1.99611580371857, output_fired_time: 0, propagate_forward_time: 181.150238037109, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.8299560546875
validation[          , genome  6104] predictions:    5594/  10000 (55.94%), best:    5594/10000 (55.94%), error:     13014.06250, best error:     13014.06250 on epoch:     0, epoch:    0/50, mu: 0.5124523640, learning_rate: 0.0018682743, weight_decay: 0.0004397306

epoch time: 1617.5474853516s, input_fired_time: 22.1674385071, output_fired_time: 9.5227947235, propagate_forward_time: 651.7507324219, propagate_backward_time: 919.9062500000, weight_update_time: 0.5931220651, other_time: 14.2001953125
epoch time: 174.4545440674s, input_fired_time: 1.9139136076, output_fired_time: 0.0000000000, propagate_forward_time: 169.7754211426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7652130127
validation[          , genome  6104] predictions:    5773/  10000 (57.73%), best:    5773/10000 (57.73%), error:     12116.42871, best error:     12116.42871 on epoch:     1, epoch:    1/50, mu: 0.5341020823, learning_rate: 0.0017067726, weight_decay: 0.0004117398

epoch time: 1569.4392089844s, input_fired_time: 21.9620590210, output_fired_time: 9.2887897491, propagate_forward_time: 633.8776245117, propagate_backward_time: 890.4781494141, weight_update_time: 0.5738413930, other_time: 13.8325805664
epoch time: 150.8043823242s, input_fired_time: 1.7225713730, output_fired_time: 0.0000000000, propagate_forward_time: 146.5277709961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5540466309
validation[          , genome  6104] predictions:    5926/  10000 (59.26%), best:    5926/10000 (59.26%), error:     11786.67383, best error:     11786.67383 on epoch:     2, epoch:    2/50, mu: 0.5547702909, learning_rate: 0.0015592318, weight_decay: 0.0003855307

epoch time: 1647.6094970703s, input_fired_time: 22.2829055786, output_fired_time: 9.7232036591, propagate_forward_time: 664.9844360352, propagate_backward_time: 936.5451049805, weight_update_time: 0.5980993509, other_time: 14.0738525391
epoch time: 167.9437713623s, input_fired_time: 1.8687295914, output_fired_time: 0.0000000000, propagate_forward_time: 163.4144897461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6605529785
validation[          , genome  6104] predictions:    5839/  10000 (58.39%), best:    5839/10000 (58.39%), error:     11688.76074, best error:     11688.76074 on epoch:     3, epoch:    3/50, mu: 0.5745015144, learning_rate: 0.0014244451, weight_decay: 0.0003609900

epoch time: 1714.0131835938s, input_fired_time: 22.6683883667, output_fired_time: 10.0705261230, propagate_forward_time: 691.2387084961, propagate_backward_time: 975.4205322266, weight_update_time: 0.6206823587, other_time: 14.6150512695
epoch time: 122.0774307251s, input_fired_time: 1.5242602825, output_fired_time: 0.0000000000, propagate_forward_time: 118.1806869507, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3724822998
validation[          , genome  6104] predictions:    5700/  10000 (57.00%), best:    5839/10000 (58.39%), error:     12101.68750, best error:     11688.76074 on epoch:     3, epoch:    4/50, mu: 0.5933381915, learning_rate: 0.0013013099, weight_decay: 0.0003380114

epoch time: 1665.8228759766s, input_fired_time: 22.4854736328, output_fired_time: 9.7957630157, propagate_forward_time: 673.7391967773, propagate_backward_time: 945.5306396484, weight_update_time: 0.6021010876, other_time: 14.2717895508
epoch time: 133.3209838867s, input_fired_time: 1.5705767870, output_fired_time: 0.0000000000, propagate_forward_time: 129.4621734619, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2882385254
validation[          , genome  6104] predictions:    6070/  10000 (60.70%), best:    6070/10000 (60.70%), error:     11176.89453, best error:     11176.89453 on epoch:     5, epoch:    5/50, mu: 0.6113209128, learning_rate: 0.0011888191, weight_decay: 0.0003164955

epoch time: 1451.4371337891s, input_fired_time: 21.8224010468, output_fired_time: 8.6427202225, propagate_forward_time: 585.1868286133, propagate_backward_time: 821.8479614258, weight_update_time: 0.5528856516, other_time: 13.9372558594
epoch time: 126.2059860229s, input_fired_time: 1.5106599331, output_fired_time: 0.0000000000, propagate_forward_time: 122.4672012329, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2281265259
validation[          , genome  6104] predictions:    6024/  10000 (60.24%), best:    6070/10000 (60.70%), error:     11431.93848, best error:     11176.89453 on epoch:     5, epoch:    6/50, mu: 0.6284884214, learning_rate: 0.0010860524, weight_decay: 0.0002963491

epoch time: 1733.7220458984s, input_fired_time: 22.7792739868, output_fired_time: 10.1661586761, propagate_forward_time: 700.8979492188, propagate_backward_time: 985.2471313477, weight_update_time: 0.6265968680, other_time: 14.6315307617
epoch time: 142.6004791260s, input_fired_time: 1.7009953260, output_fired_time: 0.0000000000, propagate_forward_time: 138.3635406494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5359497070
validation[          , genome  6104] predictions:    6043/  10000 (60.43%), best:    6070/10000 (60.70%), error:     11189.08203, best error:     11176.89453 on epoch:     5, epoch:    7/50, mu: 0.6448776126, learning_rate: 0.0009921694, weight_decay: 0.0002774852

epoch time: 1572.5877685547s, input_fired_time: 22.0081996918, output_fired_time: 9.2641792297, propagate_forward_time: 632.3471069336, propagate_backward_time: 895.3446044922, weight_update_time: 0.5707439184, other_time: 13.6237182617
epoch time: 119.7243957520s, input_fired_time: 1.4597146511, output_fired_time: 0.0000000000, propagate_forward_time: 116.1079940796, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1566848755
validation[          , genome  6104] predictions:    6013/  10000 (60.13%), best:    6070/10000 (60.70%), error:     11676.64551, best error:     11176.89453 on epoch:     5, epoch:    8/50, mu: 0.6605237722, learning_rate: 0.0009064020, weight_decay: 0.0002598220

epoch time: 1676.3709716797s, input_fired_time: 22.6456146240, output_fired_time: 9.8701906204, propagate_forward_time: 675.8655395508, propagate_backward_time: 953.5096435547, weight_update_time: 0.6095471978, other_time: 14.4799194336
epoch time: 148.9327850342s, input_fired_time: 1.7448513508, output_fired_time: 0.0000000000, propagate_forward_time: 144.6358795166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5520477295
validation[          , genome  6104] predictions:    6197/  10000 (61.97%), best:    6197/10000 (61.97%), error:     10767.65820, best error:     10767.65820 on epoch:     9, epoch:    9/50, mu: 0.6754606366, learning_rate: 0.0008280487, weight_decay: 0.0002432832

epoch time: 1540.1263427734s, input_fired_time: 22.0846309662, output_fired_time: 9.1635751724, propagate_forward_time: 618.7332153320, propagate_backward_time: 876.1931762695, weight_update_time: 0.5736545920, other_time: 13.9517822266
epoch time: 135.5722656250s, input_fired_time: 1.6038641930, output_fired_time: 0.0000000000, propagate_forward_time: 131.6366882324, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3317108154
validation[          , genome  6104] predictions:    6172/  10000 (61.72%), best:    6197/10000 (61.97%), error:     10944.30566, best error:     10767.65820 on epoch:     9, epoch:   10/50, mu: 0.6897203326, learning_rate: 0.0007564686, weight_decay: 0.0002277972

epoch time: 1649.9503173828s, input_fired_time: 22.5999240875, output_fired_time: 9.7381181717, propagate_forward_time: 666.5356445313, propagate_backward_time: 936.5989379883, weight_update_time: 0.6029847860, other_time: 14.4776000977
epoch time: 115.6820678711s, input_fired_time: 1.4740780592, output_fired_time: 0.0000000000, propagate_forward_time: 111.9062194824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3017730713
validation[          , genome  6104] predictions:    6075/  10000 (60.75%), best:    6197/10000 (61.97%), error:     11367.01563, best error:     10767.65820 on epoch:     9, epoch:   11/50, mu: 0.7033335567, learning_rate: 0.0006910762, weight_decay: 0.0002132969

epoch time: 1740.5526123047s, input_fired_time: 22.9302101135, output_fired_time: 10.2130365372, propagate_forward_time: 706.0378417969, propagate_backward_time: 986.6469116211, weight_update_time: 0.6285240650, other_time: 14.7246704102
epoch time: 134.5266723633s, input_fired_time: 1.6338170767, output_fired_time: 0.0000000000, propagate_forward_time: 130.4279327393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4649200439
validation[          , genome  6104] predictions:    6072/  10000 (60.72%), best:    6197/10000 (61.97%), error:     11509.91016, best error:     10767.65820 on epoch:     9, epoch:   12/50, mu: 0.7163296342, learning_rate: 0.0006313365, weight_decay: 0.0001997196

epoch time: 1634.0086669922s, input_fired_time: 22.4796390533, output_fired_time: 9.6076154709, propagate_forward_time: 658.9634399414, propagate_backward_time: 928.7166748047, weight_update_time: 0.5976641774, other_time: 14.2412719727
epoch time: 158.5927124023s, input_fired_time: 1.8117641211, output_fired_time: 0.0000000000, propagate_forward_time: 154.2539215088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5270233154
validation[          , genome  6104] predictions:    6438/  10000 (64.38%), best:    6438/10000 (64.38%), error:     10220.60352, best error:     10220.60352 on epoch:    13, epoch:   13/50, mu: 0.7287365198, learning_rate: 0.0005767611, weight_decay: 0.0001870065

epoch time: 1484.3282470703s, input_fired_time: 21.9372367859, output_fired_time: 8.7998189926, propagate_forward_time: 594.3565673828, propagate_backward_time: 845.4202880859, weight_update_time: 0.5599458814, other_time: 13.8143310547
epoch time: 152.3908691406s, input_fired_time: 1.7515928745, output_fired_time: 0.0000000000, propagate_forward_time: 148.1721343994, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4671478271
validation[          , genome  6104] predictions:    6292/  10000 (62.92%), best:    6438/10000 (64.38%), error:     10560.50781, best error:     10220.60352 on epoch:    13, epoch:   14/50, mu: 0.7405809164, learning_rate: 0.0005269034, weight_decay: 0.0001751028

epoch time: 1688.1495361328s, input_fired_time: 22.7611236572, output_fired_time: 9.9161148071, propagate_forward_time: 681.0426025391, propagate_backward_time: 959.8186645508, weight_update_time: 0.6146025658, other_time: 14.6110229492
epoch time: 132.4233093262s, input_fired_time: 1.6230845451, output_fired_time: 0.0000000000, propagate_forward_time: 128.3628540039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4373779297
validation[          , genome  6104] predictions:    6233/  10000 (62.33%), best:    6438/10000 (64.38%), error:     10773.95996, best error:     10220.60352 on epoch:    13, epoch:   15/50, mu: 0.7518883944, learning_rate: 0.0004813556, weight_decay: 0.0001639567

epoch time: 1753.2663574219s, input_fired_time: 23.0077743530, output_fired_time: 10.2321357727, propagate_forward_time: 708.7968750000, propagate_backward_time: 996.5665283203, weight_update_time: 0.6307215691, other_time: 14.6629638672
epoch time: 128.7871551514s, input_fired_time: 1.5851898193, output_fired_time: 0.0000000000, propagate_forward_time: 124.8566741943, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3452911377
validation[          , genome  6104] predictions:    6356/  10000 (63.56%), best:    6438/10000 (64.38%), error:     10231.29297, best error:     10220.60352 on epoch:    13, epoch:   16/50, mu: 0.7626832128, learning_rate: 0.0004397451, weight_decay: 0.0001535201

epoch time: 1599.3322753906s, input_fired_time: 22.4772453308, output_fired_time: 9.4475936890, propagate_forward_time: 644.8723754883, propagate_backward_time: 908.1937866211, weight_update_time: 0.5908198357, other_time: 14.3411865234
epoch time: 133.2908477783s, input_fired_time: 1.6385229826, output_fired_time: 0.0000000000, propagate_forward_time: 129.1975708008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4547576904
validation[          , genome  6104] predictions:    6366/  10000 (63.66%), best:    6438/10000 (64.38%), error:     10297.07715, best error:     10220.60352 on epoch:    13, epoch:   17/50, mu: 0.7729886770, learning_rate: 0.0004017316, weight_decay: 0.0001437479

epoch time: 1734.8367919922s, input_fired_time: 22.9854412079, output_fired_time: 10.1794357300, propagate_forward_time: 703.2017822266, propagate_backward_time: 983.8329467773, weight_update_time: 0.6232904792, other_time: 14.6371459961
epoch time: 155.0407714844s, input_fired_time: 1.7987982035, output_fired_time: 0.0000000000, propagate_forward_time: 150.6862945557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5556793213
validation[          , genome  6104] predictions:    6325/  10000 (63.25%), best:    6438/10000 (64.38%), error:     10539.40918, best error:     10220.60352 on epoch:    13, epoch:   18/50, mu: 0.7828269005, learning_rate: 0.0003670042, weight_decay: 0.0001345977

epoch time: 1579.8706054688s, input_fired_time: 22.3508644104, output_fired_time: 9.3274517059, propagate_forward_time: 636.5007324219, propagate_backward_time: 897.6018676758, weight_update_time: 0.5820885897, other_time: 14.0897827148
epoch time: 122.9262771606s, input_fired_time: 1.5410792828, output_fired_time: 0.0000000000, propagate_forward_time: 119.0753021240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3098983765
validation[          , genome  6104] predictions:    6192/  10000 (61.92%), best:    6438/10000 (64.38%), error:     10919.88477, best error:     10220.60352 on epoch:    13, epoch:   19/50, mu: 0.7922191024, learning_rate: 0.0003352788, weight_decay: 0.0001260299

epoch time: 1554.4754638672s, input_fired_time: 22.2312164307, output_fired_time: 9.1868333817, propagate_forward_time: 626.3176879883, propagate_backward_time: 882.8165283203, weight_update_time: 0.5742624998, other_time: 13.9231567383
epoch time: 117.3049011230s, input_fired_time: 1.4843101501, output_fired_time: 0.0000000000, propagate_forward_time: 113.6452178955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1753692627
validation[          , genome  6104] predictions:    6579/  10000 (65.79%), best:    6579/10000 (65.79%), error:      9961.17480, best error:      9961.17480 on epoch:    20, epoch:   20/50, mu: 0.8011855483, learning_rate: 0.0003062959, weight_decay: 0.0001180076

epoch time: 1558.4693603516s, input_fired_time: 22.3085899353, output_fired_time: 9.2417020798, propagate_forward_time: 628.7664794922, propagate_backward_time: 884.1080932617, weight_update_time: 0.5756744146, other_time: 14.0444946289
epoch time: 132.1304626465s, input_fired_time: 1.6120750904, output_fired_time: 0.0000000000, propagate_forward_time: 128.1078033447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4105834961
validation[          , genome  6104] predictions:    6385/  10000 (63.85%), best:    6579/10000 (65.79%), error:     10321.46680, best error:      9961.17480 on epoch:    20, epoch:   21/50, mu: 0.8097454906, learning_rate: 0.0002798183, weight_decay: 0.0001104959

epoch time: 1554.9895019531s, input_fired_time: 22.1996555328, output_fired_time: 9.1778707504, propagate_forward_time: 628.8052368164, propagate_backward_time: 881.0910644531, weight_update_time: 0.5709181428, other_time: 13.7156372070
epoch time: 122.7513198853s, input_fired_time: 1.5140877962, output_fired_time: 0.0000000000, propagate_forward_time: 119.0369186401, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2003097534
validation[          , genome  6104] predictions:    6345/  10000 (63.45%), best:    6579/10000 (65.79%), error:     10421.78125, best error:      9961.17480 on epoch:    20, epoch:   22/50, mu: 0.8179173470, learning_rate: 0.0002556296, weight_decay: 0.0001034623

epoch time: 1723.2840576172s, input_fired_time: 23.0291328430, output_fired_time: 10.1073684692, propagate_forward_time: 697.5565185547, propagate_backward_time: 977.7576904297, weight_update_time: 0.6262965202, other_time: 14.8332519531
epoch time: 116.6901779175s, input_fired_time: 1.5043632984, output_fired_time: 0.0000000000, propagate_forward_time: 112.8858566284, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2999572754
validation[          , genome  6104] predictions:    6474/  10000 (64.74%), best:    6579/10000 (65.79%), error:     10136.33789, best error:      9961.17480 on epoch:    20, epoch:   23/50, mu: 0.8257187605, learning_rate: 0.0002335319, weight_decay: 0.0000968765

epoch time: 1691.4040527344s, input_fired_time: 22.8786849976, output_fired_time: 9.9314222336, propagate_forward_time: 683.5272216797, propagate_backward_time: 960.4735107422, weight_update_time: 0.6150116324, other_time: 14.5932617188
epoch time: 110.8339309692s, input_fired_time: 1.4146718979, output_fired_time: 0.0000000000, propagate_forward_time: 107.3339233398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0853347778
validation[          , genome  6104] predictions:    6337/  10000 (63.37%), best:    6579/10000 (65.79%), error:     10471.31055, best error:      9961.17480 on epoch:    20, epoch:   24/50, mu: 0.8331664801, learning_rate: 0.0002133444, weight_decay: 0.0000907099

epoch time: 1554.1636962891s, input_fired_time: 22.3238010406, output_fired_time: 9.2159118652, propagate_forward_time: 626.0741577148, propagate_backward_time: 882.5125122070, weight_update_time: 0.5783687234, other_time: 14.0372314453
epoch time: 154.8033905029s, input_fired_time: 1.8069094419, output_fired_time: 0.0000000000, propagate_forward_time: 150.4421539307, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5543212891
validation[          , genome  6104] predictions:    6340/  10000 (63.40%), best:    6579/10000 (65.79%), error:     10473.27441, best error:      9961.17480 on epoch:    20, epoch:   25/50, mu: 0.8402765393, learning_rate: 0.0001949020, weight_decay: 0.0000849358

epoch time: 1656.8546142578s, input_fired_time: 22.6731662750, output_fired_time: 9.7843990326, propagate_forward_time: 669.4066162109, propagate_backward_time: 940.7664184570, weight_update_time: 0.5986511707, other_time: 14.2239379883
epoch time: 172.5602569580s, input_fired_time: 1.9571267366, output_fired_time: 0.0000000000, propagate_forward_time: 167.9161376953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6869964600
validation[          , genome  6104] predictions:    6321/  10000 (63.21%), best:    6579/10000 (65.79%), error:     10504.16895, best error:      9961.17480 on epoch:    20, epoch:   26/50, mu: 0.8470642567, learning_rate: 0.0001780538, weight_decay: 0.0000795293

epoch time: 1702.7093505859s, input_fired_time: 22.8702373505, output_fired_time: 10.0331192017, propagate_forward_time: 687.0801391602, propagate_backward_time: 968.2490234375, weight_update_time: 0.6163606048, other_time: 14.4768676758
epoch time: 164.5791931152s, input_fired_time: 1.9120723009, output_fired_time: 0.0000000000, propagate_forward_time: 160.0055084229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6616058350
validation[          , genome  6104] predictions:    6525/  10000 (65.25%), best:    6525/10000 (65.25%), error:      9925.30664, best error:      9925.30664 on epoch:    27, epoch:   27/50, mu: 0.8535442948, learning_rate: 0.0001626621, weight_decay: 0.0000744669

epoch time: 1650.4722900391s, input_fired_time: 22.6401214600, output_fired_time: 9.6514530182, propagate_forward_time: 665.9362182617, propagate_backward_time: 938.0547485352, weight_update_time: 0.5992524624, other_time: 14.1896972656
epoch time: 135.8057861328s, input_fired_time: 1.6306762695, output_fired_time: 0.0000000000, propagate_forward_time: 131.8846130371, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2904968262
validation[          , genome  6104] predictions:    6531/  10000 (65.31%), best:    6525/10000 (65.25%), error:     10186.86426, best error:      9925.30664 on epoch:    27, epoch:   28/50, mu: 0.8597305417, learning_rate: 0.0001486009, weight_decay: 0.0000697267

epoch time: 1617.8937988281s, input_fired_time: 22.5702877045, output_fired_time: 9.4788188934, propagate_forward_time: 651.5303955078, propagate_backward_time: 920.0442504883, weight_update_time: 0.5955750942, other_time: 14.2700805664
epoch time: 138.3454742432s, input_fired_time: 1.6850272417, output_fired_time: 0.0000000000, propagate_forward_time: 134.2444763184, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4159698486
validation[          , genome  6104] predictions:    6517/  10000 (65.17%), best:    6517/10000 (65.17%), error:      9876.65430, best error:      9876.65430 on epoch:    29, epoch:   29/50, mu: 0.8656363487, learning_rate: 0.0001357552, weight_decay: 0.0000652883

epoch time: 1690.9387207031s, input_fired_time: 22.7988758087, output_fired_time: 9.9103927612, propagate_forward_time: 683.8664550781, propagate_backward_time: 960.1177978516, weight_update_time: 0.6083201170, other_time: 14.2452392578
epoch time: 159.9643096924s, input_fired_time: 1.8623037338, output_fired_time: 0.0000000000, propagate_forward_time: 155.5346069336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5673980713
validation[          , genome  6104] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:      9866.27148, best error:      9866.27148 on epoch:    30, epoch:   30/50, mu: 0.8712744117, learning_rate: 0.0001240199, weight_decay: 0.0000611324

epoch time: 1669.1164550781s, input_fired_time: 22.6735553741, output_fired_time: 9.7880563736, propagate_forward_time: 672.4319458008, propagate_backward_time: 950.0738525391, weight_update_time: 0.6027658582, other_time: 14.1489868164
epoch time: 146.4276275635s, input_fired_time: 1.7837485075, output_fired_time: 0.0000000000, propagate_forward_time: 142.0630645752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5808105469
validation[          , genome  6104] predictions:    6611/  10000 (66.11%), best:    6611/10000 (66.11%), error:      9695.07227, best error:      9695.07227 on epoch:    31, epoch:   31/50, mu: 0.8766568303, learning_rate: 0.0001132991, weight_decay: 0.0000572411

epoch time: 1555.0648193359s, input_fired_time: 22.4373626709, output_fired_time: 9.2341165543, propagate_forward_time: 624.1937255859, propagate_backward_time: 884.9806518555, weight_update_time: 0.5794748068, other_time: 14.2189331055
epoch time: 157.5458526611s, input_fired_time: 1.8749905825, output_fired_time: 0.0000000000, propagate_forward_time: 152.9970550537, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6738128662
validation[          , genome  6104] predictions:    6626/  10000 (66.26%), best:    6611/10000 (66.11%), error:      9761.42383, best error:      9695.07227 on epoch:    31, epoch:   32/50, mu: 0.8817952871, learning_rate: 0.0001035050, weight_decay: 0.0000535974

epoch time: 1762.0028076172s, input_fired_time: 23.1539993286, output_fired_time: 10.3242015839, propagate_forward_time: 712.8818359375, propagate_backward_time: 1001.0300292969, weight_update_time: 0.6305444837, other_time: 14.6126708984
epoch time: 117.9688415527s, input_fired_time: 1.5446643829, output_fired_time: 0.0000000000, propagate_forward_time: 114.0825500488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3416290283
validation[          , genome  6104] predictions:    6649/  10000 (66.49%), best:    6649/10000 (66.49%), error:      9634.26465, best error:      9634.26465 on epoch:    33, epoch:   33/50, mu: 0.8867007494, learning_rate: 0.0000945576, weight_decay: 0.0000501857

epoch time: 1555.5278320313s, input_fired_time: 22.2850074768, output_fired_time: 9.1731958389, propagate_forward_time: 626.1785278320, propagate_backward_time: 884.1168823242, weight_update_time: 0.5729825497, other_time: 13.7741699219
epoch time: 115.1756820679s, input_fired_time: 1.5286017656, output_fired_time: 0.0000000000, propagate_forward_time: 111.3483886719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2986907959
validation[          , genome  6104] predictions:    6572/  10000 (65.72%), best:    6649/10000 (66.49%), error:      9746.67383, best error:      9634.26465 on epoch:    33, epoch:   34/50, mu: 0.8913838267, learning_rate: 0.0000863836, weight_decay: 0.0000469912

epoch time: 1618.6104736328s, input_fired_time: 22.6119976044, output_fired_time: 9.5071144104, propagate_forward_time: 650.9234619141, propagate_backward_time: 921.3674926758, weight_update_time: 0.5924604535, other_time: 14.2005004883
epoch time: 138.1678466797s, input_fired_time: 1.6604474783, output_fired_time: 0.0000000000, propagate_forward_time: 134.2003326416, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3070678711
validation[          , genome  6104] predictions:    6609/  10000 (66.09%), best:    6649/10000 (66.49%), error:      9715.60449, best error:      9634.26465 on epoch:    33, epoch:   35/50, mu: 0.8958545923, learning_rate: 0.0000789163, weight_decay: 0.0000440000

epoch time: 1622.6844482422s, input_fired_time: 22.7308654785, output_fired_time: 9.5920705795, propagate_forward_time: 654.5230102539, propagate_backward_time: 921.4132080078, weight_update_time: 0.5994135141, other_time: 14.4253540039
epoch time: 134.2909393311s, input_fired_time: 1.6756924391, output_fired_time: 0.0000000000, propagate_forward_time: 130.1693115234, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4459381104
validation[          , genome  6104] predictions:    6697/  10000 (66.97%), best:    6697/10000 (66.97%), error:      9530.93555, best error:      9530.93555 on epoch:    36, epoch:   36/50, mu: 0.9001227021, learning_rate: 0.0000720944, weight_decay: 0.0000411992

epoch time: 1651.2224121094s, input_fired_time: 22.6767520905, output_fired_time: 9.6880159378, propagate_forward_time: 667.0166625977, propagate_backward_time: 937.6676025391, weight_update_time: 0.5991963148, other_time: 14.1734008789
epoch time: 132.5716552734s, input_fired_time: 1.6674280167, output_fired_time: 0.0000000000, propagate_forward_time: 128.4446258545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4595947266
validation[          , genome  6104] predictions:    6617/  10000 (66.17%), best:    6697/10000 (66.97%), error:      9749.89453, best error:      9530.93555 on epoch:    36, epoch:   37/50, mu: 0.9041972756, learning_rate: 0.0000658622, weight_decay: 0.0000385767

epoch time: 1612.2620849609s, input_fired_time: 22.6470031738, output_fired_time: 9.5538377762, propagate_forward_time: 651.7272949219, propagate_backward_time: 914.1647949219, weight_update_time: 0.5906959176, other_time: 14.1691894531
epoch time: 118.3002853394s, input_fired_time: 1.5024174452, output_fired_time: 0.0000000000, propagate_forward_time: 114.6731872559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1246795654
validation[          , genome  6104] predictions:    6718/  10000 (67.18%), best:    6718/10000 (67.18%), error:      9429.76563, best error:      9429.76563 on epoch:    38, epoch:   38/50, mu: 0.9080871344, learning_rate: 0.0000601688, weight_decay: 0.0000361211

epoch time: 1655.0539550781s, input_fired_time: 22.7677383423, output_fired_time: 9.7144317627, propagate_forward_time: 667.1796264648, propagate_backward_time: 941.0316162109, weight_update_time: 0.6041821241, other_time: 14.3605346680
epoch time: 147.0415039063s, input_fired_time: 1.7999688387, output_fired_time: 0.0000000000, propagate_forward_time: 142.6581878662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5833435059
validation[          , genome  6104] predictions:    6702/  10000 (67.02%), best:    6718/10000 (67.18%), error:      9509.35742, best error:      9429.76563 on epoch:    38, epoch:   39/50, mu: 0.9118006825, learning_rate: 0.0000549676, weight_decay: 0.0000338218

epoch time: 1627.0396728516s, input_fired_time: 22.6853122711, output_fired_time: 9.5698652267, propagate_forward_time: 654.9788208008, propagate_backward_time: 925.5895385742, weight_update_time: 0.5930592418, other_time: 14.2161865234
epoch time: 127.7305679321s, input_fired_time: 1.6428542137, output_fired_time: 0.0000000000, propagate_forward_time: 123.7016143799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3861007690
validation[          , genome  6104] predictions:    6691/  10000 (66.91%), best:    6718/10000 (67.18%), error:      9517.79492, best error:      9429.76563 on epoch:    38, epoch:   40/50, mu: 0.9153458476, learning_rate: 0.0000502159, weight_decay: 0.0000316689

epoch time: 1537.6848144531s, input_fired_time: 22.4500465393, output_fired_time: 9.0889873505, propagate_forward_time: 618.0857543945, propagate_backward_time: 873.7973632813, weight_update_time: 0.5801067352, other_time: 14.2626342773
epoch time: 126.4729385376s, input_fired_time: 1.5902135372, output_fired_time: 0.0000000000, propagate_forward_time: 122.6124572754, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2702713013
validation[          , genome  6104] predictions:    6733/  10000 (67.33%), best:    6718/10000 (67.18%), error:      9441.39258, best error:      9429.76563 on epoch:    38, epoch:   41/50, mu: 0.9187303185, learning_rate: 0.0000458751, weight_decay: 0.0000296531

epoch time: 1689.0844726563s, input_fired_time: 22.9677104950, output_fired_time: 9.8658590317, propagate_forward_time: 682.0391845703, propagate_backward_time: 959.6875000000, weight_update_time: 0.6141887307, other_time: 14.5242919922
epoch time: 145.6335906982s, input_fired_time: 1.7662272453, output_fired_time: 0.0000000000, propagate_forward_time: 141.3638916016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5034790039
validation[          , genome  6104] predictions:    6735/  10000 (67.35%), best:    6735/10000 (67.35%), error:      9398.12793, best error:      9398.12793 on epoch:    42, epoch:   42/50, mu: 0.9219613671, learning_rate: 0.0000419094, weight_decay: 0.0000277655

epoch time: 1565.4394531250s, input_fired_time: 22.4560070038, output_fired_time: 9.2057237625, propagate_forward_time: 632.2962646484, propagate_backward_time: 887.5195312500, weight_update_time: 0.5778883100, other_time: 13.9619140625
epoch time: 156.1580047607s, input_fired_time: 1.8688886166, output_fired_time: 0.0000000000, propagate_forward_time: 151.6761322021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6129913330
validation[          , genome  6104] predictions:    6682/  10000 (66.82%), best:    6735/10000 (67.35%), error:      9587.67480, best error:      9398.12793 on epoch:    42, epoch:   43/50, mu: 0.9250459075, learning_rate: 0.0000382866, weight_decay: 0.0000259981

epoch time: 1530.4516601563s, input_fired_time: 22.3271160126, output_fired_time: 9.0155067444, propagate_forward_time: 615.2812500000, propagate_backward_time: 869.8973388672, weight_update_time: 0.5716525912, other_time: 13.9304199219
epoch time: 128.8253173828s, input_fired_time: 1.6387647390, output_fired_time: 0.0000000000, propagate_forward_time: 124.8215255737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3650283813
validation[          , genome  6104] predictions:    6731/  10000 (67.31%), best:    6735/10000 (67.35%), error:      9477.17578, best error:      9398.12793 on epoch:    42, epoch:   44/50, mu: 0.9279906154, learning_rate: 0.0000349769, weight_decay: 0.0000243432

epoch time: 1638.0122070313s, input_fired_time: 22.6719951630, output_fired_time: 9.6128425598, propagate_forward_time: 660.7617797852, propagate_backward_time: 930.9196166992, weight_update_time: 0.5933065414, other_time: 14.0460205078
epoch time: 175.0051422119s, input_fired_time: 1.9955980778, output_fired_time: 0.0000000000, propagate_forward_time: 170.3565979004, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6529388428
validation[          , genome  6104] predictions:    6715/  10000 (67.15%), best:    6735/10000 (67.35%), error:      9441.52637, best error:      9398.12793 on epoch:    42, epoch:   45/50, mu: 0.9308018088, learning_rate: 0.0000319534, weight_decay: 0.0000227937

epoch time: 1596.3873291016s, input_fired_time: 22.6000633240, output_fired_time: 9.4025297165, propagate_forward_time: 645.2303466797, propagate_backward_time: 905.0695190430, weight_update_time: 0.5856981277, other_time: 14.0847778320
epoch time: 113.1575851440s, input_fired_time: 1.4869168997, output_fired_time: 0.0000000000, propagate_forward_time: 109.5024871826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1681823730
validation[          , genome  6104] predictions:    6750/  10000 (67.50%), best:    6750/10000 (67.50%), error:      9369.13867, best error:      9369.13867 on epoch:    46, epoch:   46/50, mu: 0.9334855676, learning_rate: 0.0000291912, weight_decay: 0.0000213427

epoch time: 1551.1997070313s, input_fired_time: 22.2933979034, output_fired_time: 9.1560420990, propagate_forward_time: 623.3365478516, propagate_backward_time: 882.7817993164, weight_update_time: 0.5675077438, other_time: 13.6318969727
epoch time: 165.0606689453s, input_fired_time: 1.9602670670, output_fired_time: 0.0000000000, propagate_forward_time: 160.3505706787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7498321533
validation[          , genome  6104] predictions:    6723/  10000 (67.23%), best:    6750/10000 (67.50%), error:      9438.68359, best error:      9369.13867 on epoch:    46, epoch:   47/50, mu: 0.9360476732, learning_rate: 0.0000266678, weight_decay: 0.0000199842

epoch time: 1686.5987548828s, input_fired_time: 22.9382781982, output_fired_time: 9.9122896194, propagate_forward_time: 680.4985351563, propagate_backward_time: 958.9188842773, weight_update_time: 0.6077908278, other_time: 14.3308715820
epoch time: 135.3182373047s, input_fired_time: 1.7006354332, output_fired_time: 0.0000000000, propagate_forward_time: 131.1633758545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4542236328
validation[          , genome  6104] predictions:    6776/  10000 (67.76%), best:    6750/10000 (67.50%), error:      9393.16309, best error:      9369.13867 on epoch:    46, epoch:   48/50, mu: 0.9384936094, learning_rate: 0.0000243625, weight_decay: 0.0000187121

epoch time: 1712.9045410156s, input_fired_time: 23.0102386475, output_fired_time: 10.0696210861, propagate_forward_time: 692.5455322266, propagate_backward_time: 972.8906860352, weight_update_time: 0.6140373945, other_time: 14.3884887695
epoch time: 161.2128601074s, input_fired_time: 1.9068546295, output_fired_time: 0.0000000000, propagate_forward_time: 156.6764526367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6295471191
validation[          , genome  6104] predictions:    6716/  10000 (67.16%), best:    6716/10000 (67.16%), error:      9364.96680, best error:      9364.96680 on epoch:    49, epoch:   49/50, mu: 0.9408286810, learning_rate: 0.0000222565, weight_decay: 0.0000175210

epoch time: 1642.4395751953s, input_fired_time: 22.6440277100, output_fired_time: 9.6595048904, propagate_forward_time: 662.0361938477, propagate_backward_time: 934.1725463867, weight_update_time: 0.5919373631, other_time: 13.9272460938
epoch time: 143.8586578369s, input_fired_time: 1.7838276625, output_fired_time: 0.0000000000, propagate_forward_time: 139.4972686768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5775604248
validation[          , genome  6104] predictions:    6756/  10000 (67.56%), best:    6756/10000 (67.56%), error:      9350.97852, best error:      9350.97852 on epoch:    50, epoch:   50/50, mu: 0.9430578947, learning_rate: 0.0000203325, weight_decay: 0.0000164057

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 715.4066162109s, input_fired_time: 8.7284555435, output_fired_time: 0.0000000000, propagate_forward_time: 694.2915649414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.3865966797
best training[          , genome  6104] predictions:   37397/  50000 (74.79%), best:    6756/10000 (67.56%), error:     38121.75000, best error:      9350.97852 on epoch:    50, epoch:   51/50, mu: 0.9451860189, learning_rate: 0.0000185749, weight_decay: 0.0000153614
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 133.1325531006s, input_fired_time: 1.6463812590, output_fired_time: 0.0000000000, propagate_forward_time: 129.1713714600, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3148040771
   testing[          , genome  6104] predictions:    6756/  10000 (67.56%), best:    6756/10000 (67.56%), error:      9350.97852, best error:      9350.97852 on epoch:    50, epoch:   51/50, mu: 0.9451860189, learning_rate: 0.0000185749, weight_decay: 0.0000153614
backpropagation finished successfully!
18:57:56 (3288): called boinc_finish(0)

</stderr_txt>
]]>