Nameexact_genome_1542727803_170_6411_0
Workunit4101338
Created20 Nov 2018, 20:17:21 UTC
Sent20 Nov 2018, 20:17:42 UTC
Report deadline26 Dec 2018, 15:11:40 UTC
Received23 Nov 2018, 6:08:47 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time16 hours 41 min 13 sec
CPU time16 hours 41 min 13 sec
Validate stateValid
Credit8,946.87
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size589.98 MB
Peak swap size588.73 MB
Peak disk usage13.33 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6411.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6411_0_r1472692231_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6411.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45897
read mu: 0.45897
read mu_delta: 0.937275
read initial_learning_rate: 0.000957571
read learning_rate: 0.000957571
read learning_rate_delta: 0.904911
read initial_weight_decay: 0.000190137
read weight_decay: 0.000190137
read weight_decay_delta: 0.923964
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.0498651
read input_dropout_probability: 0.00157152
read hidden_dropout_probability: 0.0777743
read velocity_reset: 1401
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6411
read normal distribution: '1 0 0'
generator_str: '1443297185'
read generator: 1443297185
read generated_by_map:
1 crossover 1
reading 137 nodes.
reading 494 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1667995430
post shuffle 1: 1990415820
epoch time: 2158.80395507813s, input_fired_time: 32.2340507507324, output_fired_time: 16.250129699707, propagate_forward_time: 861.712463378906, propagate_backward_time: 1227.94018554688, weight_update_time: 0.602795481681824, other_time: 20.6669921875
epoch time: 207.688461303711s, input_fired_time: 2.9539098739624, output_fired_time: 0, propagate_forward_time: 200.704635620117, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.02992248535156
validation[          , genome  6411] predictions:    6253/  10000 (62.53%), best:    6253/10000 (62.53%), error:     10798.73926, best error:     10798.73926 on epoch:     0, epoch:    0/25, mu: 0.4589701295, learning_rate: 0.0009575707, weight_decay: 0.0001901371

epoch time: 2097.4536132813s, input_fired_time: 31.5989589691, output_fired_time: 15.7193918228, propagate_forward_time: 833.4790649414, propagate_backward_time: 1196.6693115234, weight_update_time: 0.5819200277, other_time: 19.9869384766
epoch time: 122.1084518433s, input_fired_time: 1.9849703312, output_fired_time: 0.0000000000, propagate_forward_time: 116.8819808960, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2415008545
validation[          , genome  6411] predictions:    6258/  10000 (62.58%), best:    6253/10000 (62.53%), error:     10819.20703, best error:     10798.73926 on epoch:     0, epoch:    1/25, mu: 0.4922789931, learning_rate: 0.0008665161, weight_decay: 0.0001756799

epoch time: 1842.1014404297s, input_fired_time: 30.1918640137, output_fired_time: 13.8739633560, propagate_forward_time: 726.5360717773, propagate_backward_time: 1052.2316894531, weight_update_time: 0.5252127051, other_time: 19.2678222656
epoch time: 205.9329681396s, input_fired_time: 2.9426345825, output_fired_time: 0.0000000000, propagate_forward_time: 198.9155731201, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0747528076
validation[          , genome  6411] predictions:    6363/  10000 (63.63%), best:    6363/10000 (63.63%), error:     10587.61719, best error:     10587.61719 on epoch:     2, epoch:    2/25, mu: 0.5234985352, learning_rate: 0.0007841198, weight_decay: 0.0001623219

epoch time: 2128.9238281250s, input_fired_time: 31.8000125885, output_fired_time: 15.9142112732, propagate_forward_time: 849.5728759766, propagate_backward_time: 1211.5592041016, weight_update_time: 0.5852500200, other_time: 20.0773925781
epoch time: 138.2780303955s, input_fired_time: 2.1593427658, output_fired_time: 0.0000000000, propagate_forward_time: 132.7808837891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3377990723
validation[          , genome  6411] predictions:    6415/  10000 (64.15%), best:    6415/10000 (64.15%), error:     10437.07227, best error:     10437.07227 on epoch:     3, epoch:    3/25, mu: 0.5527598262, learning_rate: 0.0007095584, weight_decay: 0.0001499797

epoch time: 2114.8105468750s, input_fired_time: 31.7912902832, output_fired_time: 15.9486179352, propagate_forward_time: 844.7873535156, propagate_backward_time: 1202.2834472656, weight_update_time: 0.5820159912, other_time: 19.9997558594
epoch time: 170.0564880371s, input_fired_time: 2.4765498638, output_fired_time: 0.0000000000, propagate_forward_time: 164.0367736816, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5431671143
validation[          , genome  6411] predictions:    6368/  10000 (63.68%), best:    6415/10000 (64.15%), error:     10563.49219, best error:     10437.07227 on epoch:     3, epoch:    4/25, mu: 0.5801857114, learning_rate: 0.0006420871, weight_decay: 0.0001385759

epoch time: 2060.8520507813s, input_fired_time: 31.9061069489, output_fired_time: 15.6693429947, propagate_forward_time: 819.4064331055, propagate_backward_time: 1172.7702636719, weight_update_time: 0.5943460464, other_time: 21.0998535156
epoch time: 146.7413482666s, input_fired_time: 2.3212897778, output_fired_time: 0.0000000000, propagate_forward_time: 140.7440490723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6760101318
validation[          , genome  6411] predictions:    6259/  10000 (62.59%), best:    6415/10000 (64.15%), error:     10747.44043, best error:     10437.07227 on epoch:     3, epoch:    5/25, mu: 0.6058912873, learning_rate: 0.0005810315, weight_decay: 0.0001280392

epoch time: 2121.1501464844s, input_fired_time: 31.8761901855, output_fired_time: 15.9466943741, propagate_forward_time: 847.7185058594, propagate_backward_time: 1205.4230957031, weight_update_time: 0.5863301754, other_time: 20.1855468750
epoch time: 195.5714721680s, input_fired_time: 2.7471976280, output_fired_time: 0.0000000000, propagate_forward_time: 189.1042022705, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7200775146
validation[          , genome  6411] predictions:    6242/  10000 (62.42%), best:    6415/10000 (64.15%), error:     10968.90527, best error:     10437.07227 on epoch:     3, epoch:    6/25, mu: 0.6299844980, learning_rate: 0.0005257817, weight_decay: 0.0001183037

epoch time: 2242.2661132813s, input_fired_time: 32.5154685974, output_fired_time: 16.8917694092, propagate_forward_time: 897.9127197266, propagate_backward_time: 1274.5148925781, weight_update_time: 0.6098769307, other_time: 20.4312744141
epoch time: 125.7323989868s, input_fired_time: 2.0353810787, output_fired_time: 0.0000000000, propagate_forward_time: 120.3171234131, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3798980713
validation[          , genome  6411] predictions:    6331/  10000 (63.31%), best:    6415/10000 (64.15%), error:     10537.35059, best error:     10437.07227 on epoch:     3, epoch:    7/25, mu: 0.6525664926, learning_rate: 0.0004757855, weight_decay: 0.0001093084

epoch time: 2204.1870117188s, input_fired_time: 32.3462142944, output_fired_time: 16.5479145050, propagate_forward_time: 880.9067382813, propagate_backward_time: 1253.8961181641, weight_update_time: 0.6028550863, other_time: 20.4901123047
epoch time: 186.5209350586s, input_fired_time: 2.6614630222, output_fired_time: 0.0000000000, propagate_forward_time: 180.1748352051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6846313477
validation[          , genome  6411] predictions:    6370/  10000 (63.70%), best:    6415/10000 (64.15%), error:     10446.96387, best error:     10437.07227 on epoch:     3, epoch:    8/25, mu: 0.6737319827, learning_rate: 0.0004305435, weight_decay: 0.0001009970

epoch time: 2147.8701171875s, input_fired_time: 32.0158843994, output_fired_time: 16.1749782562, propagate_forward_time: 857.6834716797, propagate_backward_time: 1221.5285644531, weight_update_time: 0.5934180617, other_time: 20.4671630859
epoch time: 181.8691253662s, input_fired_time: 2.6285843849, output_fired_time: 0.0000000000, propagate_forward_time: 175.5623321533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6782073975
validation[          , genome  6411] predictions:    6489/  10000 (64.89%), best:    6489/10000 (64.89%), error:     10265.61426, best error:     10265.61426 on epoch:     9, epoch:    9/25, mu: 0.6935698986, learning_rate: 0.0003896034, weight_decay: 0.0000933177

epoch time: 2259.1503906250s, input_fired_time: 32.8597297668, output_fired_time: 17.0524482727, propagate_forward_time: 904.4507446289, propagate_backward_time: 1283.6977539063, weight_update_time: 0.6251858473, other_time: 21.0898437500
epoch time: 177.0478668213s, input_fired_time: 2.6281077862, output_fired_time: 0.0000000000, propagate_forward_time: 170.6243286133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7954254150
validation[          , genome  6411] predictions:    6358/  10000 (63.58%), best:    6489/10000 (64.89%), error:     10549.86621, best error:     10265.61426 on epoch:     9, epoch:   10/25, mu: 0.7121635079, learning_rate: 0.0003525564, weight_decay: 0.0000862222

epoch time: 2296.4013671875s, input_fired_time: 33.0954627991, output_fired_time: 17.3816909790, propagate_forward_time: 918.7586059570, propagate_backward_time: 1306.0061035156, weight_update_time: 0.6319344640, other_time: 21.1596679688
epoch time: 119.4814147949s, input_fired_time: 1.9776197672, output_fired_time: 0.0000000000, propagate_forward_time: 114.1529083252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3508834839
validation[          , genome  6411] predictions:    6394/  10000 (63.94%), best:    6489/10000 (64.89%), error:     10401.15918, best error:     10265.61426 on epoch:     9, epoch:   11/25, mu: 0.7295908332, learning_rate: 0.0003190321, weight_decay: 0.0000796662

epoch time: 2086.9936523438s, input_fired_time: 31.6945552826, output_fired_time: 15.7144699097, propagate_forward_time: 831.7755126953, propagate_backward_time: 1187.6102294922, weight_update_time: 0.5795918107, other_time: 20.1988525391
epoch time: 126.7825164795s, input_fired_time: 2.0568714142, output_fired_time: 0.0000000000, propagate_forward_time: 121.3813552856, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3442916870
validation[          , genome  6411] predictions:    6467/  10000 (64.67%), best:    6489/10000 (64.89%), error:     10288.85449, best error:     10265.61426 on epoch:     9, epoch:   12/25, mu: 0.7459250093, learning_rate: 0.0002886956, weight_decay: 0.0000736088

epoch time: 2217.7429199219s, input_fired_time: 32.4060935974, output_fired_time: 16.6791515350, propagate_forward_time: 884.9825439453, propagate_backward_time: 1263.1737060547, weight_update_time: 0.6057251692, other_time: 20.5014648438
epoch time: 137.7956390381s, input_fired_time: 2.1540029049, output_fired_time: 0.0000000000, propagate_forward_time: 132.2882537842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3533782959
validation[          , genome  6411] predictions:    6464/  10000 (64.64%), best:    6489/10000 (64.89%), error:     10330.48730, best error:     10265.61426 on epoch:     9, epoch:   13/25, mu: 0.7612346411, learning_rate: 0.0002612437, weight_decay: 0.0000680119

epoch time: 1958.5771484375s, input_fired_time: 30.8240547180, output_fired_time: 14.6173439026, propagate_forward_time: 775.6476440430, propagate_backward_time: 1117.9643554688, weight_update_time: 0.5506234765, other_time: 19.5236816406
epoch time: 186.4280242920s, input_fired_time: 2.6529436111, output_fired_time: 0.0000000000, propagate_forward_time: 180.1133880615, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6616973877
validation[          , genome  6411] predictions:    6426/  10000 (64.26%), best:    6489/10000 (64.89%), error:     10368.33887, best error:     10265.61426 on epoch:     9, epoch:   14/25, mu: 0.7755839229, learning_rate: 0.0002364023, weight_decay: 0.0000628406

epoch time: 2166.0766601563s, input_fired_time: 32.0705909729, output_fired_time: 16.2897891998, propagate_forward_time: 862.4177856445, propagate_backward_time: 1235.0986328125, weight_update_time: 0.5955770612, other_time: 20.1999511719
epoch time: 161.9896545410s, input_fired_time: 2.4078335762, output_fired_time: 0.0000000000, propagate_forward_time: 156.0888061523, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4930114746
validation[          , genome  6411] predictions:    6439/  10000 (64.39%), best:    6489/10000 (64.89%), error:     10442.32422, best error:     10265.61426 on epoch:     9, epoch:   15/25, mu: 0.7890331745, learning_rate: 0.0002139230, weight_decay: 0.0000580624

epoch time: 1937.7030029297s, input_fired_time: 30.8624668121, output_fired_time: 14.5920953751, propagate_forward_time: 768.8359375000, propagate_backward_time: 1103.6523437500, weight_update_time: 0.5516170859, other_time: 19.7602539063
epoch time: 138.7619323730s, input_fired_time: 2.2568149567, output_fired_time: 0.0000000000, propagate_forward_time: 132.8750152588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6300964355
validation[          , genome  6411] predictions:    6537/  10000 (65.37%), best:    6537/10000 (65.37%), error:     10069.68750, best error:     10069.68750 on epoch:    16, epoch:   16/25, mu: 0.8016388416, learning_rate: 0.0001935812, weight_decay: 0.0000536476

epoch time: 2035.8764648438s, input_fired_time: 31.3922557831, output_fired_time: 15.2045917511, propagate_forward_time: 811.7917480469, propagate_backward_time: 1157.6066894531, weight_update_time: 0.5701038241, other_time: 19.8812255859
epoch time: 205.8181762695s, input_fired_time: 2.8734524250, output_fired_time: 0.0000000000, propagate_forward_time: 199.1596527100, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7850646973
validation[          , genome  6411] predictions:    6442/  10000 (64.42%), best:    6537/10000 (65.37%), error:     10289.16699, best error:     10069.68750 on epoch:    16, epoch:   17/25, mu: 0.8134537935, learning_rate: 0.0001751737, weight_decay: 0.0000495685

epoch time: 2275.5395507813s, input_fired_time: 33.1962432861, output_fired_time: 17.2059898376, propagate_forward_time: 911.0744018555, propagate_backward_time: 1292.4802246094, weight_update_time: 0.6371635795, other_time: 21.5825195313
epoch time: 209.2719573975s, input_fired_time: 2.9840009212, output_fired_time: 0.0000000000, propagate_forward_time: 202.2342987061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0536651611
validation[          , genome  6411] predictions:    6460/  10000 (64.60%), best:    6537/10000 (65.37%), error:     10141.23926, best error:     10069.68750 on epoch:    16, epoch:   18/25, mu: 0.8245276809, learning_rate: 0.0001585166, weight_decay: 0.0000457995

epoch time: 1989.2519531250s, input_fired_time: 31.0865592957, output_fired_time: 14.9791107178, propagate_forward_time: 789.2938842773, propagate_backward_time: 1134.3355712891, weight_update_time: 0.5527958870, other_time: 19.5567626953
epoch time: 168.4298553467s, input_fired_time: 2.4759182930, output_fired_time: 0.0000000000, propagate_forward_time: 162.4300689697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5238647461
validation[          , genome  6411] predictions:    6530/  10000 (65.30%), best:    6537/10000 (65.37%), error:     10207.02344, best error:     10069.68750 on epoch:    16, epoch:   19/25, mu: 0.8349069357, learning_rate: 0.0001434434, weight_decay: 0.0000423171

epoch time: 2100.7836914063s, input_fired_time: 31.7150192261, output_fired_time: 15.7135562897, propagate_forward_time: 837.2630004883, propagate_backward_time: 1196.1195068359, weight_update_time: 0.5810964704, other_time: 19.9725341797
epoch time: 153.4051361084s, input_fired_time: 2.3443975449, output_fired_time: 0.0000000000, propagate_forward_time: 147.6217498779, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4389953613
validation[          , genome  6411] predictions:    6622/  10000 (66.22%), best:    6622/10000 (66.22%), error:      9964.54199, best error:      9964.54199 on epoch:    20, epoch:   20/25, mu: 0.8446351290, learning_rate: 0.0001298035, weight_decay: 0.0000390995

epoch time: 2148.5808105469s, input_fired_time: 32.0451202393, output_fired_time: 16.0913944244, propagate_forward_time: 857.6552734375, propagate_backward_time: 1222.6741943359, weight_update_time: 0.5894635916, other_time: 20.1148681641
epoch time: 164.0362701416s, input_fired_time: 2.4480693340, output_fired_time: 0.0000000000, propagate_forward_time: 158.0802154541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5079803467
validation[          , genome  6411] predictions:    6516/  10000 (65.16%), best:    6622/10000 (66.22%), error:     10052.98926, best error:      9964.54199 on epoch:    20, epoch:   21/25, mu: 0.8537531495, learning_rate: 0.0001174606, weight_decay: 0.0000361266

epoch time: 2254.5097656250s, input_fired_time: 32.6632575989, output_fired_time: 16.9255084991, propagate_forward_time: 901.8954467773, propagate_backward_time: 1282.6014404297, weight_update_time: 0.6128833890, other_time: 20.4239501953
epoch time: 158.9658508301s, input_fired_time: 2.4000122547, output_fired_time: 0.0000000000, propagate_forward_time: 153.0388793945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5269622803
validation[          , genome  6411] predictions:    6544/  10000 (65.44%), best:    6622/10000 (66.22%), error:     10029.66504, best error:      9964.54199 on epoch:    20, epoch:   22/25, mu: 0.8622992635, learning_rate: 0.0001062913, weight_decay: 0.0000333796

epoch time: 1838.7296142578s, input_fired_time: 30.9132213593, output_fired_time: 14.0814123154, propagate_forward_time: 721.2832641602, propagate_backward_time: 1051.2475585938, weight_update_time: 0.5487155318, other_time: 21.2041015625
epoch time: 141.2556457520s, input_fired_time: 2.2969470024, output_fired_time: 0.0000000000, propagate_forward_time: 135.2230834961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7356109619
validation[          , genome  6411] predictions:    6590/  10000 (65.90%), best:    6590/10000 (65.90%), error:      9959.56934, best error:      9959.56934 on epoch:    23, epoch:   23/25, mu: 0.8703092933, learning_rate: 0.0000961842, weight_decay: 0.0000308416

epoch time: 2122.9509277344s, input_fired_time: 32.5907630920, output_fired_time: 16.2495956421, propagate_forward_time: 840.9481201172, propagate_backward_time: 1210.6457519531, weight_update_time: 0.6123022437, other_time: 22.5167236328
epoch time: 202.9554595947s, input_fired_time: 2.8656044006, output_fired_time: 0.0000000000, propagate_forward_time: 196.2572937012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8325653076
validation[          , genome  6411] predictions:    6588/  10000 (65.88%), best:    6590/10000 (65.90%), error:      9999.17676, best error:      9959.56934 on epoch:    23, epoch:   24/25, mu: 0.8778169155, learning_rate: 0.0000870381, weight_decay: 0.0000284965

epoch time: 1950.1309814453s, input_fired_time: 31.1949634552, output_fired_time: 14.7057209015, propagate_forward_time: 768.6729125977, propagate_backward_time: 1115.0821533203, weight_update_time: 0.5633375645, other_time: 20.4752197266
epoch time: 183.6660156250s, input_fired_time: 2.7403323650, output_fired_time: 0.0000000000, propagate_forward_time: 176.8879852295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0377044678
validation[          , genome  6411] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:      9897.30371, best error:      9897.30371 on epoch:    25, epoch:   25/25, mu: 0.8848536015, learning_rate: 0.0000787617, weight_decay: 0.0000263298

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 822.1071777344s, input_fired_time: 12.6244087219, output_fired_time: 0.0000000000, propagate_forward_time: 790.4161376953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.0666503906
best training[          , genome  6411] predictions:   37143/  50000 (74.29%), best:    6571/10000 (65.71%), error:     39612.73047, best error:      9897.30371 on epoch:    25, epoch:   26/25, mu: 0.8914489150, learning_rate: 0.0000712723, weight_decay: 0.0000243278
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 159.1078948975s, input_fired_time: 2.4721770287, output_fired_time: 0.0000000000, propagate_forward_time: 152.9371032715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6986083984
   testing[          , genome  6411] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:      9897.30371, best error:      9897.30371 on epoch:    25, epoch:   26/25, mu: 0.8914489150, learning_rate: 0.0000712723, weight_decay: 0.0000243278
backpropagation finished successfully!
00:06:19 (2624): called boinc_finish(0)

</stderr_txt>
]]>