Nameexact_genome_1542727803_171_6421_0
Workunit4101340
Created20 Nov 2018, 20:17:51 UTC
Sent20 Nov 2018, 20:17:56 UTC
Report deadline23 Dec 2018, 21:43:11 UTC
Received23 Nov 2018, 7:34:04 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time15 hours 40 min 13 sec
CPU time15 hours 40 min 13 sec
Validate stateValid
Credit8,264.80
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size588.92 MB
Peak swap size587.70 MB
Peak disk usage14.03 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6421.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6421_0_r116851531_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6421.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.47862
read mu: 0.47862
read mu_delta: 0.957956
read initial_learning_rate: 0.000649875
read learning_rate: 0.000649875
read learning_rate_delta: 0.930359
read initial_weight_decay: 0.000224757
read weight_decay: 0.000224757
read weight_decay_delta: 0.930701
read batch_size: 121
read epsilon: 1e-07
read alpha: 0.10145
read input_dropout_probability: 0.00127936
read hidden_dropout_probability: 0.118332
read velocity_reset: 1364
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6421
read normal distribution: '1 0 0'
generator_str: '531675637'
read generator: 531675637
read generated_by_map:
1 enable_edge 1
reading 123 nodes.
reading 481 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 192975892
post shuffle 1: 1334108851
epoch time: 1981.64636230469s, input_fired_time: 28.1498756408691, output_fired_time: 13.835506439209, propagate_forward_time: 790.071533203125, propagate_backward_time: 1130.83642578125, weight_update_time: 0.547605037689209, other_time: 18.7530517578125
epoch time: 211.422836303711s, input_fired_time: 2.65207934379578, output_fired_time: 0, propagate_forward_time: 204.975769042969, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.79498291015625
validation[          , genome  6421] predictions:    6026/  10000 (60.26%), best:    6026/10000 (60.26%), error:     11401.31738, best error:     11401.31738 on epoch:     0, epoch:    0/25, mu: 0.4786201417, learning_rate: 0.0006498750, weight_decay: 0.0002247567

epoch time: 2113.0065917969s, input_fired_time: 28.8018035889, output_fired_time: 14.5885133743, propagate_forward_time: 847.6020507813, propagate_backward_time: 1203.0953369141, weight_update_time: 0.5741342902, other_time: 18.9188232422
epoch time: 151.9852142334s, input_fired_time: 2.0260944366, output_fired_time: 0.0000000000, propagate_forward_time: 146.7615509033, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1975708008
validation[          , genome  6421] predictions:    5990/  10000 (59.90%), best:    6026/10000 (60.26%), error:     11524.66309, best error:     11401.31738 on epoch:     0, epoch:    1/25, mu: 0.5001205206, learning_rate: 0.0006046170, weight_decay: 0.0002091813

epoch time: 2075.7075195313s, input_fired_time: 28.5683555603, output_fired_time: 14.4511289597, propagate_forward_time: 830.5976562500, propagate_backward_time: 1183.3031005859, weight_update_time: 0.5639487505, other_time: 18.7872314453
epoch time: 157.4856414795s, input_fired_time: 2.1547198296, output_fired_time: 0.0000000000, propagate_forward_time: 151.8439788818, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4869384766
validation[          , genome  6421] predictions:    6236/  10000 (62.36%), best:    6236/10000 (62.36%), error:     11088.44336, best error:     11088.44336 on epoch:     2, epoch:    2/25, mu: 0.5207169056, learning_rate: 0.0005625108, weight_decay: 0.0001946852

epoch time: 2172.3181152344s, input_fired_time: 29.3224086761, output_fired_time: 15.1422901154, propagate_forward_time: 872.5987548828, propagate_backward_time: 1235.5172119141, weight_update_time: 0.5989507437, other_time: 19.7373046875
epoch time: 130.2844238281s, input_fired_time: 1.8160774708, output_fired_time: 0.0000000000, propagate_forward_time: 125.3328933716, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1354598999
validation[          , genome  6421] predictions:    6092/  10000 (60.92%), best:    6236/10000 (62.36%), error:     11413.06836, best error:     11088.44336 on epoch:     2, epoch:    3/25, mu: 0.5404473543, learning_rate: 0.0005233369, weight_decay: 0.0001811937

epoch time: 2039.0577392578s, input_fired_time: 28.5752773285, output_fired_time: 14.2867546082, propagate_forward_time: 814.8787841797, propagate_backward_time: 1162.0583496094, weight_update_time: 0.5697403550, other_time: 19.2585449219
epoch time: 191.7464599609s, input_fired_time: 2.4509890079, output_fired_time: 0.0000000000, propagate_forward_time: 185.6191101074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6763610840
validation[          , genome  6421] predictions:    5669/  10000 (56.69%), best:    6236/10000 (62.36%), error:     12726.11133, best error:     11088.44336 on epoch:     2, epoch:    4/25, mu: 0.5593482852, learning_rate: 0.0004868912, weight_decay: 0.0001686371

epoch time: 1945.3526611328s, input_fired_time: 28.2267169952, output_fired_time: 13.6375350952, propagate_forward_time: 775.0841064453, propagate_backward_time: 1108.9624023438, weight_update_time: 0.5585414767, other_time: 19.4418945313
epoch time: 141.6889801025s, input_fired_time: 1.9758391380, output_fired_time: 0.0000000000, propagate_forward_time: 136.4040985107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3090362549
validation[          , genome  6421] predictions:    5929/  10000 (59.29%), best:    6236/10000 (62.36%), error:     11673.73535, best error:     11088.44336 on epoch:     2, epoch:    5/25, mu: 0.5774545074, learning_rate: 0.0004529835, weight_decay: 0.0001569507

epoch time: 1976.8917236328s, input_fired_time: 28.4149551392, output_fired_time: 13.8403263092, propagate_forward_time: 789.1865844727, propagate_backward_time: 1125.9199218750, weight_update_time: 0.5668377876, other_time: 19.5300292969
epoch time: 153.0080871582s, input_fired_time: 2.0945699215, output_fired_time: 0.0000000000, propagate_forward_time: 147.5226593018, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3908538818
validation[          , genome  6421] predictions:    6132/  10000 (61.32%), best:    6236/10000 (62.36%), error:     11205.07617, best error:     11088.44336 on epoch:     2, epoch:    6/25, mu: 0.5947995186, learning_rate: 0.0004214372, weight_decay: 0.0001460741

epoch time: 1906.9404296875s, input_fired_time: 27.7370567322, output_fired_time: 13.3067998886, propagate_forward_time: 761.5079956055, propagate_backward_time: 1085.9285888672, weight_update_time: 0.5283834934, other_time: 18.4600830078
epoch time: 202.1985015869s, input_fired_time: 2.5237042904, output_fired_time: 0.0000000000, propagate_forward_time: 196.1131744385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5616302490
validation[          , genome  6421] predictions:    6007/  10000 (60.07%), best:    6236/10000 (62.36%), error:     11514.96777, best error:     11088.44336 on epoch:     2, epoch:    7/25, mu: 0.6114152670, learning_rate: 0.0003920879, weight_decay: 0.0001359513

epoch time: 2007.0720214844s, input_fired_time: 28.6344451904, output_fired_time: 14.1370496750, propagate_forward_time: 802.8963623047, propagate_backward_time: 1141.8360595703, weight_update_time: 0.5694370270, other_time: 19.5681152344
epoch time: 197.1278686523s, input_fired_time: 2.5200159550, output_fired_time: 0.0000000000, propagate_forward_time: 190.8919219971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7159271240
validation[          , genome  6421] predictions:    6263/  10000 (62.63%), best:    6263/10000 (62.63%), error:     11026.49121, best error:     11026.49121 on epoch:     8, epoch:    8/25, mu: 0.6273323894, learning_rate: 0.0003647824, weight_decay: 0.0001265300

epoch time: 1977.7530517578s, input_fired_time: 28.4458103180, output_fired_time: 13.8958292007, propagate_forward_time: 789.3448486328, propagate_backward_time: 1126.4914550781, weight_update_time: 0.5655657053, other_time: 19.5750732422
epoch time: 142.3495635986s, input_fired_time: 2.0001432896, output_fired_time: 0.0000000000, propagate_forward_time: 137.0247039795, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3247222900
validation[          , genome  6421] predictions:    6294/  10000 (62.94%), best:    6294/10000 (62.94%), error:     10889.36035, best error:     10889.36035 on epoch:     9, epoch:    9/25, mu: 0.6425803304, learning_rate: 0.0003393786, weight_decay: 0.0001177616

epoch time: 1866.1336669922s, input_fired_time: 27.7296123505, output_fired_time: 13.0299158096, propagate_forward_time: 744.1485595703, propagate_backward_time: 1062.3742675781, weight_update_time: 0.5336210728, other_time: 18.8513183594
epoch time: 140.8208923340s, input_fired_time: 1.9177644253, output_fired_time: 0.0000000000, propagate_forward_time: 135.7888336182, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1142883301
validation[          , genome  6421] predictions:    6009/  10000 (60.09%), best:    6294/10000 (62.94%), error:     11390.04980, best error:     10889.36035 on epoch:     9, epoch:   10/25, mu: 0.6571871638, learning_rate: 0.0003157439, weight_decay: 0.0001096008

epoch time: 2071.3718261719s, input_fired_time: 28.6903190613, output_fired_time: 14.4684820175, propagate_forward_time: 828.5099487305, propagate_backward_time: 1180.6257324219, weight_update_time: 0.5688397288, other_time: 19.0773925781
epoch time: 180.9137878418s, input_fired_time: 2.3842432499, output_fired_time: 0.0000000000, propagate_forward_time: 174.8851165771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6444244385
validation[          , genome  6421] predictions:    6281/  10000 (62.81%), best:    6294/10000 (62.94%), error:     10934.87598, best error:     10889.36035 on epoch:     9, epoch:   11/25, mu: 0.6711798906, learning_rate: 0.0002937551, weight_decay: 0.0001020055

epoch time: 1912.3874511719s, input_fired_time: 27.9043312073, output_fired_time: 13.3029546738, propagate_forward_time: 763.5618896484, propagate_backward_time: 1088.8067626953, weight_update_time: 0.5405544639, other_time: 18.8115234375
epoch time: 195.8069458008s, input_fired_time: 2.5347537994, output_fired_time: 0.0000000000, propagate_forward_time: 189.5577697754, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7144165039
validation[          , genome  6421] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     10698.66113, best error:     10698.66113 on epoch:    12, epoch:   12/25, mu: 0.6845843196, learning_rate: 0.0002732977, weight_decay: 0.0000949367

epoch time: 1821.1726074219s, input_fired_time: 27.3178215027, output_fired_time: 12.6899528503, propagate_forward_time: 720.8759765625, propagate_backward_time: 1041.9924316406, weight_update_time: 0.5138143897, other_time: 18.2963867188
epoch time: 189.2676086426s, input_fired_time: 2.3999941349, output_fired_time: 0.0000000000, propagate_forward_time: 183.4285583496, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4390563965
validation[          , genome  6421] predictions:    6246/  10000 (62.46%), best:    6340/10000 (63.40%), error:     10780.72461, best error:     10698.66113 on epoch:    12, epoch:   13/25, mu: 0.6974251866, learning_rate: 0.0002542649, weight_decay: 0.0000883576

epoch time: 2209.8603515625s, input_fired_time: 29.8213920593, output_fired_time: 15.4527978897, propagate_forward_time: 888.8196411133, propagate_backward_time: 1255.3273925781, weight_update_time: 0.6148259640, other_time: 20.4392089844
epoch time: 207.9924621582s, input_fired_time: 2.6428039074, output_fired_time: 0.0000000000, propagate_forward_time: 201.5545043945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7951507568
validation[          , genome  6421] predictions:    6299/  10000 (62.99%), best:    6340/10000 (63.40%), error:     10716.96973, best error:     10698.66113 on epoch:    12, epoch:   14/25, mu: 0.7097261548, learning_rate: 0.0002365577, weight_decay: 0.0000822345

epoch time: 2002.2176513672s, input_fired_time: 28.6967945099, output_fired_time: 14.0926113129, propagate_forward_time: 798.2432250977, propagate_backward_time: 1141.2003173828, weight_update_time: 0.5762197375, other_time: 19.9847412109
epoch time: 160.5762786865s, input_fired_time: 2.1878187656, output_fired_time: 0.0000000000, propagate_forward_time: 154.8631134033, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5253448486
validation[          , genome  6421] predictions:    6195/  10000 (61.95%), best:    6340/10000 (63.40%), error:     10890.59375, best error:     10698.66113 on epoch:    12, epoch:   15/25, mu: 0.7215099335, learning_rate: 0.0002200835, weight_decay: 0.0000765357

epoch time: 2086.5075683594s, input_fired_time: 29.2028331757, output_fired_time: 14.6329536438, propagate_forward_time: 837.2543945313, propagate_backward_time: 1185.2595214844, weight_update_time: 0.5884847641, other_time: 20.1578369141
epoch time: 123.9401931763s, input_fired_time: 1.8384708166, output_fired_time: 0.0000000000, propagate_forward_time: 118.7655868530, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3361358643
validation[          , genome  6421] predictions:    6131/  10000 (61.31%), best:    6340/10000 (63.40%), error:     11125.82422, best error:     10698.66113 on epoch:    12, epoch:   16/25, mu: 0.7327982783, learning_rate: 0.0002047567, weight_decay: 0.0000712319

epoch time: 1697.3862304688s, input_fired_time: 27.0469379425, output_fired_time: 11.9733734131, propagate_forward_time: 668.8577270508, propagate_backward_time: 970.4468383789, weight_update_time: 0.5054968596, other_time: 19.0612792969
epoch time: 199.1168365479s, input_fired_time: 2.5617792606, output_fired_time: 0.0000000000, propagate_forward_time: 192.8411102295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7139434814
validation[          , genome  6421] predictions:    6342/  10000 (63.42%), best:    6342/10000 (63.42%), error:     10635.52051, best error:     10635.52051 on epoch:    17, epoch:   17/25, mu: 0.7436119914, learning_rate: 0.0001904972, weight_decay: 0.0000662956

epoch time: 1917.1739501953s, input_fired_time: 27.7761383057, output_fired_time: 13.3879985809, propagate_forward_time: 762.4982910156, propagate_backward_time: 1095.2100830078, weight_update_time: 0.5317364931, other_time: 18.3015136719
epoch time: 181.0159759521s, input_fired_time: 2.3259832859, output_fired_time: 0.0000000000, propagate_forward_time: 175.3038635254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3861236572
validation[          , genome  6421] predictions:    6314/  10000 (63.14%), best:    6314/10000 (63.14%), error:     10531.61230, best error:     10531.61230 on epoch:    18, epoch:   18/25, mu: 0.7539710999, learning_rate: 0.0001772307, weight_decay: 0.0000617013

epoch time: 1994.1577148438s, input_fired_time: 28.4812164307, output_fired_time: 13.9388704300, propagate_forward_time: 794.0679321289, propagate_backward_time: 1138.3652343750, weight_update_time: 0.5594914556, other_time: 19.3044433594
epoch time: 134.4772796631s, input_fired_time: 1.9557704926, output_fired_time: 0.0000000000, propagate_forward_time: 129.1249389648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3965759277
validation[          , genome  6421] predictions:    6404/  10000 (64.04%), best:    6404/10000 (64.04%), error:     10479.01367, best error:     10479.01367 on epoch:    19, epoch:   19/25, mu: 0.7638946772, learning_rate: 0.0001648882, weight_decay: 0.0000574255

epoch time: 1950.1665039063s, input_fired_time: 28.5286636353, output_fired_time: 13.7674875259, propagate_forward_time: 775.9240722656, propagate_backward_time: 1111.6302490234, weight_update_time: 0.5586786270, other_time: 20.3160400391
epoch time: 166.8334960938s, input_fired_time: 2.2100129128, output_fired_time: 0.0000000000, propagate_forward_time: 161.3289642334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2945251465
validation[          , genome  6421] predictions:    6324/  10000 (63.24%), best:    6404/10000 (64.04%), error:     10607.28613, best error:     10479.01367 on epoch:    19, epoch:   20/25, mu: 0.7734010220, learning_rate: 0.0001534052, weight_decay: 0.0000534459

epoch time: 1994.0563964844s, input_fired_time: 28.2896785736, output_fired_time: 13.8485164642, propagate_forward_time: 796.7229003906, propagate_backward_time: 1136.6588134766, weight_update_time: 0.5469121933, other_time: 18.5364990234
epoch time: 198.6805267334s, input_fired_time: 2.5112164021, output_fired_time: 0.0000000000, propagate_forward_time: 192.6334838867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5358276367
validation[          , genome  6421] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10444.84863, best error:     10444.84863 on epoch:    21, epoch:   21/25, mu: 0.7825076580, learning_rate: 0.0001427219, weight_decay: 0.0000497422

epoch time: 1912.8937988281s, input_fired_time: 27.9862957001, output_fired_time: 13.3913288116, propagate_forward_time: 757.3610839844, propagate_backward_time: 1094.5396728516, weight_update_time: 0.5394262671, other_time: 19.6153564453
epoch time: 174.8077392578s, input_fired_time: 2.2774031162, output_fired_time: 0.0000000000, propagate_forward_time: 169.1311492920, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3991851807
validation[          , genome  6421] predictions:    6241/  10000 (62.41%), best:    6437/10000 (64.37%), error:     10771.08008, best error:     10444.84863 on epoch:    21, epoch:   22/25, mu: 0.7912314534, learning_rate: 0.0001327826, weight_decay: 0.0000462951

epoch time: 1910.4533691406s, input_fired_time: 27.9903907776, output_fired_time: 13.3786602020, propagate_forward_time: 759.4356689453, propagate_backward_time: 1090.3681640625, weight_update_time: 0.5335784554, other_time: 19.2805175781
epoch time: 111.5132446289s, input_fired_time: 1.6855967045, output_fired_time: 0.0000000000, propagate_forward_time: 106.7909545898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0366897583
validation[          , genome  6421] predictions:    6421/  10000 (64.21%), best:    6421/10000 (64.21%), error:     10412.14258, best error:     10412.14258 on epoch:    23, epoch:   23/25, mu: 0.7995884418, learning_rate: 0.0001235355, weight_decay: 0.0000430869

epoch time: 1802.4324951172s, input_fired_time: 27.5885143280, output_fired_time: 12.7823448181, propagate_forward_time: 711.5767822266, propagate_backward_time: 1030.4771728516, weight_update_time: 0.5284704566, other_time: 20.0076904297
epoch time: 141.1286773682s, input_fired_time: 2.0074076653, output_fired_time: 0.0000000000, propagate_forward_time: 135.8234405518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2978363037
validation[          , genome  6421] predictions:    6435/  10000 (64.35%), best:    6421/10000 (64.21%), error:     10460.15039, best error:     10412.14258 on epoch:    23, epoch:   24/25, mu: 0.8075940609, learning_rate: 0.0001149323, weight_decay: 0.0000401010

epoch time: 1922.4017333984s, input_fired_time: 28.3133182526, output_fired_time: 13.4268398285, propagate_forward_time: 765.6929931641, propagate_backward_time: 1095.0260009766, weight_update_time: 0.5565351844, other_time: 19.9425048828
epoch time: 110.4070739746s, input_fired_time: 1.7205541134, output_fired_time: 0.0000000000, propagate_forward_time: 105.5780868530, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1084365845
validation[          , genome  6421] predictions:    6486/  10000 (64.86%), best:    6486/10000 (64.86%), error:     10337.43066, best error:     10337.43066 on epoch:    25, epoch:   25/25, mu: 0.8152630925, learning_rate: 0.0001069283, weight_decay: 0.0000373220

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 681.6725463867s, input_fired_time: 9.8569488525, output_fired_time: 0.0000000000, propagate_forward_time: 654.7503051758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.0653076172
best training[          , genome  6421] predictions:   34920/  50000 (69.84%), best:    6486/10000 (64.86%), error:     44831.05078, best error:     10337.43066 on epoch:    25, epoch:   26/25, mu: 0.8226097226, learning_rate: 0.0000994817, weight_decay: 0.0000347356
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 149.2024230957s, input_fired_time: 2.1477720737, output_fired_time: 0.0000000000, propagate_forward_time: 142.8663940430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1882629395
   testing[          , genome  6421] predictions:    6486/  10000 (64.86%), best:    6486/10000 (64.86%), error:     10337.43066, best error:     10337.43066 on epoch:    25, epoch:   26/25, mu: 0.8226097226, learning_rate: 0.0000994817, weight_decay: 0.0000347356
backpropagation finished successfully!
01:31:37 (3344): called boinc_finish(0)

</stderr_txt>
]]>