Nameexact_genome_1542727803_171_6420_1
Workunit4101339
Created20 Nov 2018, 20:17:51 UTC
Sent20 Nov 2018, 20:17:55 UTC
Report deadline26 Dec 2018, 4:39:40 UTC
Received23 Nov 2018, 7:53:36 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time16 hours 40 min 49 sec
CPU time16 hours 40 min 49 sec
Validate stateValid
Credit8,837.11
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size596.30 MB
Peak swap size595.10 MB
Peak disk usage14.66 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6420.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6420_1_r468607354_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_171_6420.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.479383
read mu: 0.479383
read mu_delta: 0.961471
read initial_learning_rate: 0.000432356
read learning_rate: 0.000432356
read learning_rate_delta: 0.926564
read initial_weight_decay: 0.000209843
read weight_decay: 0.000209843
read weight_decay_delta: 0.932301
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.0963611
read input_dropout_probability: 0.00133039
read hidden_dropout_probability: 0.122916
read velocity_reset: 1436
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6420
read normal distribution: '1 0 0'
generator_str: '1249941606'
read generator: 1249941606
read generated_by_map:
1 enable_edge 1
reading 128 nodes.
reading 510 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1083537088
post shuffle 1: 1312925332
epoch time: 2060.72021484375s, input_fired_time: 29.3497085571289, output_fired_time: 14.3642501831055, propagate_forward_time: 822.216247558594, propagate_backward_time: 1176.66333007813, weight_update_time: 0.549611508846283, other_time: 18.126708984375
epoch time: 161.374465942383s, input_fired_time: 2.14935755729675, output_fired_time: 0, propagate_forward_time: 156.034378051758, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.19073486328125
validation[          , genome  6420] predictions:    6381/  10000 (63.81%), best:    6381/10000 (63.81%), error:     10593.60645, best error:     10593.60645 on epoch:     0, epoch:    0/25, mu: 0.4793827236, learning_rate: 0.0004323556, weight_decay: 0.0002098428

epoch time: 2021.8768310547s, input_fired_time: 29.4405212402, output_fired_time: 14.2766637802, propagate_forward_time: 804.6837158203, propagate_backward_time: 1154.8092041016, weight_update_time: 0.5588722825, other_time: 18.6667480469
epoch time: 184.2248687744s, input_fired_time: 2.3453762531, output_fired_time: 0.0000000000, propagate_forward_time: 178.5868835449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2926025391
validation[          , genome  6420] predictions:    6226/  10000 (62.26%), best:    6381/10000 (63.81%), error:     10970.96777, best error:     10593.60645 on epoch:     0, epoch:    1/25, mu: 0.4990563989, learning_rate: 0.0004006052, weight_decay: 0.0001956366

epoch time: 2117.4699707031s, input_fired_time: 29.5877056122, output_fired_time: 14.7610034943, propagate_forward_time: 843.0599365234, propagate_backward_time: 1211.7269287109, weight_update_time: 0.5618335009, other_time: 18.3344726563
epoch time: 207.8220825195s, input_fired_time: 2.5830526352, output_fired_time: 0.0000000000, propagate_forward_time: 201.7862243652, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4528045654
validation[          , genome  6420] predictions:    6083/  10000 (60.83%), best:    6381/10000 (63.81%), error:     11520.85352, best error:     10593.60645 on epoch:     0, epoch:    2/25, mu: 0.5179720521, learning_rate: 0.0003711865, weight_decay: 0.0001823922

epoch time: 1945.6140136719s, input_fired_time: 28.8975620270, output_fired_time: 13.7048120499, propagate_forward_time: 770.8451538086, propagate_backward_time: 1113.8439941406, weight_update_time: 0.5381429195, other_time: 18.3225097656
epoch time: 180.3707427979s, input_fired_time: 2.3751091957, output_fired_time: 0.0000000000, propagate_forward_time: 174.5839843750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4116516113
validation[          , genome  6420] predictions:    6395/  10000 (63.95%), best:    6381/10000 (63.81%), error:     10647.55664, best error:     10593.60645 on epoch:     0, epoch:    3/25, mu: 0.5361589193, learning_rate: 0.0003439281, weight_decay: 0.0001700444

epoch time: 2012.0960693359s, input_fired_time: 29.3919639587, output_fired_time: 14.2540683746, propagate_forward_time: 801.1937255859, propagate_backward_time: 1148.5524902344, weight_update_time: 0.5582761168, other_time: 18.7038574219
epoch time: 163.2964782715s, input_fired_time: 2.2125737667, output_fired_time: 0.0000000000, propagate_forward_time: 157.7597351074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3241729736
validation[          , genome  6420] predictions:    6151/  10000 (61.51%), best:    6381/10000 (63.81%), error:     11169.14648, best error:     10593.60645 on epoch:     0, epoch:    4/25, mu: 0.5536450744, learning_rate: 0.0003186715, weight_decay: 0.0001585325

epoch time: 2239.5869140625s, input_fired_time: 30.4120101929, output_fired_time: 15.6992454529, propagate_forward_time: 898.6041259766, propagate_backward_time: 1275.9642333984, weight_update_time: 0.5956274867, other_time: 18.9072265625
epoch time: 222.7494201660s, input_fired_time: 2.7832262516, output_fired_time: 0.0000000000, propagate_forward_time: 216.2556152344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7105712891
validation[          , genome  6420] predictions:    6245/  10000 (62.45%), best:    6381/10000 (63.81%), error:     10861.00293, best error:     10593.60645 on epoch:     0, epoch:    5/25, mu: 0.5704574585, learning_rate: 0.0002952696, weight_decay: 0.0001478000

epoch time: 2379.1618652344s, input_fired_time: 31.4675121307, output_fired_time: 16.7610473633, propagate_forward_time: 956.2415161133, propagate_backward_time: 1354.6838378906, weight_update_time: 0.6371477246, other_time: 20.0080566406
epoch time: 157.2378234863s, input_fired_time: 2.1124217510, output_fired_time: 0.0000000000, propagate_forward_time: 152.0193023682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1060943604
validation[          , genome  6420] predictions:    6380/  10000 (63.80%), best:    6380/10000 (63.80%), error:     10577.88867, best error:     10577.88867 on epoch:     6, epoch:    6/25, mu: 0.5866221189, learning_rate: 0.0002735863, weight_decay: 0.0001377941

epoch time: 2265.6206054688s, input_fired_time: 30.6292629242, output_fired_time: 15.9763841629, propagate_forward_time: 910.1537475586, propagate_backward_time: 1289.7225341797, weight_update_time: 0.6015180945, other_time: 19.1387939453
epoch time: 181.5780944824s, input_fired_time: 2.3191289902, output_fired_time: 0.0000000000, propagate_forward_time: 175.9823150635, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2766571045
validation[          , genome  6420] predictions:    5886/  10000 (58.86%), best:    6380/10000 (63.80%), error:     11839.50195, best error:     10577.88867 on epoch:     6, epoch:    7/25, mu: 0.6021639705, learning_rate: 0.0002534952, weight_decay: 0.0001284655

epoch time: 2089.0898437500s, input_fired_time: 29.5195674896, output_fired_time: 14.6046380997, propagate_forward_time: 831.8821411133, propagate_backward_time: 1194.7462158203, weight_update_time: 0.5587682724, other_time: 18.3372802734
epoch time: 201.2862091064s, input_fired_time: 2.5704691410, output_fired_time: 0.0000000000, propagate_forward_time: 195.1349334717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5808105469
validation[          , genome  6420] predictions:    6224/  10000 (62.24%), best:    6380/10000 (63.80%), error:     11044.45605, best error:     10577.88867 on epoch:     6, epoch:    8/25, mu: 0.6171069741, learning_rate: 0.0002348796, weight_decay: 0.0001197685

epoch time: 2077.4660644531s, input_fired_time: 29.3348331451, output_fired_time: 14.4521846771, propagate_forward_time: 827.9189453125, propagate_backward_time: 1187.7961425781, weight_update_time: 0.5512652993, other_time: 17.9641113281
epoch time: 199.6194305420s, input_fired_time: 2.4975821972, output_fired_time: 0.0000000000, propagate_forward_time: 193.7325592041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3892822266
validation[          , genome  6420] predictions:    6172/  10000 (61.72%), best:    6380/10000 (63.80%), error:     11159.27637, best error:     10577.88867 on epoch:     6, epoch:    9/25, mu: 0.6314742565, learning_rate: 0.0002176310, weight_decay: 0.0001116603

epoch time: 2054.2670898438s, input_fired_time: 29.2559375763, output_fired_time: 14.3535270691, propagate_forward_time: 821.3696899414, propagate_backward_time: 1171.3349609375, weight_update_time: 0.5468736887, other_time: 17.9528808594
epoch time: 196.1634826660s, input_fired_time: 2.4794032574, output_fired_time: 0.0000000000, propagate_forward_time: 190.2987365723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3853454590
validation[          , genome  6420] predictions:    6413/  10000 (64.13%), best:    6380/10000 (63.80%), error:     10587.05078, best error:     10577.88867 on epoch:     6, epoch:   10/25, mu: 0.6452879906, learning_rate: 0.0002016491, weight_decay: 0.0001041010

epoch time: 2081.2373046875s, input_fired_time: 29.4090137482, output_fired_time: 14.5420789719, propagate_forward_time: 830.1913452148, propagate_backward_time: 1188.9022216797, weight_update_time: 0.5539655685, other_time: 18.1927490234
epoch time: 210.2815551758s, input_fired_time: 2.6023342609, output_fired_time: 0.0000000000, propagate_forward_time: 204.2258911133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4533233643
validation[          , genome  6420] predictions:    6388/  10000 (63.88%), best:    6380/10000 (63.80%), error:     10621.90625, best error:     10577.88867 on epoch:     6, epoch:   11/25, mu: 0.6585694551, learning_rate: 0.0001868409, weight_decay: 0.0000970535

epoch time: 2144.3879394531s, input_fired_time: 29.9256076813, output_fired_time: 15.0683727264, propagate_forward_time: 857.1259765625, propagate_backward_time: 1223.6370849609, weight_update_time: 0.5705642700, other_time: 18.6309814453
epoch time: 130.6878356934s, input_fired_time: 1.9021419287, output_fired_time: 0.0000000000, propagate_forward_time: 125.6797714233, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1059188843
validation[          , genome  6420] predictions:    6158/  10000 (61.58%), best:    6380/10000 (63.80%), error:     11198.82520, best error:     10577.88867 on epoch:     6, epoch:   12/25, mu: 0.6713392138, learning_rate: 0.0001731201, weight_decay: 0.0000904830

epoch time: 2094.0783691406s, input_fired_time: 29.8469276428, output_fired_time: 14.8140430450, propagate_forward_time: 834.5053100586, propagate_backward_time: 1195.9456787109, weight_update_time: 0.5764152408, other_time: 18.9664306641
epoch time: 159.3188629150s, input_fired_time: 2.1926937103, output_fired_time: 0.0000000000, propagate_forward_time: 153.8368072510, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2893676758
validation[          , genome  6420] predictions:    6461/  10000 (64.61%), best:    6461/10000 (64.61%), error:     10551.39160, best error:     10551.39160 on epoch:    13, epoch:   13/25, mu: 0.6836169362, learning_rate: 0.0001604069, weight_decay: 0.0000843574

epoch time: 2057.1279296875s, input_fired_time: 29.6752414703, output_fired_time: 14.4804687500, propagate_forward_time: 821.5833129883, propagate_backward_time: 1172.5106201172, weight_update_time: 0.5691716671, other_time: 18.8782958984
epoch time: 179.7021789551s, input_fired_time: 2.3786063194, output_fired_time: 0.0000000000, propagate_forward_time: 173.8995819092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4239959717
validation[          , genome  6420] predictions:    6462/  10000 (64.62%), best:    6462/10000 (64.62%), error:     10435.11133, best error:     10435.11133 on epoch:    14, epoch:   14/25, mu: 0.6954216361, learning_rate: 0.0001486273, weight_decay: 0.0000786465

epoch time: 1674.9521484375s, input_fired_time: 27.2011928558, output_fired_time: 11.7331323624, propagate_forward_time: 656.3231201172, propagate_backward_time: 962.6346435547, weight_update_time: 0.4711931050, other_time: 17.0600585938
epoch time: 126.1362686157s, input_fired_time: 1.8204489946, output_fired_time: 0.0000000000, propagate_forward_time: 121.3445739746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9712448120
validation[          , genome  6420] predictions:    6412/  10000 (64.12%), best:    6462/10000 (64.62%), error:     10496.21191, best error:     10435.11133 on epoch:    14, epoch:   15/25, mu: 0.7067714930, learning_rate: 0.0001377127, weight_decay: 0.0000733222

epoch time: 1978.1866455078s, input_fired_time: 28.8975582123, output_fired_time: 13.8378334045, propagate_forward_time: 785.7489013672, propagate_backward_time: 1131.7792968750, weight_update_time: 0.5358271599, other_time: 17.9229736328
epoch time: 152.8026733398s, input_fired_time: 2.1208047867, output_fired_time: 0.0000000000, propagate_forward_time: 147.4452514648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2366180420
validation[          , genome  6420] predictions:    6364/  10000 (63.64%), best:    6462/10000 (64.62%), error:     10641.51563, best error:     10435.11133 on epoch:    14, epoch:   16/25, mu: 0.7176840901, learning_rate: 0.0001275996, weight_decay: 0.0000683583

epoch time: 2122.7890625000s, input_fired_time: 30.0487518311, output_fired_time: 14.9593458176, propagate_forward_time: 848.9421386719, propagate_backward_time: 1209.6976318359, weight_update_time: 0.5856558681, other_time: 19.1412353516
epoch time: 224.5738525391s, input_fired_time: 2.7959942818, output_fired_time: 0.0000000000, propagate_forward_time: 218.0596771240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7181854248
validation[          , genome  6420] predictions:    6114/  10000 (61.14%), best:    6462/10000 (64.62%), error:     11461.09961, best error:     10435.11133 on epoch:    14, epoch:   17/25, mu: 0.7281762362, learning_rate: 0.0001182293, weight_decay: 0.0000637305

epoch time: 1889.6390380859s, input_fired_time: 28.7029685974, output_fired_time: 13.2956418991, propagate_forward_time: 748.9320678711, propagate_backward_time: 1080.4014892578, weight_update_time: 0.5313283801, other_time: 18.3067626953
epoch time: 168.4709014893s, input_fired_time: 2.2677435875, output_fired_time: 0.0000000000, propagate_forward_time: 162.8390045166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3641510010
validation[          , genome  6420] predictions:    6447/  10000 (64.47%), best:    6447/10000 (64.47%), error:     10423.06934, best error:     10423.06934 on epoch:    18, epoch:   18/25, mu: 0.7382640839, learning_rate: 0.0001095470, weight_decay: 0.0000594160

epoch time: 2219.2507324219s, input_fired_time: 30.4608421326, output_fired_time: 15.5847883224, propagate_forward_time: 888.5389404297, propagate_backward_time: 1265.4987792969, weight_update_time: 0.5950371623, other_time: 19.1673583984
epoch time: 180.3027191162s, input_fired_time: 2.3333609104, output_fired_time: 0.0000000000, propagate_forward_time: 174.6966247559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2727355957
validation[          , genome  6420] predictions:    6412/  10000 (64.12%), best:    6447/10000 (64.47%), error:     10456.18652, best error:     10423.06934 on epoch:    18, epoch:   19/25, mu: 0.7479632497, learning_rate: 0.0001015023, weight_decay: 0.0000553936

epoch time: 2236.1804199219s, input_fired_time: 30.3507499695, output_fired_time: 15.5620393753, propagate_forward_time: 895.9869384766, propagate_backward_time: 1275.5889892578, weight_update_time: 0.5862720609, other_time: 18.6916503906
epoch time: 175.5009155273s, input_fired_time: 2.2948167324, output_fired_time: 0.0000000000, propagate_forward_time: 169.9066009521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2994995117
validation[          , genome  6420] predictions:    6379/  10000 (63.79%), best:    6447/10000 (64.47%), error:     10670.05566, best error:     10423.06934 on epoch:    18, epoch:   20/25, mu: 0.7572887540, learning_rate: 0.0000940484, weight_decay: 0.0000516435

epoch time: 1967.7871093750s, input_fired_time: 28.9084377289, output_fired_time: 13.7886056900, propagate_forward_time: 779.2952880859, propagate_backward_time: 1127.7965087891, weight_update_time: 0.5322851539, other_time: 17.9982910156
epoch time: 157.1211853027s, input_fired_time: 2.1528017521, output_fired_time: 0.0000000000, propagate_forward_time: 151.2845611572, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6838226318
validation[          , genome  6420] predictions:    6413/  10000 (64.13%), best:    6413/10000 (64.13%), error:     10361.75391, best error:     10361.75391 on epoch:    21, epoch:   21/25, mu: 0.7662549615, learning_rate: 0.0000871419, weight_decay: 0.0000481473

epoch time: 2217.4277343750s, input_fired_time: 30.9427871704, output_fired_time: 15.8159608841, propagate_forward_time: 880.5747680664, propagate_backward_time: 1269.4046630859, weight_update_time: 0.6095783710, other_time: 20.6895751953
epoch time: 224.0463409424s, input_fired_time: 2.8214828968, output_fired_time: 0.0000000000, propagate_forward_time: 217.4812011719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7436523438
validation[          , genome  6420] predictions:    6492/  10000 (64.92%), best:    6492/10000 (64.92%), error:     10221.82715, best error:     10221.82715 on epoch:    22, epoch:   22/25, mu: 0.7748757005, learning_rate: 0.0000807426, weight_decay: 0.0000448878

epoch time: 2256.7680664063s, input_fired_time: 30.9291305542, output_fired_time: 15.9441576004, propagate_forward_time: 905.2410888672, propagate_backward_time: 1284.9578857422, weight_update_time: 0.6084620953, other_time: 19.6958007813
epoch time: 203.1762542725s, input_fired_time: 2.5536389351, output_fired_time: 0.0000000000, propagate_forward_time: 197.1879119873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4347076416
validation[          , genome  6420] predictions:    6360/  10000 (63.60%), best:    6492/10000 (64.92%), error:     10522.43066, best error:     10221.82715 on epoch:    22, epoch:   23/25, mu: 0.7831642628, learning_rate: 0.0000748132, weight_decay: 0.0000418489

epoch time: 2006.6925048828s, input_fired_time: 29.0688457489, output_fired_time: 13.9463348389, propagate_forward_time: 800.3139038086, propagate_backward_time: 1145.4637451172, weight_update_time: 0.5394819379, other_time: 17.8997802734
epoch time: 155.8019104004s, input_fired_time: 2.0976388454, output_fired_time: 0.0000000000, propagate_forward_time: 150.5991973877, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1050720215
validation[          , genome  6420] predictions:    6291/  10000 (62.91%), best:    6492/10000 (64.92%), error:     10671.36426, best error:     10221.82715 on epoch:    22, epoch:   24/25, mu: 0.7911334634, learning_rate: 0.0000693192, weight_decay: 0.0000390158

epoch time: 2113.3125000000s, input_fired_time: 30.4812984467, output_fired_time: 15.1039009094, propagate_forward_time: 836.7329711914, propagate_backward_time: 1210.0123291016, weight_update_time: 0.5918645263, other_time: 20.9820556641
epoch time: 186.9231109619s, input_fired_time: 2.4409298897, output_fired_time: 0.0000000000, propagate_forward_time: 181.0187683105, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4634094238
validation[          , genome  6420] predictions:    6433/  10000 (64.33%), best:    6492/10000 (64.92%), error:     10436.20117, best error:     10221.82715 on epoch:    22, epoch:   25/25, mu: 0.7987956405, learning_rate: 0.0000642287, weight_decay: 0.0000363744

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 821.8022460938s, input_fired_time: 10.9800128937, output_fired_time: 0.0000000000, propagate_forward_time: 793.7595825195, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.0626220703
best training[          , genome  6420] predictions:   35248/  50000 (70.50%), best:    6492/10000 (64.92%), error:     44751.05859, best error:     10221.82715 on epoch:    22, epoch:   26/25, mu: 0.8061625957, learning_rate: 0.0000595120, weight_decay: 0.0000339119
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 193.4340209961s, input_fired_time: 2.4677042961, output_fired_time: 0.0000000000, propagate_forward_time: 187.4300842285, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5362396240
   testing[          , genome  6420] predictions:    6492/  10000 (64.92%), best:    6492/10000 (64.92%), error:     10221.82715, best error:     10221.82715 on epoch:    22, epoch:   26/25, mu: 0.8061625957, learning_rate: 0.0000595120, weight_decay: 0.0000339119
backpropagation finished successfully!
01:51:03 (3304): called boinc_finish(0)

</stderr_txt>
]]>