Nameexact_genome_1542649503_172_6325_0
Workunit4099013
Created19 Nov 2018, 23:45:26 UTC
Sent19 Nov 2018, 23:45:29 UTC
Report deadline19 Dec 2018, 7:58:50 UTC
Received20 Nov 2018, 11:06:40 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60660
Run time9 hours 12 min 23 sec
CPU time9 hours 2 min 13 sec
Validate stateValid
Credit7,335.65
Device peak FLOPS3.44 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size581.77 MB
Peak swap size601.51 MB
Peak disk usage13.22 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6325.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6325_0_r2076998858_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6325.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.500958
read mu: 0.500958
read mu_delta: 0.931334
read initial_learning_rate: 0.000251415
read learning_rate: 0.000251415
read learning_rate_delta: 0.930738
read initial_weight_decay: 0.000224188
read weight_decay: 0.000224188
read weight_decay_delta: 0.940267
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0942871
read input_dropout_probability: 0.00138528
read hidden_dropout_probability: 0.079676
read velocity_reset: 1858
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6325
read normal distribution: '0 -0.722341179847717 0.490794479846954'
generator_str: '869765801'
read generator: 869765801
read generated_by_map:
1 add_node 1
reading 107 nodes.
reading 435 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 232632278
post shuffle 1: 906251073
epoch time: 1161.84558105469s, input_fired_time: 25.467212677002, output_fired_time: 8.82350540161133, propagate_forward_time: 425.126922607422, propagate_backward_time: 687.73291015625, weight_update_time: 0.346359997987747, other_time: 14.695068359375
epoch time: 89.6271286010742s, input_fired_time: 1.60708487033844, output_fired_time: 0, propagate_forward_time: 85.5182571411133, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.50178527832031
validation[          , genome  6325] predictions:    6491/  10000 (64.91%), best:    6491/10000 (64.91%), error:     10253.12988, best error:     10253.12988 on epoch:     0, epoch:    0/25, mu: 0.5009581447, learning_rate: 0.0002514150, weight_decay: 0.0002241884

epoch time: 1161.3731689453s, input_fired_time: 25.4713115692, output_fired_time: 8.8114328384, propagate_forward_time: 424.3841247559, propagate_backward_time: 688.2335205078, weight_update_time: 0.3455685973, other_time: 14.4727783203
epoch time: 90.4065093994s, input_fired_time: 1.6728290319, output_fired_time: 0.0000000000, propagate_forward_time: 86.2037200928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5299606323
validation[          , genome  6325] predictions:    6309/  10000 (63.09%), best:    6491/10000 (64.91%), error:     10753.73730, best error:     10253.12988 on epoch:     0, epoch:    1/25, mu: 0.5345386267, learning_rate: 0.0002340014, weight_decay: 0.0002107970

epoch time: 1360.1425781250s, input_fired_time: 25.5173187256, output_fired_time: 8.8437585831, propagate_forward_time: 620.0769653320, propagate_backward_time: 690.8432006836, weight_update_time: 0.3676890731, other_time: 14.8613281250
epoch time: 90.1225128174s, input_fired_time: 1.6346093416, output_fired_time: 0.0000000000, propagate_forward_time: 85.9139709473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5739288330
validation[          , genome  6325] predictions:    6392/  10000 (63.92%), best:    6491/10000 (64.91%), error:     10504.60547, best error:     10253.12988 on epoch:     0, epoch:    2/25, mu: 0.5658133030, learning_rate: 0.0002177940, weight_decay: 0.0001982055

epoch time: 1170.8389892578s, input_fired_time: 25.5479488373, output_fired_time: 8.9207525253, propagate_forward_time: 428.9657592773, propagate_backward_time: 692.7015380859, weight_update_time: 0.3452035189, other_time: 14.7029418945
epoch time: 89.4855728149s, input_fired_time: 1.6067548990, output_fired_time: 0.0000000000, propagate_forward_time: 85.4061203003, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4726943970
validation[          , genome  6325] predictions:    6338/  10000 (63.38%), best:    6491/10000 (64.91%), error:     10648.10059, best error:     10253.12988 on epoch:     0, epoch:    3/25, mu: 0.5949404836, learning_rate: 0.0002027091, weight_decay: 0.0001863662

epoch time: 1176.2396240234s, input_fired_time: 25.4948749542, output_fired_time: 9.0506420135, propagate_forward_time: 430.4730224609, propagate_backward_time: 696.6725463867, weight_update_time: 0.3649711013, other_time: 14.5485229492
epoch time: 89.8674774170s, input_fired_time: 1.6378669739, output_fired_time: 0.0000000000, propagate_forward_time: 85.6108703613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6187438965
validation[          , genome  6325] predictions:    6339/  10000 (63.39%), best:    6491/10000 (64.91%), error:     10646.78125, best error:     10253.12988 on epoch:     0, epoch:    4/25, mu: 0.6220676303, learning_rate: 0.0001886691, weight_decay: 0.0001752340

epoch time: 1164.4095458984s, input_fired_time: 25.4542064667, output_fired_time: 8.9225654602, propagate_forward_time: 426.2592468262, propagate_backward_time: 688.9761352539, weight_update_time: 0.3496505618, other_time: 14.7973022461
epoch time: 89.8374862671s, input_fired_time: 1.6043000221, output_fired_time: 0.0000000000, propagate_forward_time: 85.6965637207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5366210938
validation[          , genome  6325] predictions:    6277/  10000 (62.77%), best:    6491/10000 (64.91%), error:     10831.11523, best error:     10253.12988 on epoch:     0, epoch:    5/25, mu: 0.6473320723, learning_rate: 0.0001756014, weight_decay: 0.0001647668

epoch time: 1163.8317871094s, input_fired_time: 25.5000476837, output_fired_time: 8.7932443619, propagate_forward_time: 425.5238342285, propagate_backward_time: 689.4609375000, weight_update_time: 0.3455368280, other_time: 14.5538330078
epoch time: 89.1392364502s, input_fired_time: 1.5994739532, output_fired_time: 0.0000000000, propagate_forward_time: 85.0203399658, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5194244385
validation[          , genome  6325] predictions:    6315/  10000 (63.15%), best:    6491/10000 (64.91%), error:     10917.21875, best error:     10253.12988 on epoch:     0, epoch:    6/25, mu: 0.6708616614, learning_rate: 0.0001634389, weight_decay: 0.0001549248

epoch time: 1160.0773925781s, input_fired_time: 25.3076152802, output_fired_time: 8.8327598572, propagate_forward_time: 424.6415100098, propagate_backward_time: 686.6151123047, weight_update_time: 0.3557875156, other_time: 14.6804199219
epoch time: 89.9691772461s, input_fired_time: 1.6135320663, output_fired_time: 0.0000000000, propagate_forward_time: 85.8905792236, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4650650024
validation[          , genome  6325] predictions:    6135/  10000 (61.35%), best:    6491/10000 (64.91%), error:     11299.51758, best error:     10253.12988 on epoch:     0, epoch:    7/25, mu: 0.6927756071, learning_rate: 0.0001521188, weight_decay: 0.0001456707

epoch time: 1164.0968017578s, input_fired_time: 25.4030666351, output_fired_time: 8.8469171524, propagate_forward_time: 426.3078613281, propagate_backward_time: 688.9388427734, weight_update_time: 0.3480581641, other_time: 14.6000976563
epoch time: 89.9382553101s, input_fired_time: 1.6333038807, output_fired_time: 0.0000000000, propagate_forward_time: 85.8286743164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4762802124
validation[          , genome  6325] predictions:    6339/  10000 (63.39%), best:    6491/10000 (64.91%), error:     10793.05762, best error:     10253.12988 on epoch:     0, epoch:    8/25, mu: 0.7131847739, learning_rate: 0.0001415827, weight_decay: 0.0001369694

epoch time: 3147.6804199219s, input_fired_time: 25.2246646881, output_fired_time: 8.8726673126, propagate_forward_time: 2412.0000000000, propagate_backward_time: 687.0489501953, weight_update_time: 0.3608972132, other_time: 14.5343017578
epoch time: 89.6892013550s, input_fired_time: 1.5587403774, output_fired_time: 0.0000000000, propagate_forward_time: 85.6234283447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5070343018
validation[          , genome  6325] predictions:    5761/  10000 (57.61%), best:    6491/10000 (64.91%), error:     12337.45215, best error:     10253.12988 on epoch:     0, epoch:    9/25, mu: 0.7321925759, learning_rate: 0.0001317764, weight_decay: 0.0001287879

epoch time: 1202.9222412109s, input_fired_time: 25.2028713226, output_fired_time: 8.8330230713, propagate_forward_time: 465.9727783203, propagate_backward_time: 688.4721069336, weight_update_time: 0.3472552299, other_time: 14.4414672852
epoch time: 89.7957458496s, input_fired_time: 1.6299104691, output_fired_time: 0.0000000000, propagate_forward_time: 85.6396865845, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5261459351
validation[          , genome  6325] predictions:    6359/  10000 (63.59%), best:    6491/10000 (64.91%), error:     10765.08496, best error:     10253.12988 on epoch:     0, epoch:   10/25, mu: 0.7498951554, learning_rate: 0.0001226493, weight_decay: 0.0001210950

epoch time: 1163.1846923828s, input_fired_time: 25.5351200104, output_fired_time: 8.8956403732, propagate_forward_time: 425.4541320801, propagate_backward_time: 688.8738403320, weight_update_time: 0.3565064371, other_time: 14.4259643555
epoch time: 89.9310226440s, input_fired_time: 1.6185203791, output_fired_time: 0.0000000000, propagate_forward_time: 85.8569869995, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4555130005
validation[          , genome  6325] predictions:    6262/  10000 (62.62%), best:    6491/10000 (64.91%), error:     10983.71582, best error:     10253.12988 on epoch:     0, epoch:   11/25, mu: 0.7663821578, learning_rate: 0.0001141543, weight_decay: 0.0001138617

epoch time: 1158.5345458984s, input_fired_time: 25.0985546112, output_fired_time: 8.7662448883, propagate_forward_time: 423.9862670898, propagate_backward_time: 686.4288330078, weight_update_time: 0.3514618874, other_time: 14.2546997070
epoch time: 89.3674468994s, input_fired_time: 1.5689975023, output_fired_time: 0.0000000000, propagate_forward_time: 85.3597640991, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4386825562
validation[          , genome  6325] predictions:    6195/  10000 (61.95%), best:    6491/10000 (64.91%), error:     11098.54980, best error:     10253.12988 on epoch:     0, epoch:   12/25, mu: 0.7817370892, learning_rate: 0.0001062478, weight_decay: 0.0001070604

epoch time: 1164.0146484375s, input_fired_time: 25.2443542480, output_fired_time: 8.8657913208, propagate_forward_time: 426.5653991699, propagate_backward_time: 688.9163208008, weight_update_time: 0.3452118337, other_time: 14.4226684570
epoch time: 90.0159530640s, input_fired_time: 1.6131449938, output_fired_time: 0.0000000000, propagate_forward_time: 85.9069595337, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4958496094
validation[          , genome  6325] predictions:    6191/  10000 (61.91%), best:    6491/10000 (64.91%), error:     11046.55371, best error:     10253.12988 on epoch:     0, epoch:   13/25, mu: 0.7960376740, learning_rate: 0.0000988888, weight_decay: 0.0001006654

epoch time: 1159.4503173828s, input_fired_time: 25.1760692596, output_fired_time: 8.8991861343, propagate_forward_time: 424.5267639160, propagate_backward_time: 686.2997436523, weight_update_time: 0.3474035859, other_time: 14.5486450195
epoch time: 89.8319015503s, input_fired_time: 1.5695029497, output_fired_time: 0.0000000000, propagate_forward_time: 85.8438873291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4185104370
validation[          , genome  6325] predictions:    6187/  10000 (61.87%), best:    6491/10000 (64.91%), error:     11163.69336, best error:     10253.12988 on epoch:     0, epoch:   14/25, mu: 0.8093562722, learning_rate: 0.0000920396, weight_decay: 0.0000946524

epoch time: 1304.9678955078s, input_fired_time: 25.1562652588, output_fired_time: 8.7436103821, propagate_forward_time: 572.1145019531, propagate_backward_time: 684.5425415039, weight_update_time: 0.3758474588, other_time: 14.4109497070
epoch time: 89.4982528687s, input_fired_time: 1.6060822010, output_fired_time: 0.0000000000, propagate_forward_time: 85.4276351929, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4645385742
validation[          , genome  6325] predictions:    6362/  10000 (63.62%), best:    6491/10000 (64.91%), error:     10818.42285, best error:     10253.12988 on epoch:     0, epoch:   15/25, mu: 0.8217603564, learning_rate: 0.0000856647, weight_decay: 0.0000889985

epoch time: 1986.4495849609s, input_fired_time: 25.3265495300, output_fired_time: 8.8772115707, propagate_forward_time: 1249.4875488281, propagate_backward_time: 688.0475463867, weight_update_time: 0.3477292359, other_time: 14.7107543945
epoch time: 90.1375885010s, input_fired_time: 1.6172548532, output_fired_time: 0.0000000000, propagate_forward_time: 86.0088806152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5114517212
validation[          , genome  6325] predictions:    6201/  10000 (62.01%), best:    6491/10000 (64.91%), error:     11066.80859, best error:     10253.12988 on epoch:     0, epoch:   16/25, mu: 0.8333126903, learning_rate: 0.0000797314, weight_decay: 0.0000836824

epoch time: 1161.7033691406s, input_fired_time: 25.4096527100, output_fired_time: 8.8759546280, propagate_forward_time: 425.2966613770, propagate_backward_time: 687.4826049805, weight_update_time: 0.3501347005, other_time: 14.6384887695
epoch time: 89.8561401367s, input_fired_time: 1.5944755077, output_fired_time: 0.0000000000, propagate_forward_time: 85.7370605469, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5246047974
validation[          , genome  6325] predictions:    6309/  10000 (63.09%), best:    6491/10000 (64.91%), error:     10695.92871, best error:     10253.12988 on epoch:     0, epoch:   17/25, mu: 0.8440717459, learning_rate: 0.0000742090, weight_decay: 0.0000786838

epoch time: 1183.9112548828s, input_fired_time: 25.7441234589, output_fired_time: 9.0376148224, propagate_forward_time: 433.1010437012, propagate_backward_time: 701.1785888672, weight_update_time: 0.3571553528, other_time: 14.8498535156
epoch time: 91.0148849487s, input_fired_time: 1.6161111593, output_fired_time: 0.0000000000, propagate_forward_time: 86.7686309814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6301422119
validation[          , genome  6325] predictions:    6323/  10000 (63.23%), best:    6491/10000 (64.91%), error:     10709.30664, best error:     10253.12988 on epoch:     0, epoch:   18/25, mu: 0.8540920615, learning_rate: 0.0000690691, weight_decay: 0.0000739838

epoch time: 1183.8526611328s, input_fired_time: 25.6316280365, output_fired_time: 9.1108570099, propagate_forward_time: 432.5820922852, propagate_backward_time: 701.5100097656, weight_update_time: 0.3661132455, other_time: 15.0181274414
epoch time: 89.0505218506s, input_fired_time: 1.5931636095, output_fired_time: 0.0000000000, propagate_forward_time: 84.8837738037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5735855103
validation[          , genome  6325] predictions:    6050/  10000 (60.50%), best:    6491/10000 (64.91%), error:     11329.22461, best error:     10253.12988 on epoch:     0, epoch:   19/25, mu: 0.8634243011, learning_rate: 0.0000642853, weight_decay: 0.0000695646

epoch time: 1156.3980712891s, input_fired_time: 25.1326522827, output_fired_time: 8.8361024857, propagate_forward_time: 423.4763793945, propagate_backward_time: 684.2330932617, weight_update_time: 0.3523454070, other_time: 14.7198486328
epoch time: 89.7272644043s, input_fired_time: 1.5902776718, output_fired_time: 0.0000000000, propagate_forward_time: 85.6107406616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5262451172
validation[          , genome  6325] predictions:    6282/  10000 (62.82%), best:    6491/10000 (64.91%), error:     10844.20215, best error:     10253.12988 on epoch:     0, epoch:   20/25, mu: 0.8721157312, learning_rate: 0.0000598327, weight_decay: 0.0000654093

epoch time: 1157.0321044922s, input_fired_time: 25.2428627014, output_fired_time: 8.8276634216, propagate_forward_time: 423.6493835449, propagate_backward_time: 684.6690673828, weight_update_time: 0.3488380313, other_time: 14.6430664063
epoch time: 88.7781524658s, input_fired_time: 1.5926250219, output_fired_time: 0.0000000000, propagate_forward_time: 84.6499710083, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5355529785
validation[          , genome  6325] predictions:    6327/  10000 (63.27%), best:    6491/10000 (64.91%), error:     10707.12598, best error:     10253.12988 on epoch:     0, epoch:   21/25, mu: 0.8802103400, learning_rate: 0.0000556886, weight_decay: 0.0000615022

epoch time: 1799.2806396484s, input_fired_time: 25.6891098022, output_fired_time: 8.9154863358, propagate_forward_time: 425.2739868164, propagate_backward_time: 1324.5996093750, weight_update_time: 0.3461572826, other_time: 14.8024902344
epoch time: 90.0685501099s, input_fired_time: 1.6494888067, output_fired_time: 0.0000000000, propagate_forward_time: 85.8959197998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5231399536
validation[          , genome  6325] predictions:    6340/  10000 (63.40%), best:    6491/10000 (64.91%), error:     10724.00684, best error:     10253.12988 on epoch:     0, epoch:   22/25, mu: 0.8877491355, learning_rate: 0.0000518315, weight_decay: 0.0000578285

epoch time: 1162.5981445313s, input_fired_time: 25.3956756592, output_fired_time: 8.8312778473, propagate_forward_time: 425.9956970215, propagate_backward_time: 687.8850708008, weight_update_time: 0.3486708701, other_time: 14.4904174805
epoch time: 89.3912353516s, input_fired_time: 1.5878483057, output_fired_time: 0.0000000000, propagate_forward_time: 85.3117065430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4916839600
validation[          , genome  6325] predictions:    6161/  10000 (61.61%), best:    6491/10000 (64.91%), error:     11085.77246, best error:     10253.12988 on epoch:     0, epoch:   23/25, mu: 0.8947702646, learning_rate: 0.0000482415, weight_decay: 0.0000543743

epoch time: 1158.7460937500s, input_fired_time: 25.1299839020, output_fired_time: 8.7910509109, propagate_forward_time: 424.4733276367, propagate_backward_time: 685.8735351563, weight_update_time: 0.3432945907, other_time: 14.4782104492
epoch time: 89.9126052856s, input_fired_time: 1.6021853685, output_fired_time: 0.0000000000, propagate_forward_time: 85.8313064575, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4791107178
validation[          , genome  6325] predictions:    6352/  10000 (63.52%), best:    6491/10000 (64.91%), error:     10694.31641, best error:     10253.12988 on epoch:     0, epoch:   24/25, mu: 0.9013093114, learning_rate: 0.0000449002, weight_decay: 0.0000511263

epoch time: 1846.5770263672s, input_fired_time: 25.1610031128, output_fired_time: 8.7294054031, propagate_forward_time: 422.5588684082, propagate_backward_time: 1375.6479492188, weight_update_time: 0.3436514139, other_time: 14.4798583984
epoch time: 88.9484329224s, input_fired_time: 1.5884059668, output_fired_time: 0.0000000000, propagate_forward_time: 84.8521499634, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5078735352
validation[          , genome  6325] predictions:    6320/  10000 (63.20%), best:    6491/10000 (64.91%), error:     10500.96875, best error:     10253.12988 on epoch:     0, epoch:   25/25, mu: 0.9073993564, learning_rate: 0.0000417903, weight_decay: 0.0000480724

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 443.2117004395s, input_fired_time: 8.0513477325, output_fired_time: 0.0000000000, propagate_forward_time: 422.5489807129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.6113586426
best training[          , genome  6325] predictions:   36146/  50000 (72.29%), best:    6491/10000 (64.91%), error:     42731.48047, best error:     10253.12988 on epoch:     0, epoch:   26/25, mu: 0.9130712152, learning_rate: 0.0000388958, weight_decay: 0.0000452009
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 88.8467102051s, input_fired_time: 1.6051914692, output_fired_time: 0.0000000000, propagate_forward_time: 84.7067718506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5347442627
   testing[          , genome  6325] predictions:    6491/  10000 (64.91%), best:    6491/10000 (64.91%), error:     10253.12988, best error:     10253.12988 on epoch:     0, epoch:   26/25, mu: 0.9130712152, learning_rate: 0.0000388958, weight_decay: 0.0000452009
backpropagation finished successfully!
05:29:08 (11924): called boinc_finish(0)

</stderr_txt>
]]>