Nameexact_genome_1542649503_172_6350_0
Workunit4099590
Created20 Nov 2018, 2:56:59 UTC
Sent20 Nov 2018, 3:01:13 UTC
Report deadline16 Dec 2018, 19:35:42 UTC
Received20 Nov 2018, 20:02:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84860
Run time11 hours 8 min 55 sec
CPU time11 hours 2 min 4 sec
Validate stateValid
Credit6,672.65
Device peak FLOPS4.41 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size563.76 MB
Peak swap size566.50 MB
Peak disk usage8.47 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6350.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6350_0_r696849019_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6350.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.509598
read mu: 0.509598
read mu_delta: 0.914469
read initial_learning_rate: 0.00103368
read learning_rate: 0.00103368
read learning_rate_delta: 0.932228
read initial_weight_decay: 0.000227453
read weight_decay: 0.000227453
read weight_decay_delta: 0.93482
read batch_size: 104
read epsilon: 1e-07
read alpha: 0.093868
read input_dropout_probability: 0.00155485
read hidden_dropout_probability: 0.0658336
read velocity_reset: 1714
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6350
read normal distribution: '1 0.667510211467743 0.263040661811829'
generator_str: '128297434'
read generator: 128297434
read generated_by_map:
1 change_size 1
reading 103 nodes.
reading 420 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 221391650
post shuffle 1: 1487126142
epoch time: 1398.83605957031s, input_fired_time: 22.0784492492676, output_fired_time: 7.81234550476074, propagate_forward_time: 529.2119140625, propagate_backward_time: 825.247497558594, weight_update_time: 0.404838532209396, other_time: 14.4857788085938
epoch time: 110.998817443848s, input_fired_time: 1.24986720085144, output_fired_time: 0, propagate_forward_time: 107.193496704102, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.55545043945312
validation[          , genome  6350] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10358.93066, best error:     10358.93066 on epoch:     0, epoch:    0/25, mu: 0.5095980167, learning_rate: 0.0010336818, weight_decay: 0.0002274530

epoch time: 1421.5511474609s, input_fired_time: 21.8766593933, output_fired_time: 7.9585900307, propagate_forward_time: 535.9790039062, propagate_backward_time: 840.9515991211, weight_update_time: 0.3901962936, other_time: 14.7852172852
epoch time: 109.0702667236s, input_fired_time: 1.1643989086, output_fired_time: 0.0000000000, propagate_forward_time: 105.4312362671, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4746322632
validation[          , genome  6350] predictions:    6286/  10000 (62.86%), best:    6437/10000 (64.37%), error:     10640.85254, best error:     10358.93066 on epoch:     0, epoch:    1/25, mu: 0.5506873727, learning_rate: 0.0009636272, weight_decay: 0.0002126275

epoch time: 1367.5290527344s, input_fired_time: 21.4674568176, output_fired_time: 7.7115550041, propagate_forward_time: 513.1367797852, propagate_backward_time: 811.1464843750, weight_update_time: 0.3770723939, other_time: 14.0668334961
epoch time: 108.6112518311s, input_fired_time: 1.1389249563, output_fired_time: 0.0000000000, propagate_forward_time: 105.2132568359, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2590713501
validation[          , genome  6350] predictions:    6412/  10000 (64.12%), best:    6437/10000 (64.37%), error:     10502.49609, best error:     10358.93066 on epoch:     0, epoch:    2/25, mu: 0.5882623196, learning_rate: 0.0008983205, weight_decay: 0.0001987683

epoch time: 1379.6534423828s, input_fired_time: 21.1080265045, output_fired_time: 7.7576470375, propagate_forward_time: 512.1238403320, propagate_backward_time: 825.5219726562, weight_update_time: 0.3599865437, other_time: 13.1419067383
epoch time: 108.3955841064s, input_fired_time: 1.1308494806, output_fired_time: 0.0000000000, propagate_forward_time: 104.9580841064, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3066482544
validation[          , genome  6350] predictions:    6212/  10000 (62.12%), best:    6437/10000 (64.37%), error:     11086.25684, best error:     10358.93066 on epoch:     0, epoch:    3/25, mu: 0.6226234436, learning_rate: 0.0008374396, weight_decay: 0.0001858125

epoch time: 1406.4289550781s, input_fired_time: 21.4189796448, output_fired_time: 7.8465561867, propagate_forward_time: 525.7830810547, propagate_backward_time: 837.6710205078, weight_update_time: 0.3691792190, other_time: 13.7093505859
epoch time: 113.6144561768s, input_fired_time: 1.2309874296, output_fired_time: 0.0000000000, propagate_forward_time: 109.8121643066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5713043213
validation[          , genome  6350] predictions:    6371/  10000 (63.71%), best:    6437/10000 (64.37%), error:     10617.79199, best error:     10358.93066 on epoch:     0, epoch:    4/25, mu: 0.6540456414, learning_rate: 0.0007806848, weight_decay: 0.0001737012

epoch time: 1387.1041259766s, input_fired_time: 21.6778831482, output_fired_time: 7.8194913864, propagate_forward_time: 521.9594116211, propagate_backward_time: 822.0073852539, weight_update_time: 0.3773535490, other_time: 13.6400146484
epoch time: 102.8910140991s, input_fired_time: 1.1702220440, output_fired_time: 0.0000000000, propagate_forward_time: 99.5640182495, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1567764282
validation[          , genome  6350] predictions:    6347/  10000 (63.47%), best:    6437/10000 (64.37%), error:     10690.91406, best error:     10358.93066 on epoch:     0, epoch:    5/25, mu: 0.6827802062, learning_rate: 0.0007277763, weight_decay: 0.0001623793

epoch time: 1391.6804199219s, input_fired_time: 21.4005336761, output_fired_time: 7.7062206268, propagate_forward_time: 522.2050781250, propagate_backward_time: 826.6561889648, weight_update_time: 0.3671178520, other_time: 13.7124633789
epoch time: 109.5249404907s, input_fired_time: 1.2212522030, output_fired_time: 0.0000000000, propagate_forward_time: 105.8017044067, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5019836426
validation[          , genome  6350] predictions:    6404/  10000 (64.04%), best:    6404/10000 (64.04%), error:     10346.77930, best error:     10346.77930 on epoch:     6, epoch:    6/25, mu: 0.7090570927, learning_rate: 0.0006784536, weight_decay: 0.0001517953

epoch time: 1386.0777587891s, input_fired_time: 21.6369228363, output_fired_time: 7.7725663185, propagate_forward_time: 521.7757568359, propagate_backward_time: 820.8655395508, weight_update_time: 0.3780079186, other_time: 14.0269165039
epoch time: 109.0917205811s, input_fired_time: 1.1698893309, output_fired_time: 0.0000000000, propagate_forward_time: 105.5056533813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4161758423
validation[          , genome  6350] predictions:    6211/  10000 (62.11%), best:    6404/10000 (64.04%), error:     11075.79395, best error:     10346.77930 on epoch:     6, epoch:    7/25, mu: 0.7330864668, learning_rate: 0.0006324736, weight_decay: 0.0001419013

epoch time: 1418.6032714844s, input_fired_time: 22.1856403351, output_fired_time: 7.8681387901, propagate_forward_time: 541.7764892578, propagate_backward_time: 831.8857421875, weight_update_time: 0.3996279538, other_time: 14.8872070312
epoch time: 117.9062194824s, input_fired_time: 1.3014614582, output_fired_time: 0.0000000000, propagate_forward_time: 114.0250854492, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5796737671
validation[          , genome  6350] predictions:    6294/  10000 (62.94%), best:    6404/10000 (64.04%), error:     10693.92676, best error:     10346.77930 on epoch:     6, epoch:    8/25, mu: 0.7550606132, learning_rate: 0.0005896097, weight_decay: 0.0001326521

epoch time: 1430.1831054688s, input_fired_time: 22.4983310699, output_fired_time: 7.9245471954, propagate_forward_time: 548.8080444336, propagate_backward_time: 835.9950561523, weight_update_time: 0.3981431127, other_time: 14.9571533203
epoch time: 111.9127960205s, input_fired_time: 1.2530641556, output_fired_time: 0.0000000000, propagate_forward_time: 108.0488281250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6109008789
validation[          , genome  6350] predictions:    6260/  10000 (62.60%), best:    6404/10000 (64.04%), error:     10795.18359, best error:     10346.77930 on epoch:     6, epoch:    9/25, mu: 0.7751552463, learning_rate: 0.0005496508, weight_decay: 0.0001240058

epoch time: 1420.2139892578s, input_fired_time: 21.7006626129, output_fired_time: 7.9634137154, propagate_forward_time: 533.8546752930, propagate_backward_time: 841.8777465820, weight_update_time: 0.3874910474, other_time: 14.8175048828
epoch time: 110.4006958008s, input_fired_time: 1.1911474466, output_fired_time: 0.0000000000, propagate_forward_time: 106.6514053345, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5581436157
validation[          , genome  6350] predictions:    6409/  10000 (64.09%), best:    6404/10000 (64.04%), error:     10391.95703, best error:     10346.77930 on epoch:     6, epoch:   10/25, mu: 0.7935311794, learning_rate: 0.0005123999, weight_decay: 0.0001159230

epoch time: 1465.4747314453s, input_fired_time: 23.2186393738, output_fired_time: 7.9744477272, propagate_forward_time: 571.8865356445, propagate_backward_time: 847.4268188477, weight_update_time: 0.4097570479, other_time: 14.9682617188
epoch time: 108.9589004517s, input_fired_time: 1.1648229361, output_fired_time: 0.0000000000, propagate_forward_time: 105.2332916260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5607833862
validation[          , genome  6350] predictions:    6425/  10000 (64.25%), best:    6425/10000 (64.25%), error:     10240.41211, best error:     10240.41211 on epoch:    11, epoch:   11/25, mu: 0.8103353977, learning_rate: 0.0004776736, weight_decay: 0.0001083671

epoch time: 1408.0628662109s, input_fired_time: 21.4388141632, output_fired_time: 7.8703651428, propagate_forward_time: 526.3893432617, propagate_backward_time: 837.3943481445, weight_update_time: 0.3838365674, other_time: 14.9699707031
epoch time: 111.8408660889s, input_fired_time: 1.2448797226, output_fired_time: 0.0000000000, propagate_forward_time: 108.0074005127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5885848999
validation[          , genome  6350] predictions:    6473/  10000 (64.73%), best:    6473/10000 (64.73%), error:     10153.50879, best error:     10153.50879 on epoch:    12, epoch:   12/25, mu: 0.8257023096, learning_rate: 0.0004453008, weight_decay: 0.0001013037

epoch time: 1437.7009277344s, input_fired_time: 22.6340808868, output_fired_time: 8.0008296967, propagate_forward_time: 549.5103759766, propagate_backward_time: 842.2658081055, weight_update_time: 0.4201725721, other_time: 15.2898559570
epoch time: 115.3495864868s, input_fired_time: 1.2921029329, output_fired_time: 0.0000000000, propagate_forward_time: 111.3383865356, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7190933228
validation[          , genome  6350] predictions:    6479/  10000 (64.79%), best:    6479/10000 (64.79%), error:     10062.99219, best error:     10062.99219 on epoch:    13, epoch:   13/25, mu: 0.8397548795, learning_rate: 0.0004151219, weight_decay: 0.0000947007

epoch time: 1418.8239746094s, input_fired_time: 22.0690135956, output_fired_time: 7.9043946266, propagate_forward_time: 535.4578247070, propagate_backward_time: 838.5386962891, weight_update_time: 0.3925876021, other_time: 14.8540649414
epoch time: 109.8083343506s, input_fired_time: 1.1823127270, output_fired_time: 0.0000000000, propagate_forward_time: 106.0973510742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5286712646
validation[          , genome  6350] predictions:    6475/  10000 (64.75%), best:    6479/10000 (64.79%), error:     10221.13770, best error:     10062.99219 on epoch:    13, epoch:   14/25, mu: 0.8526055217, learning_rate: 0.0003869883, weight_decay: 0.0000885280

epoch time: 1424.5677490234s, input_fired_time: 22.1278209686, output_fired_time: 7.9373011589, propagate_forward_time: 538.8197631836, propagate_backward_time: 841.0293579102, weight_update_time: 0.3945935667, other_time: 14.6535644531
epoch time: 116.1362380981s, input_fired_time: 1.2877525091, output_fired_time: 0.0000000000, propagate_forward_time: 112.3115768433, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5369110107
validation[          , genome  6350] predictions:    6472/  10000 (64.72%), best:    6479/10000 (64.79%), error:     10156.03809, best error:     10062.99219 on epoch:    13, epoch:   15/25, mu: 0.8643570542, learning_rate: 0.0003607614, weight_decay: 0.0000827578

epoch time: 1423.8001708984s, input_fired_time: 22.3779888153, output_fired_time: 8.0415134430, propagate_forward_time: 540.1544799805, propagate_backward_time: 838.1978759766, weight_update_time: 0.4072106481, other_time: 15.0283813477
epoch time: 119.2807922363s, input_fired_time: 1.3033615351, output_fired_time: 0.0000000000, propagate_forward_time: 115.2790756226, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6983566284
validation[          , genome  6350] predictions:    6447/  10000 (64.47%), best:    6479/10000 (64.79%), error:     10232.28418, best error:     10062.99219 on epoch:    13, epoch:   16/25, mu: 0.8751034737, learning_rate: 0.0003363120, weight_decay: 0.0000773636

epoch time: 1464.9909667969s, input_fired_time: 23.3689613342, output_fired_time: 8.2208042145, propagate_forward_time: 561.3133544922, propagate_backward_time: 856.1629638672, weight_update_time: 0.4286016524, other_time: 15.9248046875
epoch time: 115.1186370850s, input_fired_time: 1.3071717024, output_fired_time: 0.0000000000, propagate_forward_time: 111.0462265015, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7652359009
validation[          , genome  6350] predictions:    6517/  10000 (65.17%), best:    6479/10000 (64.79%), error:     10073.52930, best error:     10062.99219 on epoch:    13, epoch:   17/25, mu: 0.8849307299, learning_rate: 0.0003135195, weight_decay: 0.0000723210

epoch time: 1408.6156005859s, input_fired_time: 22.3503036499, output_fired_time: 7.9045615196, propagate_forward_time: 535.0883789062, propagate_backward_time: 828.6990356445, weight_update_time: 0.3934023082, other_time: 14.5733032227
epoch time: 108.6038742065s, input_fired_time: 1.2397575378, output_fired_time: 0.0000000000, propagate_forward_time: 104.8464660645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5176544189
validation[          , genome  6350] predictions:    6481/  10000 (64.81%), best:    6479/10000 (64.79%), error:     10088.76953, best error:     10062.99219 on epoch:    13, epoch:   18/25, mu: 0.8939174414, learning_rate: 0.0002922717, weight_decay: 0.0000676071

epoch time: 1405.9342041016s, input_fired_time: 21.5133914948, output_fired_time: 7.8229756355, propagate_forward_time: 527.2094116211, propagate_backward_time: 835.1664428711, weight_update_time: 0.3742752075, other_time: 14.2219238281
epoch time: 110.4426727295s, input_fired_time: 1.2656742334, output_fired_time: 0.0000000000, propagate_forward_time: 106.6086196899, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5683822632
validation[          , genome  6350] predictions:    6483/  10000 (64.83%), best:    6479/10000 (64.79%), error:     10146.86523, best error:     10062.99219 on epoch:    13, epoch:   19/25, mu: 0.9021354914, learning_rate: 0.0002724639, weight_decay: 0.0000632004

epoch time: 1406.3572998047s, input_fired_time: 21.8938064575, output_fired_time: 7.8210387230, propagate_forward_time: 532.1708984375, propagate_backward_time: 830.0686645508, weight_update_time: 0.3859241605, other_time: 14.4028930664
epoch time: 109.3349304199s, input_fired_time: 1.1772680283, output_fired_time: 0.0000000000, propagate_forward_time: 105.7243957520, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4332656860
validation[          , genome  6350] predictions:    6525/  10000 (65.25%), best:    6525/10000 (65.25%), error:      9979.55859, best error:      9979.55859 on epoch:    20, epoch:   20/25, mu: 0.9096506238, learning_rate: 0.0002539985, weight_decay: 0.0000590810

epoch time: 3589.2792968750s, input_fired_time: 21.4410400391, output_fired_time: 7.8682827950, propagate_forward_time: 520.9057617188, propagate_backward_time: 3025.5786132812, weight_update_time: 0.3810716867, other_time: 13.4858398438
epoch time: 112.7516937256s, input_fired_time: 1.2277116776, output_fired_time: 0.0000000000, propagate_forward_time: 109.1110992432, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4128799438
validation[          , genome  6350] predictions:    6476/  10000 (64.76%), best:    6525/10000 (65.25%), error:     10037.21094, best error:      9979.55859 on epoch:    20, epoch:   21/25, mu: 0.9165229797, learning_rate: 0.0002367845, weight_decay: 0.0000552301

epoch time: 1391.5886230469s, input_fired_time: 21.3392906189, output_fired_time: 7.8338756561, propagate_forward_time: 518.0812377930, propagate_backward_time: 831.1193847656, weight_update_time: 0.3729399145, other_time: 13.2149047852
epoch time: 107.6156921387s, input_fired_time: 1.1792469025, output_fired_time: 0.0000000000, propagate_forward_time: 104.2253494263, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2110977173
validation[          , genome  6350] predictions:    6466/  10000 (64.66%), best:    6525/10000 (65.25%), error:     10125.45215, best error:      9979.55859 on epoch:    20, epoch:   22/25, mu: 0.9228075743, learning_rate: 0.0002207372, weight_decay: 0.0000516301

epoch time: 1366.7320556641s, input_fired_time: 21.5131645203, output_fired_time: 7.7451100349, propagate_forward_time: 509.2561340332, propagate_backward_time: 815.1610107422, weight_update_time: 0.3736163974, other_time: 13.0566406250
epoch time: 108.9935989380s, input_fired_time: 1.2436679602, output_fired_time: 0.0000000000, propagate_forward_time: 105.2867355347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4631958008
validation[          , genome  6350] predictions:    6407/  10000 (64.07%), best:    6525/10000 (65.25%), error:     10142.74707, best error:      9979.55859 on epoch:    20, epoch:   23/25, mu: 0.9285546541, learning_rate: 0.0002057774, weight_decay: 0.0000482649

epoch time: 1381.7550048828s, input_fired_time: 21.5424041748, output_fired_time: 7.7429804802, propagate_forward_time: 517.2786254883, propagate_backward_time: 821.8471069336, weight_update_time: 0.3682105839, other_time: 13.3439941406
epoch time: 108.2813568115s, input_fired_time: 1.2446022034, output_fired_time: 0.0000000000, propagate_forward_time: 104.7648773193, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2718811035
validation[          , genome  6350] predictions:    6541/  10000 (65.41%), best:    6541/10000 (65.41%), error:      9974.40918, best error:      9974.40918 on epoch:    24, epoch:   24/25, mu: 0.9338101745, learning_rate: 0.0001918315, weight_decay: 0.0000451189

epoch time: 1394.5644531250s, input_fired_time: 21.5610790253, output_fired_time: 7.7961978912, propagate_forward_time: 521.9957885742, propagate_backward_time: 829.6934814453, weight_update_time: 0.3665590584, other_time: 13.5180053711
epoch time: 106.5224838257s, input_fired_time: 1.2395632267, output_fired_time: 0.0000000000, propagate_forward_time: 102.9229125977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3600082397
validation[          , genome  6350] predictions:    6452/  10000 (64.52%), best:    6541/10000 (65.41%), error:     10167.08008, best error:      9974.40918 on epoch:    24, epoch:   25/25, mu: 0.9386161566, learning_rate: 0.0001788307, weight_decay: 0.0000421781

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 530.6838378906s, input_fired_time: 6.1032547951, output_fired_time: 0.0000000000, propagate_forward_time: 513.0099487305, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.5706176758
best training[          , genome  6350] predictions:   36574/  50000 (73.15%), best:    6541/10000 (65.41%), error:     39670.03906, best error:      9974.40918 on epoch:    24, epoch:   26/25, mu: 0.9430111051, learning_rate: 0.0001667110, weight_decay: 0.0000394289
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 108.7614135742s, input_fired_time: 1.2304513454, output_fired_time: 0.0000000000, propagate_forward_time: 105.1851425171, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3458175659
   testing[          , genome  6350] predictions:    6541/  10000 (65.41%), best:    6541/10000 (65.41%), error:      9974.40918, best error:      9974.40918 on epoch:    24, epoch:   26/25, mu: 0.9430111051, learning_rate: 0.0001667110, weight_decay: 0.0000394289
backpropagation finished successfully!
10:09:39 (27782): called boinc_finish(0)

</stderr_txt>
]]>