Nameexact_genome_1542334203_172_6029_1
Workunit4091929
Created16 Nov 2018, 3:06:10 UTC
Sent16 Nov 2018, 3:15:44 UTC
Report deadline14 Dec 2018, 14:29:06 UTC
Received18 Nov 2018, 22:24:44 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID75310
Run time11 hours 9 min 54 sec
CPU time11 hours 9 min 42 sec
Validate stateValid
Credit7,116.91
Device peak FLOPS3.28 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size568.86 MB
Peak swap size571.54 MB
Peak disk usage8.67 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_172_6029.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542334203_172_6029_1_r852841017_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542334203_172_6029.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.498469
read mu: 0.498469
read mu_delta: 0.937128
read initial_learning_rate: 0.000842057
read learning_rate: 0.000842057
read learning_rate_delta: 0.929698
read initial_weight_decay: 0.000241872
read weight_decay: 0.000241872
read weight_decay_delta: 0.941312
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0944993
read input_dropout_probability: 0.00135003
read hidden_dropout_probability: 0.0839317
read velocity_reset: 1866
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6029
read normal distribution: '0 -1.03473246097565 0.404400318861008'
generator_str: '946402868'
read generator: 946402868
read generated_by_map:
1 change_size_x 1
reading 105 nodes.
reading 426 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1929112794
post shuffle 1: 1860016321
epoch time: 1406.68957519531s, input_fired_time: 27.6499500274658, output_fired_time: 8.15473461151123, propagate_forward_time: 546.791320800781, propagate_backward_time: 808.603210449219, weight_update_time: 0.386081129312515, other_time: 15.4903564453125
epoch time: 113.875221252441s, input_fired_time: 1.51933312416077, output_fired_time: 0, propagate_forward_time: 109.84001159668, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.51587677001953
validation[          , genome  6029] predictions:    6199/  10000 (61.99%), best:    6199/10000 (61.99%), error:     11040.50293, best error:     11040.50293 on epoch:     0, epoch:    0/25, mu: 0.4984686077, learning_rate: 0.0008420574, weight_decay: 0.0002418719

epoch time: 1405.7923583984s, input_fired_time: 27.7550888062, output_fired_time: 8.2717647552, propagate_forward_time: 545.3896484375, propagate_backward_time: 808.4213256836, weight_update_time: 0.3990652263, other_time: 15.9545288086
epoch time: 114.7519912720s, input_fired_time: 1.5087376833, output_fired_time: 0.0000000000, propagate_forward_time: 110.7057266235, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5375289917
validation[          , genome  6029] predictions:    5979/  10000 (59.79%), best:    6199/10000 (61.99%), error:     11545.69824, best error:     11040.50293 on epoch:     0, epoch:    1/25, mu: 0.5293722749, learning_rate: 0.0007828595, weight_decay: 0.0002276770

epoch time: 1404.1533203125s, input_fired_time: 27.6478061676, output_fired_time: 8.1109962463, propagate_forward_time: 545.1245117188, propagate_backward_time: 807.9972534180, weight_update_time: 0.3834321797, other_time: 15.2727661133
epoch time: 113.5662689209s, input_fired_time: 1.4947695732, output_fired_time: 0.0000000000, propagate_forward_time: 109.5468902588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5246124268
validation[          , genome  6029] predictions:    5787/  10000 (57.87%), best:    6199/10000 (61.99%), error:     11914.25781, best error:     11040.50293 on epoch:     0, epoch:    2/25, mu: 0.5583329201, learning_rate: 0.0007278232, weight_decay: 0.0002143151

epoch time: 1404.4396972656s, input_fired_time: 27.6325664520, output_fired_time: 8.1558618546, propagate_forward_time: 545.6774902344, propagate_backward_time: 807.4567871094, weight_update_time: 0.3904693425, other_time: 15.5169677734
epoch time: 113.8213653564s, input_fired_time: 1.5033761263, output_fired_time: 0.0000000000, propagate_forward_time: 109.8178482056, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5001373291
validation[          , genome  6029] predictions:    5960/  10000 (59.60%), best:    6199/10000 (61.99%), error:     11503.13574, best error:     11040.50293 on epoch:     0, epoch:    3/25, mu: 0.5854727626, learning_rate: 0.0006766561, weight_decay: 0.0002017374

epoch time: 1397.4659423828s, input_fired_time: 27.5626125336, output_fired_time: 8.0509204865, propagate_forward_time: 542.9335327148, propagate_backward_time: 803.8231811523, weight_update_time: 0.3808974028, other_time: 15.0957031250
epoch time: 113.9251785278s, input_fired_time: 1.5204551220, output_fired_time: 0.0000000000, propagate_forward_time: 109.9084320068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4962921143
validation[          , genome  6029] predictions:    6222/  10000 (62.22%), best:    6222/10000 (62.22%), error:     10935.46289, best error:     10935.46289 on epoch:     4, epoch:    4/25, mu: 0.6109062433, learning_rate: 0.0006290862, weight_decay: 0.0001898978

epoch time: 1405.9560546875s, input_fired_time: 27.6744518280, output_fired_time: 8.2332897186, propagate_forward_time: 545.9277343750, propagate_backward_time: 808.4071655273, weight_update_time: 0.3950159848, other_time: 15.7134399414
epoch time: 113.8634567261s, input_fired_time: 1.5223447084, output_fired_time: 0.0000000000, propagate_forward_time: 109.8360595703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5050506592
validation[          , genome  6029] predictions:    6207/  10000 (62.07%), best:    6222/10000 (62.22%), error:     10973.50000, best error:     10935.46289 on epoch:     4, epoch:    5/25, mu: 0.6347407103, learning_rate: 0.0005848604, weight_decay: 0.0001787531

epoch time: 1405.9205322266s, input_fired_time: 27.6876964569, output_fired_time: 8.2367811203, propagate_forward_time: 544.2756347656, propagate_backward_time: 809.6376342773, weight_update_time: 0.3976745903, other_time: 16.0827026367
epoch time: 113.6947174072s, input_fired_time: 1.5053232908, output_fired_time: 0.0000000000, propagate_forward_time: 109.6845169067, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5048751831
validation[          , genome  6029] predictions:    6197/  10000 (61.97%), best:    6222/10000 (62.22%), error:     10961.93262, best error:     10935.46289 on epoch:     4, epoch:    6/25, mu: 0.6570766568, learning_rate: 0.0005437438, weight_decay: 0.0001682625

epoch time: 1406.4497070312s, input_fired_time: 27.6850738525, output_fired_time: 8.1915225983, propagate_forward_time: 546.6821289062, propagate_backward_time: 808.2008056641, weight_update_time: 0.3926905692, other_time: 15.6901855469
epoch time: 113.9846649170s, input_fired_time: 1.4986953735, output_fired_time: 0.0000000000, propagate_forward_time: 109.9945983887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4913711548
validation[          , genome  6029] predictions:    6017/  10000 (60.17%), best:    6222/10000 (62.22%), error:     11358.09375, best error:     10935.46289 on epoch:     4, epoch:    7/25, mu: 0.6780082583, learning_rate: 0.0005055178, weight_decay: 0.0001583875

epoch time: 1407.3319091797s, input_fired_time: 27.6281833649, output_fired_time: 8.2130432129, propagate_forward_time: 545.8789672852, propagate_backward_time: 809.9910278320, weight_update_time: 0.3940994143, other_time: 15.6207275391
epoch time: 116.3418731689s, input_fired_time: 1.5902470350, output_fired_time: 0.0000000000, propagate_forward_time: 111.9908294678, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7607955933
validation[          , genome  6029] predictions:    6289/  10000 (62.89%), best:    6289/10000 (62.89%), error:     10743.94824, best error:     10743.94824 on epoch:     8, epoch:    8/25, mu: 0.6976238489, learning_rate: 0.0004699791, weight_decay: 0.0001490921

epoch time: 1402.9427490234s, input_fired_time: 27.6695327759, output_fired_time: 8.1524734497, propagate_forward_time: 544.6946411133, propagate_backward_time: 806.9772949219, weight_update_time: 0.3859049678, other_time: 15.4487915039
epoch time: 113.0966644287s, input_fired_time: 1.5140486956, output_fired_time: 0.0000000000, propagate_forward_time: 109.0872039795, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4954147339
validation[          , genome  6029] predictions:    6215/  10000 (62.15%), best:    6289/10000 (62.89%), error:     10952.50879, best error:     10743.94824 on epoch:     8, epoch:    9/25, mu: 0.7160061598, learning_rate: 0.0004369389, weight_decay: 0.0001403421

epoch time: 1403.8209228516s, input_fired_time: 27.6288108826, output_fired_time: 8.1343803406, propagate_forward_time: 545.4653320312, propagate_backward_time: 807.3065185547, weight_update_time: 0.3818555474, other_time: 15.2858886719
epoch time: 113.3170471191s, input_fired_time: 1.5136163235, output_fired_time: 0.0000000000, propagate_forward_time: 109.2944412231, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5089874268
validation[          , genome  6029] predictions:    6119/  10000 (61.19%), best:    6289/10000 (62.89%), error:     10929.55371, best error:     10743.94824 on epoch:     8, epoch:   10/25, mu: 0.7332327366, learning_rate: 0.0004062214, weight_decay: 0.0001321058

epoch time: 1404.0513916016s, input_fired_time: 27.7394027710, output_fired_time: 8.1322479248, propagate_forward_time: 545.0260009766, propagate_backward_time: 807.7510986328, weight_update_time: 0.3870103955, other_time: 15.4027099609
epoch time: 114.0330200195s, input_fired_time: 1.5092424154, output_fired_time: 0.0000000000, propagate_forward_time: 110.0096359253, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5141448975
validation[          , genome  6029] predictions:    6040/  10000 (60.40%), best:    6289/10000 (62.89%), error:     11311.95703, best error:     10743.94824 on epoch:     8, epoch:   11/25, mu: 0.7493762374, learning_rate: 0.0003776634, weight_decay: 0.0001243528

epoch time: 1406.6890869141s, input_fired_time: 27.8129234314, output_fired_time: 8.2591800690, propagate_forward_time: 546.6515502930, propagate_backward_time: 807.8400878906, weight_update_time: 0.4052404761, other_time: 16.1254272461
epoch time: 114.0140991211s, input_fired_time: 1.5255608559, output_fired_time: 0.0000000000, propagate_forward_time: 109.9771881104, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5113525391
validation[          , genome  6029] predictions:    6239/  10000 (62.39%), best:    6289/10000 (62.89%), error:     10745.31836, best error:     10743.94824 on epoch:     8, epoch:   12/25, mu: 0.7645047903, learning_rate: 0.0003511131, weight_decay: 0.0001170547

epoch time: 1406.1866455078s, input_fired_time: 27.7326564789, output_fired_time: 8.2524433136, propagate_forward_time: 546.3281860352, propagate_backward_time: 808.0243530273, weight_update_time: 0.3938575089, other_time: 15.8489990234
epoch time: 114.5364990234s, input_fired_time: 1.5333354473, output_fired_time: 0.0000000000, propagate_forward_time: 110.4860000610, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5171661377
validation[          , genome  6029] predictions:    6199/  10000 (61.99%), best:    6289/10000 (62.89%), error:     10871.15137, best error:     10743.94824 on epoch:     8, epoch:   13/25, mu: 0.7786821723, learning_rate: 0.0003264293, weight_decay: 0.0001101851

epoch time: 1406.1274414062s, input_fired_time: 27.7021751404, output_fired_time: 8.2074089050, propagate_forward_time: 545.6669311523, propagate_backward_time: 809.0880126953, weight_update_time: 0.3869810402, other_time: 15.4629516602
epoch time: 113.3488769531s, input_fired_time: 1.5187803507, output_fired_time: 0.0000000000, propagate_forward_time: 109.3322372437, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4978561401
validation[          , genome  6029] predictions:    6357/  10000 (63.57%), best:    6357/10000 (63.57%), error:     10444.52148, best error:     10444.52148 on epoch:    14, epoch:   14/25, mu: 0.7919681668, learning_rate: 0.0003034808, weight_decay: 0.0001037185

epoch time: 1406.9162597656s, input_fired_time: 27.7305717468, output_fired_time: 8.2185964584, propagate_forward_time: 545.4818725586, propagate_backward_time: 809.4514160156, weight_update_time: 0.3981516659, other_time: 16.0337524414
epoch time: 113.7910766602s, input_fired_time: 1.5229395628, output_fired_time: 0.0000000000, propagate_forward_time: 109.7435760498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5245590210
validation[          , genome  6029] predictions:    6291/  10000 (62.91%), best:    6357/10000 (63.57%), error:     10686.82324, best error:     10444.52148 on epoch:    14, epoch:   15/25, mu: 0.8044188619, learning_rate: 0.0002821457, weight_decay: 0.0000976315

epoch time: 1404.6157226562s, input_fired_time: 27.6725711823, output_fired_time: 8.1241807938, propagate_forward_time: 546.1175537109, propagate_backward_time: 807.3994140625, weight_update_time: 0.3833932579, other_time: 15.3020019531
epoch time: 115.1015396118s, input_fired_time: 1.5839775801, output_fired_time: 0.0000000000, propagate_forward_time: 110.7657623291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7518005371
validation[          , genome  6029] predictions:    6348/  10000 (63.48%), best:    6357/10000 (63.57%), error:     10539.05273, best error:     10444.52148 on epoch:    14, epoch:   16/25, mu: 0.8160867691, learning_rate: 0.0002623104, weight_decay: 0.0000919017

epoch time: 1401.8778076172s, input_fired_time: 27.6727542877, output_fired_time: 8.1417980194, propagate_forward_time: 544.4481811523, propagate_backward_time: 806.1375122070, weight_update_time: 0.3846782744, other_time: 15.4775390625
epoch time: 113.2588653564s, input_fired_time: 1.5111625195, output_fired_time: 0.0000000000, propagate_forward_time: 109.2234344482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5242691040
validation[          , genome  6029] predictions:    6264/  10000 (62.64%), best:    6357/10000 (63.57%), error:     10790.49707, best error:     10444.52148 on epoch:    14, epoch:   17/25, mu: 0.8270210624, learning_rate: 0.0002438696, weight_decay: 0.0000865082

epoch time: 1404.2187500000s, input_fired_time: 27.7834892273, output_fired_time: 8.1761388779, propagate_forward_time: 545.4489746094, propagate_backward_time: 807.1940917969, weight_update_time: 0.3899286091, other_time: 15.6160888672
epoch time: 114.9436950684s, input_fired_time: 1.5881503820, output_fired_time: 0.0000000000, propagate_forward_time: 110.6662368774, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6893081665
validation[          , genome  6029] predictions:    6322/  10000 (63.22%), best:    6357/10000 (63.57%), error:     10526.74805, best error:     10444.52148 on epoch:    14, epoch:   18/25, mu: 0.8372678757, learning_rate: 0.0002267252, weight_decay: 0.0000814312

epoch time: 1407.9793701172s, input_fired_time: 27.7919960022, output_fired_time: 8.1581983566, propagate_forward_time: 548.7604980469, propagate_backward_time: 807.7020263672, weight_update_time: 0.3896282911, other_time: 15.5666503906
epoch time: 114.9202270508s, input_fired_time: 1.5495693684, output_fired_time: 0.0000000000, propagate_forward_time: 110.8400421143, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5306167603
validation[          , genome  6029] predictions:    6293/  10000 (62.93%), best:    6357/10000 (63.57%), error:     10480.28027, best error:     10444.52148 on epoch:    14, epoch:   19/25, mu: 0.8468704820, learning_rate: 0.0002107860, weight_decay: 0.0000766522

epoch time: 1406.3444824219s, input_fired_time: 27.7290115356, output_fired_time: 8.2330656052, propagate_forward_time: 545.6860961914, propagate_backward_time: 809.1635742188, weight_update_time: 0.3917182982, other_time: 15.5327758789
epoch time: 112.8445129395s, input_fired_time: 1.5559138060, output_fired_time: 0.0000000000, propagate_forward_time: 108.7420349121, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5465621948
validation[          , genome  6029] predictions:    6471/  10000 (64.71%), best:    6471/10000 (64.71%), error:     10241.01074, best error:     10241.01074 on epoch:    20, epoch:   20/25, mu: 0.8558693528, learning_rate: 0.0001959675, weight_decay: 0.0000721536

epoch time: 1405.8476562500s, input_fired_time: 27.7512760162, output_fired_time: 8.0941772461, propagate_forward_time: 546.7502441406, propagate_backward_time: 807.9221801758, weight_update_time: 0.3835864365, other_time: 15.3297729492
epoch time: 113.7209091187s, input_fired_time: 1.5434768200, output_fired_time: 0.0000000000, propagate_forward_time: 109.6454544067, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5319747925
validation[          , genome  6029] predictions:    6317/  10000 (63.17%), best:    6471/10000 (64.71%), error:     10516.79395, best error:     10241.01074 on epoch:    20, epoch:   21/25, mu: 0.8643024564, learning_rate: 0.0001821906, weight_decay: 0.0000679191

epoch time: 1406.8104248047s, input_fired_time: 27.7343368530, output_fired_time: 8.1602716446, propagate_forward_time: 544.9799804688, propagate_backward_time: 810.5220336914, weight_update_time: 0.3875985444, other_time: 15.4137573242
epoch time: 114.6618881226s, input_fired_time: 1.5437890291, output_fired_time: 0.0000000000, propagate_forward_time: 110.5503463745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5677490234
validation[          , genome  6029] predictions:    6425/  10000 (64.25%), best:    6425/10000 (64.25%), error:     10231.43750, best error:     10231.43750 on epoch:    22, epoch:   22/25, mu: 0.8722053170, learning_rate: 0.0001693824, weight_decay: 0.0000639330

epoch time: 1403.8493652344s, input_fired_time: 27.7758312225, output_fired_time: 8.1229038239, propagate_forward_time: 545.9262084961, propagate_backward_time: 806.6911010742, weight_update_time: 0.3841515183, other_time: 15.3332519531
epoch time: 113.6885833740s, input_fired_time: 1.5515245199, output_fired_time: 0.0000000000, propagate_forward_time: 109.5248413086, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6122207642
validation[          , genome  6029] predictions:    6289/  10000 (62.89%), best:    6425/10000 (64.25%), error:     10491.27246, best error:     10231.43750 on epoch:    22, epoch:   23/25, mu: 0.8796113133, learning_rate: 0.0001574745, weight_decay: 0.0000601809

epoch time: 1404.1557617188s, input_fired_time: 27.7741718292, output_fired_time: 8.0853338242, propagate_forward_time: 545.2921142578, propagate_backward_time: 807.8627929688, weight_update_time: 0.3771646619, other_time: 15.1413574219
epoch time: 113.0936126709s, input_fired_time: 1.5429894924, output_fired_time: 0.0000000000, propagate_forward_time: 109.0438919067, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5067291260
validation[          , genome  6029] predictions:    6374/  10000 (63.74%), best:    6425/10000 (64.25%), error:     10359.77148, best error:     10231.43750 on epoch:    22, epoch:   24/25, mu: 0.8865516782, learning_rate: 0.0001464038, weight_decay: 0.0000566490

epoch time: 1405.7315673828s, input_fired_time: 27.8444423676, output_fired_time: 8.1344900131, propagate_forward_time: 546.0284423828, propagate_backward_time: 808.2804565430, weight_update_time: 0.3875089288, other_time: 15.4436645508
epoch time: 115.4666748047s, input_fired_time: 1.5598586798, output_fired_time: 0.0000000000, propagate_forward_time: 111.3638458252, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5429687500
validation[          , genome  6029] predictions:    6363/  10000 (63.63%), best:    6425/10000 (64.25%), error:     10361.59375, best error:     10231.43750 on epoch:    22, epoch:   25/25, mu: 0.8930556774, learning_rate: 0.0001361114, weight_decay: 0.0000533244

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 570.5582275391s, input_fired_time: 7.8027644157, output_fired_time: 0.0000000000, propagate_forward_time: 549.7730712891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.9824218750
best training[          , genome  6029] predictions:   35454/  50000 (70.91%), best:    6425/10000 (64.25%), error:     43212.33594, best error:     10231.43750 on epoch:    22, epoch:   26/25, mu: 0.8991507888, learning_rate: 0.0001265426, weight_decay: 0.0000501949
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 114.9846954346s, input_fired_time: 1.5831257105, output_fired_time: 0.0000000000, propagate_forward_time: 110.8038711548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5977020264
   testing[          , genome  6029] predictions:    6425/  10000 (64.25%), best:    6425/10000 (64.25%), error:     10231.43750, best error:     10231.43750 on epoch:    22, epoch:   26/25, mu: 0.8991507888, learning_rate: 0.0001265426, weight_decay: 0.0000501949
backpropagation finished successfully!
22:24:07 (24779): called boinc_finish(0)

</stderr_txt>
]]>