Nameexact_genome_1542727803_172_6419_1
Workunit4101315
Created20 Nov 2018, 20:14:39 UTC
Sent20 Nov 2018, 20:15:03 UTC
Report deadline17 Dec 2018, 7:24:34 UTC
Received21 Nov 2018, 6:16:47 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60660
Run time8 hours 27 min
CPU time8 hours 8 min 46 sec
Validate stateValid
Credit6,616.24
Device peak FLOPS3.44 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size580.73 MB
Peak swap size600.57 MB
Peak disk usage12.61 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6419.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6419_1_r1196985608_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_172_6419.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.485532
read mu: 0.485532
read mu_delta: 0.942106
read initial_learning_rate: 0.000272663
read learning_rate: 0.000272663
read learning_rate_delta: 0.926093
read initial_weight_decay: 0.000218318
read weight_decay: 0.000218318
read weight_decay_delta: 0.944322
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0923005
read input_dropout_probability: 0.00130696
read hidden_dropout_probability: 0.0857913
read velocity_reset: 1823
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6419
read normal distribution: '0 -0.181744173169136 0.264517575502396'
generator_str: '742624285'
read generator: 742624285
read generated_by_map:
1 change_size 1
reading 110 nodes.
reading 427 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 111401631
post shuffle 1: 678699660
epoch time: 1048.57739257813s, input_fired_time: 26.2677364349365, output_fired_time: 9.06993961334229, propagate_forward_time: 380.644348144531, propagate_backward_time: 618.310607910156, weight_update_time: 0.332075208425522, other_time: 14.2847290039063
epoch time: 80.1516876220703s, input_fired_time: 1.6151762008667, output_fired_time: 0, propagate_forward_time: 76.0172119140625, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.51930236816406
validation[          , genome  6419] predictions:    6137/  10000 (61.37%), best:    6137/10000 (61.37%), error:     11290.29785, best error:     11290.29785 on epoch:     0, epoch:    0/25, mu: 0.4855318964, learning_rate: 0.0002726633, weight_decay: 0.0002183181

epoch time: 1042.1555175781s, input_fired_time: 26.0522766113, output_fired_time: 9.0453138351, propagate_forward_time: 378.1955871582, propagate_backward_time: 614.5750732422, weight_update_time: 0.3281364143, other_time: 14.2873535156
epoch time: 80.2273635864s, input_fired_time: 1.5900572538, output_fired_time: 0.0000000000, propagate_forward_time: 76.2048034668, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4325027466
validation[          , genome  6419] predictions:    6043/  10000 (60.43%), best:    6137/10000 (61.37%), error:     11635.88477, best error:     11290.29785 on epoch:     0, epoch:    1/25, mu: 0.5147375464, learning_rate: 0.0002525116, weight_decay: 0.0002061627

epoch time: 1042.3636474609s, input_fired_time: 26.0198287964, output_fired_time: 9.0156049728, propagate_forward_time: 378.4909057617, propagate_backward_time: 614.6118164063, weight_update_time: 0.3276381791, other_time: 14.2254638672
epoch time: 80.5129928589s, input_fired_time: 1.5975921154, output_fired_time: 0.0000000000, propagate_forward_time: 76.4973297119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4180679321
validation[          , genome  6419] predictions:    5913/  10000 (59.13%), best:    6137/10000 (61.37%), error:     12165.53711, best error:     11290.29785 on epoch:     0, epoch:    2/25, mu: 0.5422523618, learning_rate: 0.0002338493, weight_decay: 0.0001946840

epoch time: 1044.1557617188s, input_fired_time: 26.0088233948, output_fired_time: 9.0154628754, propagate_forward_time: 378.7397766113, propagate_backward_time: 616.1799926758, weight_update_time: 0.3283547163, other_time: 14.2117309570
epoch time: 80.3354568481s, input_fired_time: 1.5855222940, output_fired_time: 0.0000000000, propagate_forward_time: 76.1943893433, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5555419922
validation[          , genome  6419] predictions:    5717/  10000 (57.17%), best:    6137/10000 (61.37%), error:     12853.62598, best error:     11290.29785 on epoch:     0, epoch:    3/25, mu: 0.5681742430, learning_rate: 0.0002165662, weight_decay: 0.0001838444

epoch time: 1044.5518798828s, input_fired_time: 26.0636253357, output_fired_time: 9.0343875885, propagate_forward_time: 379.4208679199, propagate_backward_time: 615.7048950195, weight_update_time: 0.3310417533, other_time: 14.3281860352
epoch time: 80.8081359863s, input_fired_time: 1.6488301754, output_fired_time: 0.0000000000, propagate_forward_time: 76.6453857422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5139236450
validation[          , genome  6419] predictions:    5956/  10000 (59.56%), best:    6137/10000 (61.37%), error:     11860.34863, best error:     11290.29785 on epoch:     0, epoch:    4/25, mu: 0.5925953984, learning_rate: 0.0002005605, weight_decay: 0.0001736084

epoch time: 1274.4122314453s, input_fired_time: 26.1029758453, output_fired_time: 9.1557970047, propagate_forward_time: 605.4116210938, propagate_backward_time: 619.3856201172, weight_update_time: 0.3473211229, other_time: 14.3562011719
epoch time: 80.9187316895s, input_fired_time: 1.6028522253, output_fired_time: 0.0000000000, propagate_forward_time: 76.7918701172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5240097046
validation[          , genome  6419] predictions:    5741/  10000 (57.41%), best:    6137/10000 (61.37%), error:     12507.02832, best error:     11290.29785 on epoch:     0, epoch:    5/25, mu: 0.6156027317, learning_rate: 0.0001857378, weight_decay: 0.0001639423

epoch time: 1221.0709228516s, input_fired_time: 25.9708938599, output_fired_time: 9.0767183304, propagate_forward_time: 555.0612792969, propagate_backward_time: 616.6892089844, weight_update_time: 0.3281304836, other_time: 14.2728271484
epoch time: 80.1678466797s, input_fired_time: 1.5816890001, output_fired_time: 0.0000000000, propagate_forward_time: 76.0567398071, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5294189453
validation[          , genome  6419] predictions:    5905/  10000 (59.05%), best:    6137/10000 (61.37%), error:     12048.58301, best error:     11290.29785 on epoch:     0, epoch:    6/25, mu: 0.6372780800, learning_rate: 0.0001720105, weight_decay: 0.0001548143

epoch time: 1043.2972412109s, input_fired_time: 25.9564552307, output_fired_time: 9.0764980316, propagate_forward_time: 378.8719787598, propagate_backward_time: 615.0681762695, weight_update_time: 0.3308190405, other_time: 14.3241577148
epoch time: 79.9811248779s, input_fired_time: 1.5801371336, output_fired_time: 0.0000000000, propagate_forward_time: 75.9619216919, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4390640259
validation[          , genome  6419] predictions:    6102/  10000 (61.02%), best:    6137/10000 (61.37%), error:     11518.57324, best error:     11290.29785 on epoch:     0, epoch:    7/25, mu: 0.6576985121, learning_rate: 0.0001592977, weight_decay: 0.0001461946

epoch time: 1040.5926513672s, input_fired_time: 25.8523406982, output_fired_time: 9.0586881638, propagate_forward_time: 377.2083740234, propagate_backward_time: 614.2272338867, weight_update_time: 0.3372672200, other_time: 14.2459716797
epoch time: 80.5108184814s, input_fired_time: 1.6270610094, output_fired_time: 0.0000000000, propagate_forward_time: 76.3272476196, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5565109253
validation[          , genome  6419] predictions:    6094/  10000 (60.94%), best:    6137/10000 (61.37%), error:     11382.77734, best error:     11290.29785 on epoch:     0, epoch:    8/25, mu: 0.6769367456, learning_rate: 0.0001475246, weight_decay: 0.0001380548

epoch time: 1043.8723144531s, input_fired_time: 25.9242649078, output_fired_time: 9.0325250626, propagate_forward_time: 378.9309082031, propagate_backward_time: 615.5878906250, weight_update_time: 0.3310315609, other_time: 14.3967285156
epoch time: 79.9671859741s, input_fired_time: 1.5917090178, output_fired_time: 0.0000000000, propagate_forward_time: 75.8592453003, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5162353516
validation[          , genome  6419] predictions:    6059/  10000 (60.59%), best:    6137/10000 (61.37%), error:     11475.56641, best error:     11290.29785 on epoch:     0, epoch:    9/25, mu: 0.6950612068, learning_rate: 0.0001366215, weight_decay: 0.0001303682

epoch time: 1225.9066162109s, input_fired_time: 25.8445301056, output_fired_time: 194.1647644043, propagate_forward_time: 377.8607482910, propagate_backward_time: 613.8817138672, weight_update_time: 0.3265384734, other_time: 14.1549072266
epoch time: 80.7995376587s, input_fired_time: 1.6252309084, output_fired_time: 0.0000000000, propagate_forward_time: 76.6881713867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4861373901
validation[          , genome  6419] predictions:    6129/  10000 (61.29%), best:    6137/10000 (61.37%), error:     11512.64063, best error:     11290.29785 on epoch:     0, epoch:   10/25, mu: 0.7121363878, learning_rate: 0.0001265242, weight_decay: 0.0001231096

epoch time: 1487.0449218750s, input_fired_time: 25.9451866150, output_fired_time: 9.0395908356, propagate_forward_time: 379.5637512207, propagate_backward_time: 1058.0692138672, weight_update_time: 0.3268683851, other_time: 14.4272460938
epoch time: 81.4614868164s, input_fired_time: 1.6044204235, output_fired_time: 0.0000000000, propagate_forward_time: 77.3277893066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5292739868
validation[          , genome  6419] predictions:    5932/  10000 (59.32%), best:    6137/10000 (61.37%), error:     11990.58887, best error:     11290.29785 on epoch:     0, epoch:   11/25, mu: 0.7282230258, learning_rate: 0.0001171732, weight_decay: 0.0001162552

epoch time: 1042.3144531250s, input_fired_time: 25.8693370819, output_fired_time: 9.0571746826, propagate_forward_time: 378.5500488281, propagate_backward_time: 614.5744018555, weight_update_time: 0.3290677667, other_time: 14.2634887695
epoch time: 80.0362319946s, input_fired_time: 1.5764162540, output_fired_time: 0.0000000000, propagate_forward_time: 75.9932174683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4665985107
validation[          , genome  6419] predictions:    6191/  10000 (61.91%), best:    6191/10000 (61.91%), error:     11213.85449, best error:     11213.85449 on epoch:    12, epoch:   12/25, mu: 0.7433783412, learning_rate: 0.0001085133, weight_decay: 0.0001097823

epoch time: 1044.3647460938s, input_fired_time: 25.8585510254, output_fired_time: 9.0688972473, propagate_forward_time: 379.0252075195, propagate_backward_time: 616.1293945313, weight_update_time: 0.3277069032, other_time: 14.2827148438
epoch time: 79.9003677368s, input_fired_time: 1.5804975033, output_fired_time: 0.0000000000, propagate_forward_time: 75.8198089600, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5000610352
validation[          , genome  6419] predictions:    6023/  10000 (60.23%), best:    6191/10000 (61.91%), error:     11470.87207, best error:     11213.85449 on epoch:    12, epoch:   13/25, mu: 0.7576562166, learning_rate: 0.0001004935, weight_decay: 0.0001036699

epoch time: 1414.4432373047s, input_fired_time: 25.8335399628, output_fired_time: 9.0340156555, propagate_forward_time: 377.6057128906, propagate_backward_time: 987.9078369141, weight_update_time: 0.3405061662, other_time: 14.0621337891
epoch time: 84.4928741455s, input_fired_time: 1.7663402557, output_fired_time: 0.0000000000, propagate_forward_time: 80.0775527954, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6489791870
validation[          , genome  6419] predictions:    5993/  10000 (59.93%), best:    6191/10000 (61.91%), error:     11672.32813, best error:     11213.85449 on epoch:    12, epoch:   14/25, mu: 0.7711074948, learning_rate: 0.0000930663, weight_decay: 0.0000978978

epoch time: 1642.5335693359s, input_fired_time: 25.9874801636, output_fired_time: 9.1048965454, propagate_forward_time: 821.5155029297, propagate_backward_time: 771.7093505859, weight_update_time: 0.3256409168, other_time: 14.2164306641
epoch time: 79.9220275879s, input_fired_time: 1.5597965717, output_fired_time: 0.0000000000, propagate_forward_time: 75.9119644165, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4502639771
validation[          , genome  6419] predictions:    6097/  10000 (60.97%), best:    6097/10000 (60.97%), error:     11205.58008, best error:     11205.58008 on epoch:    15, epoch:   15/25, mu: 0.7837800384, learning_rate: 0.0000861881, weight_decay: 0.0000924470

epoch time: 1038.9360351563s, input_fired_time: 25.7697601318, output_fired_time: 8.9911222458, propagate_forward_time: 377.1219787598, propagate_backward_time: 612.8886108398, weight_update_time: 0.3279871643, other_time: 14.1644897461
epoch time: 80.4785614014s, input_fired_time: 1.5829323530, output_fired_time: 0.0000000000, propagate_forward_time: 76.2955017090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6001281738
validation[          , genome  6419] predictions:    6012/  10000 (60.12%), best:    6097/10000 (60.97%), error:     11735.62500, best error:     11205.58008 on epoch:    15, epoch:   16/25, mu: 0.7957189083, learning_rate: 0.0000798182, weight_decay: 0.0000872998

epoch time: 1039.8558349609s, input_fired_time: 25.7640876770, output_fired_time: 9.0301713943, propagate_forward_time: 377.5374450684, propagate_backward_time: 613.2373657227, weight_update_time: 0.3319708407, other_time: 14.2868041992
epoch time: 80.0541534424s, input_fired_time: 1.5808084011, output_fired_time: 0.0000000000, propagate_forward_time: 76.0119018555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4614410400
validation[          , genome  6419] predictions:    6087/  10000 (60.87%), best:    6097/10000 (60.97%), error:     11555.31738, best error:     11205.58008 on epoch:    15, epoch:   17/25, mu: 0.8069666028, learning_rate: 0.0000739191, weight_decay: 0.0000824391

epoch time: 1039.8822021484s, input_fired_time: 25.8026733398, output_fired_time: 8.9897937775, propagate_forward_time: 377.4546203613, propagate_backward_time: 613.3894042969, weight_update_time: 0.3282964528, other_time: 14.2457275391
epoch time: 79.7225112915s, input_fired_time: 1.5667867661, output_fired_time: 0.0000000000, propagate_forward_time: 75.7514343262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4042892456
validation[          , genome  6419] predictions:    6326/  10000 (63.26%), best:    6326/10000 (63.26%), error:     10702.18555, best error:     10702.18555 on epoch:    18, epoch:   18/25, mu: 0.8175631166, learning_rate: 0.0000684560, weight_decay: 0.0000778491

epoch time: 2000.8041992188s, input_fired_time: 25.9592361450, output_fired_time: 8.9808101654, propagate_forward_time: 1119.1722412109, propagate_backward_time: 832.4458618164, weight_update_time: 0.3260491490, other_time: 14.2460327148
epoch time: 82.8371582031s, input_fired_time: 1.6387958527, output_fired_time: 0.0000000000, propagate_forward_time: 78.6813812256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5169830322
validation[          , genome  6419] predictions:    6128/  10000 (61.28%), best:    6326/10000 (63.26%), error:     11130.22852, best error:     10702.18555 on epoch:    18, epoch:   19/25, mu: 0.8275461793, learning_rate: 0.0000633966, weight_decay: 0.0000735146

epoch time: 1040.4261474609s, input_fired_time: 25.8698329926, output_fired_time: 9.0124015808, propagate_forward_time: 377.8096008301, propagate_backward_time: 613.5162353516, weight_update_time: 0.3443441093, other_time: 14.2181396484
epoch time: 79.9979324341s, input_fired_time: 1.5975760221, output_fired_time: 0.0000000000, propagate_forward_time: 75.8511047363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5492553711
validation[          , genome  6419] predictions:    6375/  10000 (63.75%), best:    6326/10000 (63.26%), error:     10726.91992, best error:     10702.18555 on epoch:    18, epoch:   20/25, mu: 0.8369512558, learning_rate: 0.0000587112, weight_decay: 0.0000694215

epoch time: 1037.5317382813s, input_fired_time: 25.7667484283, output_fired_time: 9.0284147263, propagate_forward_time: 376.7315673828, propagate_backward_time: 611.8504638672, weight_update_time: 0.3256774843, other_time: 14.1545410156
epoch time: 79.7130126953s, input_fired_time: 1.5671715736, output_fired_time: 0.0000000000, propagate_forward_time: 75.7291488647, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4166946411
validation[          , genome  6419] predictions:    6111/  10000 (61.11%), best:    6326/10000 (63.26%), error:     11356.31445, best error:     10702.18555 on epoch:    18, epoch:   21/25, mu: 0.8458118439, learning_rate: 0.0000543720, weight_decay: 0.0000655563

epoch time: 1185.7425537109s, input_fired_time: 29.7743549347, output_fired_time: 10.2903566360, propagate_forward_time: 431.7470092773, propagate_backward_time: 697.6196289063, weight_update_time: 0.3587478995, other_time: 16.3110961914
epoch time: 99.9770889282s, input_fired_time: 2.1031672955, output_fired_time: 0.0000000000, propagate_forward_time: 94.7271347046, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1467895508
validation[          , genome  6419] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10682.41602, best error:     10682.41602 on epoch:    22, epoch:   22/25, mu: 0.8541594744, learning_rate: 0.0000503536, weight_decay: 0.0000619062

epoch time: 1290.7707519531s, input_fired_time: 32.1848297119, output_fired_time: 11.3963937759, propagate_forward_time: 467.9369201660, propagate_backward_time: 760.6414794922, weight_update_time: 0.4190379977, other_time: 18.6112060547
epoch time: 110.5678482056s, input_fired_time: 2.2581026554, output_fired_time: 0.0000000000, propagate_forward_time: 104.7552947998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5544509888
validation[          , genome  6419] predictions:    6245/  10000 (62.45%), best:    6388/10000 (63.88%), error:     11014.61914, best error:     10682.41602 on epoch:    22, epoch:   23/25, mu: 0.8620238304, learning_rate: 0.0000466321, weight_decay: 0.0000584594

epoch time: 1275.2939453125s, input_fired_time: 32.1363372803, output_fired_time: 11.2125539780, propagate_forward_time: 464.0421142578, propagate_backward_time: 750.0720825195, weight_update_time: 0.3686217964, other_time: 17.8308715820
epoch time: 111.4018859863s, input_fired_time: 2.1483995914, output_fired_time: 0.0000000000, propagate_forward_time: 105.6845474243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5689392090
validation[          , genome  6419] predictions:    6238/  10000 (62.38%), best:    6388/10000 (63.88%), error:     11043.56934, best error:     10682.41602 on epoch:    22, epoch:   24/25, mu: 0.8694328666, learning_rate: 0.0000431857, weight_decay: 0.0000552046

epoch time: 1060.2108154297s, input_fired_time: 26.4889411926, output_fired_time: 9.2275199890, propagate_forward_time: 384.3596496582, propagate_backward_time: 625.3415527344, weight_update_time: 0.3348132968, other_time: 14.7932128906
epoch time: 335.3815307617s, input_fired_time: 1.5778051615, output_fired_time: 0.0000000000, propagate_forward_time: 331.3364562988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4672546387
validation[          , genome  6419] predictions:    6214/  10000 (62.14%), best:    6388/10000 (63.88%), error:     11172.21094, best error:     10682.41602 on epoch:    22, epoch:   25/25, mu: 0.8764129877, learning_rate: 0.0000399939, weight_decay: 0.0000521309

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 398.4335021973s, input_fired_time: 7.9613742828, output_fired_time: 0.0000000000, propagate_forward_time: 377.3800964355, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.0920410156
best training[          , genome  6419] predictions:   33742/  50000 (67.48%), best:    6388/10000 (63.88%), error:     48775.25391, best error:     10682.41602 on epoch:    22, epoch:   26/25, mu: 0.8829889894, learning_rate: 0.0000370381, weight_decay: 0.0000492284
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 81.1111145020s, input_fired_time: 1.5970202684, output_fired_time: 0.0000000000, propagate_forward_time: 77.0693206787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4447708130
   testing[          , genome  6419] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10682.41602, best error:     10682.41602 on epoch:    22, epoch:   26/25, mu: 0.8829889894, learning_rate: 0.0000370381, weight_decay: 0.0000492284
backpropagation finished successfully!
00:37:13 (9024): called boinc_finish(0)

</stderr_txt>
]]>