Nameexact_genome_1542649503_173_6349_1
Workunit4099585
Created20 Nov 2018, 2:56:23 UTC
Sent20 Nov 2018, 3:01:13 UTC
Report deadline30 Dec 2018, 21:04:05 UTC
Received20 Nov 2018, 20:02:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84860
Run time16 hours 35 min 42 sec
CPU time16 hours 25 min 31 sec
Validate stateValid
Credit10,188.00
Device peak FLOPS4.41 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size589.19 MB
Peak swap size591.93 MB
Peak disk usage9.61 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6349.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6349_1_r1540657257_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6349.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549379
read mu: 0.549379
read mu_delta: 0.973744
read initial_learning_rate: 0.00115369
read learning_rate: 0.00115369
read learning_rate_delta: 0.915069
read initial_weight_decay: 0.000311155
read weight_decay: 0.000311155
read weight_decay_delta: 0.914211
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.12192
read input_dropout_probability: 0.00169745
read hidden_dropout_probability: 0.0613399
read velocity_reset: 1770
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6349
read normal distribution: '0 1.18591678142548 -0.0242864247411489'
generator_str: '2140485144'
read generator: 2140485144
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 475 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 487760664
post shuffle 1: 742448003
epoch time: 2096.23022460938s, input_fired_time: 22.9341621398926, output_fired_time: 8.49130725860596, propagate_forward_time: 786.278625488281, propagate_backward_time: 1262.31628417969, weight_update_time: 0.392232894897461, other_time: 16.2100830078125
epoch time: 162.723571777344s, input_fired_time: 1.24988734722137, output_fired_time: 0, propagate_forward_time: 158.73193359375, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.74174499511719
validation[          , genome  6349] predictions:    6129/  10000 (61.29%), best:    6129/10000 (61.29%), error:     11044.06152, best error:     11044.06152 on epoch:     0, epoch:    0/25, mu: 0.5493788123, learning_rate: 0.0011536898, weight_decay: 0.0003111550

epoch time: 2110.8754882812s, input_fired_time: 22.6795940399, output_fired_time: 8.4087972641, propagate_forward_time: 792.4154052734, propagate_backward_time: 1271.2576904297, weight_update_time: 0.3787801266, other_time: 16.1140136719
epoch time: 169.5423126221s, input_fired_time: 1.2450871468, output_fired_time: 0.0000000000, propagate_forward_time: 165.4272155762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8700103760
validation[          , genome  6349] predictions:    6339/  10000 (63.39%), best:    6339/10000 (63.39%), error:     10595.34863, best error:     10595.34863 on epoch:     1, epoch:    1/25, mu: 0.5609479547, learning_rate: 0.0010557062, weight_decay: 0.0002844615

epoch time: 2094.2563476562s, input_fired_time: 22.3070602417, output_fired_time: 8.3616819382, propagate_forward_time: 778.1920776367, propagate_backward_time: 1270.2016601562, weight_update_time: 0.3573777080, other_time: 15.1938476562
epoch time: 159.4197998047s, input_fired_time: 1.1950194836, output_fired_time: 0.0000000000, propagate_forward_time: 155.5650939941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6596832275
validation[          , genome  6349] predictions:    6285/  10000 (62.85%), best:    6339/10000 (63.39%), error:     10780.43555, best error:     10595.34863 on epoch:     1, epoch:    2/25, mu: 0.5722133517, learning_rate: 0.0009660444, weight_decay: 0.0002600579

epoch time: 2052.6591796875s, input_fired_time: 22.3692340851, output_fired_time: 8.2051286697, propagate_forward_time: 770.2693481445, propagate_backward_time: 1236.7404785156, weight_update_time: 0.3622337282, other_time: 15.0749511719
epoch time: 166.7881774902s, input_fired_time: 1.1984206438, output_fired_time: 0.0000000000, propagate_forward_time: 162.9106750488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6790771484
validation[          , genome  6349] predictions:    6045/  10000 (60.45%), best:    6339/10000 (63.39%), error:     11618.10352, best error:     10595.34863 on epoch:     1, epoch:    3/25, mu: 0.5831829309, learning_rate: 0.0008839976, weight_decay: 0.0002377480

epoch time: 2103.9016113281s, input_fired_time: 22.6947536469, output_fired_time: 8.4306392670, propagate_forward_time: 786.9185791016, propagate_backward_time: 1270.6855468750, weight_update_time: 0.3644966781, other_time: 15.1719970703
epoch time: 162.1764068604s, input_fired_time: 1.2176347971, output_fired_time: 0.0000000000, propagate_forward_time: 158.3080596924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6507110596
validation[          , genome  6349] predictions:    6062/  10000 (60.62%), best:    6339/10000 (63.39%), error:     11402.83691, best error:     10595.34863 on epoch:     1, epoch:    4/25, mu: 0.5938645005, learning_rate: 0.0008089191, weight_decay: 0.0002173519

epoch time: 2110.3420410156s, input_fired_time: 22.8127326965, output_fired_time: 8.4206085205, propagate_forward_time: 792.3712768555, propagate_backward_time: 1270.9207763672, weight_update_time: 0.3768076599, other_time: 15.8165283203
epoch time: 161.5469055176s, input_fired_time: 1.1636074781, output_fired_time: 0.0000000000, propagate_forward_time: 157.3696289062, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0136718750
validation[          , genome  6349] predictions:    6187/  10000 (61.87%), best:    6339/10000 (63.39%), error:     11002.00879, best error:     10595.34863 on epoch:     1, epoch:    5/25, mu: 0.6042656302, learning_rate: 0.0007402170, weight_decay: 0.0001987056

epoch time: 2110.7509765625s, input_fired_time: 22.3309516907, output_fired_time: 8.4192905426, propagate_forward_time: 784.0140991211, propagate_backward_time: 1279.4320068359, weight_update_time: 0.3769541979, other_time: 16.5545654297
epoch time: 164.0465240479s, input_fired_time: 1.2381349802, output_fired_time: 0.0000000000, propagate_forward_time: 159.8843841553, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9240112305
validation[          , genome  6349] predictions:    6259/  10000 (62.59%), best:    6339/10000 (63.39%), error:     10956.75488, best error:     10595.34863 on epoch:     1, epoch:    6/25, mu: 0.6143936515, learning_rate: 0.0006773499, weight_decay: 0.0001816589

epoch time: 2141.1081542969s, input_fired_time: 22.9295578003, output_fired_time: 8.5111169815, propagate_forward_time: 805.6174926758, propagate_backward_time: 1287.2083740234, weight_update_time: 0.3835526407, other_time: 16.8416748047
epoch time: 167.4804840088s, input_fired_time: 1.2530915737, output_fired_time: 0.0000000000, propagate_forward_time: 163.2239074707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0034790039
validation[          , genome  6349] predictions:    6261/  10000 (62.61%), best:    6339/10000 (63.39%), error:     10842.03516, best error:     10595.34863 on epoch:     1, epoch:    7/25, mu: 0.6242557168, learning_rate: 0.0006198221, weight_decay: 0.0001660747

epoch time: 2122.3479003906s, input_fired_time: 22.9488773346, output_fired_time: 8.4567899704, propagate_forward_time: 799.4044799805, propagate_backward_time: 1274.8947753906, weight_update_time: 0.3796109557, other_time: 16.6428222656
epoch time: 169.4133453369s, input_fired_time: 1.2420161963, output_fired_time: 0.0000000000, propagate_forward_time: 165.1478118896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0235137939
validation[          , genome  6349] predictions:    6295/  10000 (62.95%), best:    6339/10000 (63.39%), error:     10658.69531, best error:     10595.34863 on epoch:     1, epoch:    8/25, mu: 0.6338588595, learning_rate: 0.0005671802, weight_decay: 0.0001518274

epoch time: 2131.6953125000s, input_fired_time: 22.6541576385, output_fired_time: 8.4918298721, propagate_forward_time: 799.5784301758, propagate_backward_time: 1284.4503173828, weight_update_time: 0.3773371577, other_time: 16.5206298828
epoch time: 177.9395751953s, input_fired_time: 1.2534874678, output_fired_time: 0.0000000000, propagate_forward_time: 173.8183441162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8677368164
validation[          , genome  6349] predictions:    6318/  10000 (63.18%), best:    6339/10000 (63.39%), error:     10654.10840, best error:     10595.34863 on epoch:     1, epoch:    9/25, mu: 0.6432098746, learning_rate: 0.0005190092, weight_decay: 0.0001388023

epoch time: 2105.6201171875s, input_fired_time: 23.1931400299, output_fired_time: 8.4386854172, propagate_forward_time: 795.4609985352, propagate_backward_time: 1262.4217529297, weight_update_time: 0.3860329986, other_time: 16.1055908203
epoch time: 164.7317352295s, input_fired_time: 1.2305305004, output_fired_time: 0.0000000000, propagate_forward_time: 160.6101989746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8910064697
validation[          , genome  6349] predictions:    6359/  10000 (63.59%), best:    6359/10000 (63.59%), error:     10524.25781, best error:     10524.25781 on epoch:    10, epoch:   10/25, mu: 0.6523153782, learning_rate: 0.0004749295, weight_decay: 0.0001268947

epoch time: 2148.7397460938s, input_fired_time: 23.4340724945, output_fired_time: 8.6090869904, propagate_forward_time: 812.4062500000, propagate_backward_time: 1287.3505859375, weight_update_time: 0.4013805389, other_time: 16.9396972656
epoch time: 168.6935119629s, input_fired_time: 1.3127435446, output_fired_time: 0.0000000000, propagate_forward_time: 164.3457031250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0350646973
validation[          , genome  6349] predictions:    6456/  10000 (64.56%), best:    6456/10000 (64.56%), error:     10391.16797, best error:     10391.16797 on epoch:    11, epoch:   11/25, mu: 0.6611817479, learning_rate: 0.0004345934, weight_decay: 0.0001160086

epoch time: 2123.3452148438s, input_fired_time: 23.0586872101, output_fired_time: 8.4354763031, propagate_forward_time: 801.5289916992, propagate_backward_time: 1273.4685058594, weight_update_time: 0.3986449838, other_time: 16.8535156250
epoch time: 162.5545806885s, input_fired_time: 1.2068866491, output_fired_time: 0.0000000000, propagate_forward_time: 158.4751892090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8724975586
validation[          , genome  6349] predictions:    6352/  10000 (63.52%), best:    6456/10000 (64.56%), error:     10478.18945, best error:     10391.16797 on epoch:    11, epoch:   12/25, mu: 0.6698153615, learning_rate: 0.0003976831, weight_decay: 0.0001060564

epoch time: 2094.4851074219s, input_fired_time: 22.4629344940, output_fired_time: 8.4197769165, propagate_forward_time: 782.0588378906, propagate_backward_time: 1265.8428955078, weight_update_time: 0.3669719696, other_time: 15.7008056641
epoch time: 161.0393981934s, input_fired_time: 1.1927289963, output_fired_time: 0.0000000000, propagate_forward_time: 157.0537567139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7929077148
validation[          , genome  6349] predictions:    6448/  10000 (64.48%), best:    6448/10000 (64.48%), error:     10367.14648, best error:     10367.14648 on epoch:    13, epoch:   13/25, mu: 0.6782222986, learning_rate: 0.0003639076, weight_decay: 0.0000969579

epoch time: 2082.5937500000s, input_fired_time: 22.5819473267, output_fired_time: 8.3403749466, propagate_forward_time: 780.1329345703, propagate_backward_time: 1255.8208007812, weight_update_time: 0.3681010008, other_time: 15.7176513672
epoch time: 159.8383483887s, input_fired_time: 1.2018076181, output_fired_time: 0.0000000000, propagate_forward_time: 155.9662475586, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6702880859
validation[          , genome  6349] predictions:    6401/  10000 (64.01%), best:    6448/10000 (64.48%), error:     10549.75977, best error:     10367.14648 on epoch:    13, epoch:   14/25, mu: 0.6864084601, learning_rate: 0.0003330007, weight_decay: 0.0000886401

epoch time: 2148.5844726562s, input_fired_time: 22.2997455597, output_fired_time: 8.2927055359, propagate_forward_time: 771.8627319336, propagate_backward_time: 1331.0869140625, weight_update_time: 0.3629570007, other_time: 15.0422363281
epoch time: 156.2911682129s, input_fired_time: 1.1779482365, output_fired_time: 0.0000000000, propagate_forward_time: 152.5664367676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5467834473
validation[          , genome  6349] predictions:    6423/  10000 (64.23%), best:    6448/10000 (64.48%), error:     10545.81152, best error:     10367.14648 on epoch:    13, epoch:   15/25, mu: 0.6943796873, learning_rate: 0.0003047187, weight_decay: 0.0000810358

epoch time: 2056.8869628906s, input_fired_time: 22.3082618713, output_fired_time: 8.3129272461, propagate_forward_time: 763.0551147461, propagate_backward_time: 1248.5668945312, weight_update_time: 0.3609482944, other_time: 14.6437988281
epoch time: 158.5438842773s, input_fired_time: 1.2161433697, output_fired_time: 0.0000000000, propagate_forward_time: 154.7488098145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5789337158
validation[          , genome  6349] predictions:    6440/  10000 (64.40%), best:    6448/10000 (64.48%), error:     10405.10059, best error:     10367.14648 on epoch:    13, epoch:   16/25, mu: 0.7021416426, learning_rate: 0.0002788388, weight_decay: 0.0000740838

epoch time: 2034.6223144531s, input_fired_time: 22.4437141418, output_fired_time: 8.2313909531, propagate_forward_time: 759.1974487305, propagate_backward_time: 1230.1408691406, weight_update_time: 0.3694476485, other_time: 14.6088867188
epoch time: 155.9836425781s, input_fired_time: 1.2234659195, output_fired_time: 0.0000000000, propagate_forward_time: 152.2886047363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4715728760
validation[          , genome  6349] predictions:    6377/  10000 (63.77%), best:    6448/10000 (64.48%), error:     10600.06152, best error:     10367.14648 on epoch:    13, epoch:   17/25, mu: 0.7096997499, learning_rate: 0.0002551568, weight_decay: 0.0000677283

epoch time: 2078.1774902344s, input_fired_time: 22.3700122833, output_fired_time: 8.3567924500, propagate_forward_time: 773.6594238281, propagate_backward_time: 1259.0662841797, weight_update_time: 0.3587261438, other_time: 14.7248535156
epoch time: 156.4751434326s, input_fired_time: 1.1642888784, output_fired_time: 0.0000000000, propagate_forward_time: 152.7984771729, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5123748779
validation[          , genome  6349] predictions:    6504/  10000 (65.04%), best:    6504/10000 (65.04%), error:     10153.65332, best error:     10153.65332 on epoch:    18, epoch:   18/25, mu: 0.7170594335, learning_rate: 0.0002334862, weight_decay: 0.0000619180

epoch time: 2085.7602539062s, input_fired_time: 22.7102184296, output_fired_time: 8.4711952209, propagate_forward_time: 781.6052856445, propagate_backward_time: 1257.0058593750, weight_update_time: 0.3764972091, other_time: 15.9677734375
epoch time: 159.1625976562s, input_fired_time: 1.2846792936, output_fired_time: 0.0000000000, propagate_forward_time: 155.0829467773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7949676514
validation[          , genome  6349] predictions:    6499/  10000 (64.99%), best:    6499/10000 (64.99%), error:     10075.09863, best error:     10075.09863 on epoch:    19, epoch:   19/25, mu: 0.7242258787, learning_rate: 0.0002136560, weight_decay: 0.0000566061

epoch time: 2090.8740234375s, input_fired_time: 23.2922935486, output_fired_time: 8.3949489594, propagate_forward_time: 795.8522338867, propagate_backward_time: 1246.6168212891, weight_update_time: 0.3926490843, other_time: 16.7176513672
epoch time: 163.7100372314s, input_fired_time: 1.2620308399, output_fired_time: 0.0000000000, propagate_forward_time: 159.5042114258, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9438018799
validation[          , genome  6349] predictions:    6474/  10000 (64.74%), best:    6499/10000 (64.99%), error:     10302.35449, best error:     10075.09863 on epoch:    19, epoch:   20/25, mu: 0.7312041521, learning_rate: 0.0001955101, weight_decay: 0.0000517500

epoch time: 2097.2109375000s, input_fired_time: 23.2447528839, output_fired_time: 8.4035701752, propagate_forward_time: 797.7194824219, propagate_backward_time: 1251.2785644531, weight_update_time: 0.3884321153, other_time: 16.5644531250
epoch time: 157.2117309570s, input_fired_time: 1.2895162106, output_fired_time: 0.0000000000, propagate_forward_time: 153.0293731689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8928375244
validation[          , genome  6349] predictions:    6465/  10000 (64.65%), best:    6499/10000 (64.99%), error:     10245.69629, best error:     10075.09863 on epoch:    19, epoch:   21/25, mu: 0.7379992008, learning_rate: 0.0001789053, weight_decay: 0.0000473104

epoch time: 2086.0290527344s, input_fired_time: 22.7941207886, output_fired_time: 8.3637037277, propagate_forward_time: 782.8809204102, propagate_backward_time: 1255.2752685547, weight_update_time: 0.3935633898, other_time: 16.7149658203
epoch time: 158.4740905762s, input_fired_time: 1.2902408838, output_fired_time: 0.0000000000, propagate_forward_time: 154.3323211670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8515319824
validation[          , genome  6349] predictions:    6481/  10000 (64.81%), best:    6499/10000 (64.99%), error:     10182.17285, best error:     10075.09863 on epoch:    19, epoch:   22/25, mu: 0.7446158528, learning_rate: 0.0001637107, weight_decay: 0.0000432517

epoch time: 2076.6706542969s, input_fired_time: 22.7856178284, output_fired_time: 8.3247194290, propagate_forward_time: 782.0892944336, propagate_backward_time: 1246.8201904297, weight_update_time: 0.3830730617, other_time: 16.6507568359
epoch time: 161.3066406250s, input_fired_time: 1.2947483063, output_fired_time: 0.0000000000, propagate_forward_time: 156.8953857422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1165008545
validation[          , genome  6349] predictions:    6541/  10000 (65.41%), best:    6541/10000 (65.41%), error:     10041.51074, best error:     10041.51074 on epoch:    23, epoch:   23/25, mu: 0.7510587573, learning_rate: 0.0001498067, weight_decay: 0.0000395412

epoch time: 2102.1569824219s, input_fired_time: 23.0923767090, output_fired_time: 8.4002552032, propagate_forward_time: 794.1383056641, propagate_backward_time: 1259.5249023438, weight_update_time: 0.3895688057, other_time: 17.0013427734
epoch time: 164.5405426025s, input_fired_time: 1.3137412071, output_fired_time: 0.0000000000, propagate_forward_time: 160.3296508789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8971557617
validation[          , genome  6349] predictions:    6512/  10000 (65.12%), best:    6541/10000 (65.41%), error:     10142.33691, best error:     10041.51074 on epoch:    23, epoch:   24/25, mu: 0.7573325038, learning_rate: 0.0001370835, weight_decay: 0.0000361490

epoch time: 2115.7583007812s, input_fired_time: 22.9010944366, output_fired_time: 8.4752426147, propagate_forward_time: 797.9278564453, propagate_backward_time: 1269.5316162109, weight_update_time: 0.3902875483, other_time: 16.9223632812
epoch time: 160.0192565918s, input_fired_time: 1.2364923954, output_fired_time: 0.0000000000, propagate_forward_time: 155.7938385010, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9889221191
validation[          , genome  6349] predictions:    6555/  10000 (65.55%), best:    6555/10000 (65.55%), error:     10009.56836, best error:     10009.56836 on epoch:    25, epoch:   25/25, mu: 0.7634415030, learning_rate: 0.0001254409, weight_decay: 0.0000330479

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 820.1784667969s, input_fired_time: 6.2993431091, output_fired_time: 0.0000000000, propagate_forward_time: 799.6173095703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 14.2618408203
best training[          , genome  6349] predictions:   36544/  50000 (73.09%), best:    6555/10000 (65.55%), error:     40875.62109, best error:     10009.56836 on epoch:    25, epoch:   26/25, mu: 0.7693901062, learning_rate: 0.0001147871, weight_decay: 0.0000302127
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 159.8561096191s, input_fired_time: 1.2151397467, output_fired_time: 0.0000000000, propagate_forward_time: 155.8964996338, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7444763184
   testing[          , genome  6349] predictions:    6555/  10000 (65.55%), best:    6555/10000 (65.55%), error:     10009.56836, best error:     10009.56836 on epoch:    25, epoch:   26/25, mu: 0.7693901062, learning_rate: 0.0001147871, weight_decay: 0.0000302127
backpropagation finished successfully!
14:47:30 (27699): called boinc_finish(0)

</stderr_txt>
]]>