Nameexact_genome_1542531604_171_6258_0
Workunit4097405
Created18 Nov 2018, 13:09:52 UTC
Sent18 Nov 2018, 13:51:25 UTC
Report deadline22 Dec 2018, 21:21:52 UTC
Received19 Nov 2018, 14:18:21 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84153
Run time15 hours 28 min 36 sec
CPU time14 hours 36 min 35 sec
Validate stateValid
Credit8,578.21
Device peak FLOPS4.74 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size596.45 MB
Peak swap size600.16 MB
Peak disk usage9.45 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_171_6258.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_171_6258_0_r970777848_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542531604_171_6258.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486783
read mu: 0.486783
read mu_delta: 0.956984
read initial_learning_rate: 0.00178742
read learning_rate: 0.00178742
read learning_rate_delta: 0.930083
read initial_weight_decay: 0.000170829
read weight_decay: 0.000170829
read weight_decay_delta: 0.933111
read batch_size: 131
read epsilon: 1e-07
read alpha: 0.101117
read input_dropout_probability: 0.0012671
read hidden_dropout_probability: 0.125051
read velocity_reset: 1226
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6258
read normal distribution: '1 0 0'
generator_str: '2126013485'
read generator: 2126013485
read generated_by_map:
1 crossover 1
reading 127 nodes.
reading 487 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2075723609
post shuffle 1: 860091895
epoch time: 2089.52319335938s, input_fired_time: 28.4390735626221, output_fired_time: 10.8651790618896, propagate_forward_time: 779.88427734375, propagate_backward_time: 1248.28112792969, weight_update_time: 0.47975766658783, other_time: 22.0535888671875
epoch time: 164.720779418945s, input_fired_time: 1.57186663150787, output_fired_time: 0, propagate_forward_time: 159.274459838867, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.87445068359375
validation[          , genome  6258] predictions:    5992/  10000 (59.92%), best:    5992/10000 (59.92%), error:     11494.61328, best error:     11494.61328 on epoch:     0, epoch:    0/25, mu: 0.4867834151, learning_rate: 0.0017874151, weight_decay: 0.0001708294

epoch time: 2100.3647460938s, input_fired_time: 28.4487304688, output_fired_time: 10.7932100296, propagate_forward_time: 787.5693359375, propagate_backward_time: 1250.3750000000, weight_update_time: 0.4714400470, other_time: 23.1784667969
epoch time: 163.9592437744s, input_fired_time: 2.6274778843, output_fired_time: 0.0000000000, propagate_forward_time: 157.4971008301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8346710205
validation[          , genome  6258] predictions:    6056/  10000 (60.56%), best:    6056/10000 (60.56%), error:     11446.38770, best error:     11446.38770 on epoch:     1, epoch:    1/25, mu: 0.5084295273, learning_rate: 0.0016624443, weight_decay: 0.0001594027

epoch time: 2102.0307617188s, input_fired_time: 28.5007114410, output_fired_time: 11.0551033020, propagate_forward_time: 774.6240234375, propagate_backward_time: 1263.3099365234, weight_update_time: 0.4717476368, other_time: 24.5408935547
epoch time: 161.7660369873s, input_fired_time: 2.5770614147, output_fired_time: 0.0000000000, propagate_forward_time: 154.3029632568, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8860168457
validation[          , genome  6258] predictions:    6267/  10000 (62.67%), best:    6267/10000 (62.67%), error:     10916.93848, best error:     10916.93848 on epoch:     2, epoch:    2/25, mu: 0.5291445255, learning_rate: 0.0015462112, weight_decay: 0.0001487404

epoch time: 2126.3891601562s, input_fired_time: 30.9640502930, output_fired_time: 10.3240423203, propagate_forward_time: 788.5515747070, propagate_backward_time: 1273.6198730469, weight_update_time: 0.5185489655, other_time: 22.9296875000
epoch time: 164.7819824219s, input_fired_time: 1.6783275604, output_fired_time: 0.0000000000, propagate_forward_time: 158.1972045898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9064483643
validation[          , genome  6258] predictions:    6307/  10000 (63.07%), best:    6307/10000 (63.07%), error:     10730.37207, best error:     10730.37207 on epoch:     3, epoch:    3/25, mu: 0.5489684939, learning_rate: 0.0014381047, weight_decay: 0.0001387912

epoch time: 2103.0075683594s, input_fired_time: 26.7498302460, output_fired_time: 10.2383556366, propagate_forward_time: 787.4025268555, propagate_backward_time: 1256.5802001953, weight_update_time: 0.4852822125, other_time: 22.0367431641
epoch time: 162.5327911377s, input_fired_time: 1.6285787821, output_fired_time: 0.0000000000, propagate_forward_time: 155.9075469971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9966583252
validation[          , genome  6258] predictions:    6185/  10000 (61.85%), best:    6307/10000 (63.07%), error:     10899.70898, best error:     10730.37207 on epoch:     3, epoch:    4/25, mu: 0.5679396987, learning_rate: 0.0013375566, weight_decay: 0.0001295076

epoch time: 2160.5549316406s, input_fired_time: 30.6792087555, output_fired_time: 11.8422765732, propagate_forward_time: 799.1364135742, propagate_backward_time: 1291.9621582031, weight_update_time: 0.5677691102, other_time: 26.9348144531
epoch time: 167.1982116699s, input_fired_time: 1.7466912270, output_fired_time: 0.0000000000, propagate_forward_time: 160.3329467773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1185760498
validation[          , genome  6258] predictions:    6258/  10000 (62.58%), best:    6307/10000 (63.07%), error:     10775.94922, best error:     10730.37207 on epoch:     3, epoch:    5/25, mu: 0.5860948563, learning_rate: 0.0012440387, weight_decay: 0.0001208449

epoch time: 2160.3247070312s, input_fired_time: 27.9937534332, output_fired_time: 11.7208023071, propagate_forward_time: 803.8814086914, propagate_backward_time: 1290.9196777344, weight_update_time: 0.5192313194, other_time: 25.8093261719
epoch time: 163.5940856934s, input_fired_time: 1.7097675800, output_fired_time: 0.0000000000, propagate_forward_time: 156.9571228027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9272003174
validation[          , genome  6258] predictions:    6324/  10000 (63.24%), best:    6324/10000 (63.24%), error:     10648.68652, best error:     10648.68652 on epoch:     6, epoch:    6/25, mu: 0.6034690738, learning_rate: 0.0011570592, weight_decay: 0.0001127617

epoch time: 2153.1787109375s, input_fired_time: 30.4855060577, output_fired_time: 12.6004877090, propagate_forward_time: 798.6916503906, propagate_backward_time: 1285.5405273438, weight_update_time: 1.5237867832, other_time: 25.8603515625
epoch time: 166.3582611084s, input_fired_time: 1.7105723619, output_fired_time: 0.0000000000, propagate_forward_time: 160.5267639160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1209259033
validation[          , genome  6258] predictions:    6293/  10000 (62.93%), best:    6324/10000 (63.24%), error:     10702.12207, best error:     10648.68652 on epoch:     6, epoch:    7/25, mu: 0.6200959086, learning_rate: 0.0010761610, weight_decay: 0.0001052192

epoch time: 2153.6987304688s, input_fired_time: 31.3711929321, output_fired_time: 12.7798414230, propagate_forward_time: 794.5832519531, propagate_backward_time: 1291.3729248047, weight_update_time: 0.5130026937, other_time: 23.5916748047
epoch time: 163.8907012939s, input_fired_time: 2.7157101631, output_fired_time: 0.0000000000, propagate_forward_time: 157.1253356934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0496520996
validation[          , genome  6258] predictions:    6250/  10000 (62.50%), best:    6324/10000 (63.24%), error:     10756.15137, best error:     10648.68652 on epoch:     6, epoch:    8/25, mu: 0.6360075474, learning_rate: 0.0010009190, weight_decay: 0.0000981811

epoch time: 2136.8432617188s, input_fired_time: 27.5851230621, output_fired_time: 10.6748352051, propagate_forward_time: 801.8793945312, propagate_backward_time: 1272.0839843750, weight_update_time: 0.5185075402, other_time: 24.6198730469
epoch time: 167.3226928711s, input_fired_time: 2.7608203888, output_fired_time: 0.0000000000, propagate_forward_time: 160.3247222900, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2371520996
validation[          , genome  6258] predictions:    6342/  10000 (63.42%), best:    6342/10000 (63.42%), error:     10555.76367, best error:     10555.76367 on epoch:     9, epoch:    9/25, mu: 0.6512347460, learning_rate: 0.0009309378, weight_decay: 0.0000916139

epoch time: 2153.2846679688s, input_fired_time: 31.3710803986, output_fired_time: 11.5145740509, propagate_forward_time: 794.1499633789, propagate_backward_time: 1292.6513671875, weight_update_time: 0.5201675296, other_time: 23.5976562500
epoch time: 165.4430694580s, input_fired_time: 1.6858389378, output_fired_time: 0.0000000000, propagate_forward_time: 159.6352081299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1220245361
validation[          , genome  6258] predictions:    6413/  10000 (64.13%), best:    6413/10000 (64.13%), error:     10453.96094, best error:     10453.96094 on epoch:    10, epoch:   10/25, mu: 0.6658068895, learning_rate: 0.0008658494, weight_decay: 0.0000854859

epoch time: 2131.7841796875s, input_fired_time: 28.2576446533, output_fired_time: 12.5848617554, propagate_forward_time: 792.5087280273, propagate_backward_time: 1273.8980712891, weight_update_time: 0.5026229024, other_time: 24.5347900391
epoch time: 162.0665893555s, input_fired_time: 1.6621522903, output_fired_time: 0.0000000000, propagate_forward_time: 156.2185516357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1858825684
validation[          , genome  6258] predictions:    6446/  10000 (64.46%), best:    6446/10000 (64.46%), error:     10353.51953, best error:     10353.51953 on epoch:    11, epoch:   11/25, mu: 0.6797522306, learning_rate: 0.0008053118, weight_decay: 0.0000797678

epoch time: 2090.2363281250s, input_fired_time: 29.8389453888, output_fired_time: 11.1777257919, propagate_forward_time: 776.0929565430, propagate_backward_time: 1251.1512451172, weight_update_time: 0.4967896938, other_time: 21.9754638672
epoch time: 163.1646881104s, input_fired_time: 2.6645905972, output_fired_time: 0.0000000000, propagate_forward_time: 156.5918731689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9082183838
validation[          , genome  6258] predictions:    6299/  10000 (62.99%), best:    6446/10000 (64.46%), error:     10783.22754, best error:     10353.51953 on epoch:    11, epoch:   12/25, mu: 0.6930977106, learning_rate: 0.0007490068, weight_decay: 0.0000744322

epoch time: 2114.6804199219s, input_fired_time: 28.9644050598, output_fired_time: 11.3389043808, propagate_forward_time: 784.3208618164, propagate_backward_time: 1267.2895507812, weight_update_time: 0.5461081862, other_time: 22.7668457031
epoch time: 167.8509826660s, input_fired_time: 2.7852566242, output_fired_time: 0.0000000000, propagate_forward_time: 160.9460296631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1196899414
validation[          , genome  6258] predictions:    6484/  10000 (64.84%), best:    6484/10000 (64.84%), error:     10275.70898, best error:     10275.70898 on epoch:    13, epoch:   13/25, mu: 0.7058691382, learning_rate: 0.0006966384, weight_decay: 0.0000694535

epoch time: 2139.3208007812s, input_fired_time: 30.6483173370, output_fired_time: 10.5668067932, propagate_forward_time: 794.1110839844, propagate_backward_time: 1278.4705810547, weight_update_time: 0.5276235938, other_time: 25.5238037109
epoch time: 166.0641021729s, input_fired_time: 2.6477077007, output_fired_time: 0.0000000000, propagate_forward_time: 159.3124237061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1039733887
validation[          , genome  6258] predictions:    6466/  10000 (64.66%), best:    6484/10000 (64.84%), error:     10316.87012, best error:     10275.70898 on epoch:    13, epoch:   14/25, mu: 0.7180911899, learning_rate: 0.0006479315, weight_decay: 0.0000648078

epoch time: 2159.2617187500s, input_fired_time: 28.6949977875, output_fired_time: 10.6137304306, propagate_forward_time: 802.3476562500, propagate_backward_time: 1291.4188232422, weight_update_time: 0.5404222012, other_time: 26.1864013672
epoch time: 168.9312438965s, input_fired_time: 1.7506977320, output_fired_time: 0.0000000000, propagate_forward_time: 162.0744171143, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1061248779
validation[          , genome  6258] predictions:    6432/  10000 (64.32%), best:    6484/10000 (64.84%), error:     10433.80762, best error:     10275.70898 on epoch:    13, epoch:   15/25, mu: 0.7297874689, learning_rate: 0.0006026301, weight_decay: 0.0000604728

epoch time: 2166.5019531250s, input_fired_time: 30.9424972534, output_fired_time: 10.6167783737, propagate_forward_time: 803.8899536133, propagate_backward_time: 1296.7473144531, weight_update_time: 0.5309589505, other_time: 24.3056640625
epoch time: 164.8425903320s, input_fired_time: 1.7652842999, output_fired_time: 0.0000000000, propagate_forward_time: 157.8588867188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2184143066
validation[          , genome  6258] predictions:    6420/  10000 (64.20%), best:    6484/10000 (64.84%), error:     10424.86816, best error:     10275.70898 on epoch:    13, epoch:   16/25, mu: 0.7409806252, learning_rate: 0.0005604960, weight_decay: 0.0000564278

epoch time: 2115.3635253906s, input_fired_time: 29.2002239227, output_fired_time: 10.2639102936, propagate_forward_time: 788.0673217773, propagate_backward_time: 1263.3668212891, weight_update_time: 0.5135673881, other_time: 24.4652099609
epoch time: 167.0682373047s, input_fired_time: 1.7282885313, output_fired_time: 0.0000000000, propagate_forward_time: 160.2657775879, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0741729736
validation[          , genome  6258] predictions:    6553/  10000 (65.53%), best:    6553/10000 (65.53%), error:     10011.82129, best error:     10011.82129 on epoch:    17, epoch:   17/25, mu: 0.7516923547, learning_rate: 0.0005213078, weight_decay: 0.0000526534

epoch time: 2135.3918457031s, input_fired_time: 28.3149890900, output_fired_time: 11.1889247894, propagate_forward_time: 792.8124389648, propagate_backward_time: 1276.0876464844, weight_update_time: 0.5103689432, other_time: 26.9877929688
epoch time: 163.9440002441s, input_fired_time: 1.6735457182, output_fired_time: 0.0000000000, propagate_forward_time: 157.3573760986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9130859375
validation[          , genome  6258] predictions:    6356/  10000 (63.56%), best:    6553/10000 (65.53%), error:     10486.94434, best error:     10011.82129 on epoch:    17, epoch:   18/25, mu: 0.7619432807, learning_rate: 0.0004848595, weight_decay: 0.0000491315

epoch time: 2076.6760253906s, input_fired_time: 29.8222827911, output_fired_time: 10.0779600143, propagate_forward_time: 769.5750732422, propagate_backward_time: 1245.6198730469, weight_update_time: 0.4909871221, other_time: 21.5808105469
epoch time: 162.6151733398s, input_fired_time: 1.6886187792, output_fired_time: 0.0000000000, propagate_forward_time: 156.9175415039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0090179443
validation[          , genome  6258] predictions:    6469/  10000 (64.69%), best:    6553/10000 (65.53%), error:     10259.00879, best error:     10011.82129 on epoch:    17, epoch:   19/25, mu: 0.7717532516, learning_rate: 0.0004509596, weight_decay: 0.0000458451

epoch time: 2114.4274902344s, input_fired_time: 30.9192085266, output_fired_time: 10.2772779465, propagate_forward_time: 779.1788330078, propagate_backward_time: 1269.6119384766, weight_update_time: 0.4889223576, other_time: 24.4401855469
epoch time: 158.5862121582s, input_fired_time: 2.6402845383, output_fired_time: 0.0000000000, propagate_forward_time: 151.7149505615, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2309722900
validation[          , genome  6258] predictions:    6410/  10000 (64.10%), best:    6553/10000 (65.53%), error:     10297.68262, best error:     10011.82129 on epoch:    17, epoch:   20/25, mu: 0.7811412811, learning_rate: 0.0004194298, weight_decay: 0.0000427786

epoch time: 2112.2451171875s, input_fired_time: 27.2342109680, output_fired_time: 11.1445875168, propagate_forward_time: 789.1690063477, propagate_backward_time: 1259.0961914062, weight_update_time: 0.5183407068, other_time: 25.6013183594
epoch time: 167.8644409180s, input_fired_time: 1.7207230330, output_fired_time: 0.0000000000, propagate_forward_time: 162.0353851318, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1083374023
validation[          , genome  6258] predictions:    6544/  10000 (65.44%), best:    6553/10000 (65.53%), error:     10016.07031, best error:     10011.82129 on epoch:    17, epoch:   21/25, mu: 0.7901254296, learning_rate: 0.0003901045, weight_decay: 0.0000399171

epoch time: 2111.1589355469s, input_fired_time: 29.3403587341, output_fired_time: 12.3722696304, propagate_forward_time: 791.1964111328, propagate_backward_time: 1255.3005371094, weight_update_time: 0.4995939434, other_time: 22.9493408203
epoch time: 160.6023712158s, input_fired_time: 1.7189670801, output_fired_time: 0.0000000000, propagate_forward_time: 153.9098358154, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9735717773
validation[          , genome  6258] predictions:    6572/  10000 (65.72%), best:    6553/10000 (65.53%), error:     10079.01758, best error:     10011.82129 on epoch:    17, epoch:   22/25, mu: 0.7987231612, learning_rate: 0.0003628296, weight_decay: 0.0000372471

epoch time: 2102.7424316406s, input_fired_time: 28.2128849030, output_fired_time: 11.2369260788, propagate_forward_time: 781.2410888672, propagate_backward_time: 1260.2239990234, weight_update_time: 0.5004069209, other_time: 21.8276367188
epoch time: 166.3315734863s, input_fired_time: 1.7239639759, output_fired_time: 0.0000000000, propagate_forward_time: 160.6521301270, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9554748535
validation[          , genome  6258] predictions:    6535/  10000 (65.35%), best:    6553/10000 (65.53%), error:     10085.26660, best error:     10011.82129 on epoch:    17, epoch:   23/25, mu: 0.8069510460, learning_rate: 0.0003374616, weight_decay: 0.0000347557

epoch time: 2118.8012695312s, input_fired_time: 32.2181701660, output_fired_time: 10.3497085571, propagate_forward_time: 780.7006835938, propagate_backward_time: 1273.1594238281, weight_update_time: 0.5025650859, other_time: 22.3732910156
epoch time: 169.3593444824s, input_fired_time: 1.8324035406, output_fired_time: 0.0000000000, propagate_forward_time: 163.3162536621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2106933594
validation[          , genome  6258] predictions:    6481/  10000 (64.81%), best:    6553/10000 (65.53%), error:     10284.58984, best error:     10011.82129 on epoch:    17, epoch:   24/25, mu: 0.8148249984, learning_rate: 0.0003138673, weight_decay: 0.0000324309

epoch time: 2135.6579589844s, input_fired_time: 34.5746955872, output_fired_time: 10.6627264023, propagate_forward_time: 780.9935913086, propagate_backward_time: 1283.7498779297, weight_update_time: 0.5226869583, other_time: 25.6768798828
epoch time: 167.2143554688s, input_fired_time: 2.8155543804, output_fired_time: 0.0000000000, propagate_forward_time: 160.3234863281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0753173828
validation[          , genome  6258] predictions:    6583/  10000 (65.83%), best:    6583/10000 (65.83%), error:      9907.43945, best error:      9907.43945 on epoch:    25, epoch:   25/25, mu: 0.8223602176, learning_rate: 0.0002919227, weight_decay: 0.0000302616

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 822.6953735352s, input_fired_time: 11.0521793365, output_fired_time: 0.0000000000, propagate_forward_time: 788.2778320312, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 23.3653564453
best training[          , genome  6258] predictions:   37364/  50000 (74.73%), best:    6583/10000 (65.83%), error:     38310.26172, best error:      9907.43945 on epoch:    25, epoch:   26/25, mu: 0.8295713067, learning_rate: 0.0002715123, weight_decay: 0.0000282374
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 169.0881195068s, input_fired_time: 2.8506886959, output_fired_time: 0.0000000000, propagate_forward_time: 162.0273895264, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2100372314
   testing[          , genome  6258] predictions:    6583/  10000 (65.83%), best:    6583/10000 (65.83%), error:      9907.43945, best error:      9907.43945 on epoch:    25, epoch:   26/25, mu: 0.8295713067, learning_rate: 0.0002715123, weight_decay: 0.0000282374
backpropagation finished successfully!
08:17:35 (17098): called boinc_finish(0)

</stderr_txt>
]]>