Nameexact_genome_1542229504_179_5954_1
Workunit4090153
Created15 Nov 2018, 6:11:52 UTC
Sent15 Nov 2018, 6:11:58 UTC
Report deadline2 Jan 2019, 17:20:00 UTC
Received18 Nov 2018, 14:19:18 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID72866
Run time1 days 16 hours 11 min 46 sec
CPU time1 days 15 hours 38 min 22 sec
Validate stateValid
Credit12,115.98
Device peak FLOPS2.78 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size552.68 MB
Peak swap size550.91 MB
Peak disk usage12.74 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542229504_179_5954.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542229504_179_5954_1_r192446899_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542229504_179_5954.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.513285
read mu: 0.513285
read mu_delta: 0.957716
read initial_learning_rate: 0.00327527
read learning_rate: 0.00327527
read learning_rate_delta: 0.918657
read initial_weight_decay: 0.000408505
read weight_decay: 0.000408505
read weight_decay_delta: 0.945165
read batch_size: 109
read epsilon: 1e-07
read alpha: 0.113522
read input_dropout_probability: 0.00118054
read hidden_dropout_probability: 0.11859
read velocity_reset: 1831
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5954
read normal distribution: '1 -2.01429486274719 1.53504621982574'
generator_str: '851178033'
read generator: 851178033
read generated_by_map:
1 change_size 1
reading 87 nodes.
reading 427 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1360627964
post shuffle 1: 1082089631
epoch time: 2545.27734375s, input_fired_time: 27.518970489502, output_fired_time: 15.8456630706787, propagate_forward_time: 1031.74865722656, propagate_backward_time: 1449.48620605469, weight_update_time: 0.926026046276093, other_time: 20.677734375
epoch time: 213.743240356445s, input_fired_time: 2.28961563110352, output_fired_time: 0, propagate_forward_time: 207.913131713867, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.54049682617188
validation[          , genome  5954] predictions:    5556/  10000 (55.56%), best:    5556/10000 (55.56%), error:     12817.44043, best error:     12817.44043 on epoch:     0, epoch:    0/50, mu: 0.5132846832, learning_rate: 0.0032752710, weight_decay: 0.0004085052

epoch time: 2677.9924316406s, input_fired_time: 29.1089229584, output_fired_time: 16.6517009735, propagate_forward_time: 1079.4351806641, propagate_backward_time: 1530.9730224609, weight_update_time: 0.9914464355, other_time: 21.8237304688
epoch time: 227.1614532471s, input_fired_time: 2.4519784451, output_fired_time: 0.0000000000, propagate_forward_time: 220.8672485352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8422241211
validation[          , genome  5954] predictions:    5927/  10000 (59.27%), best:    5927/10000 (59.27%), error:     11752.31836, best error:     11752.31836 on epoch:     1, epoch:    1/50, mu: 0.5334422588, learning_rate: 0.0030088499, weight_decay: 0.0003861048

epoch time: 2749.5410156250s, input_fired_time: 30.1438407898, output_fired_time: 17.0851821899, propagate_forward_time: 1111.6331787109, propagate_backward_time: 1568.0058593750, weight_update_time: 1.0093997717, other_time: 22.6729736328
epoch time: 223.0616149902s, input_fired_time: 2.4274544716, output_fired_time: 0.0000000000, propagate_forward_time: 216.8730316162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7611236572
validation[          , genome  5954] predictions:    6039/  10000 (60.39%), best:    6039/10000 (60.39%), error:     11340.31445, best error:     11340.31445 on epoch:     2, epoch:    2/50, mu: 0.5527474880, learning_rate: 0.0027641004, weight_decay: 0.0003649327

epoch time: 2566.1826171875s, input_fired_time: 27.9577903748, output_fired_time: 15.9296007156, propagate_forward_time: 1037.2359619141, propagate_backward_time: 1464.3768310547, weight_update_time: 0.9179516435, other_time: 20.6823730469
epoch time: 214.6654815674s, input_fired_time: 2.3257584572, output_fired_time: 0.0000000000, propagate_forward_time: 208.7730560303, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5666656494
validation[          , genome  5954] predictions:    6072/  10000 (60.72%), best:    6072/10000 (60.72%), error:     11296.87305, best error:     11296.87305 on epoch:     3, epoch:    3/50, mu: 0.5712363720, learning_rate: 0.0025392596, weight_decay: 0.0003449216

epoch time: 2551.8989257813s, input_fired_time: 27.8570442200, output_fired_time: 15.8307647705, propagate_forward_time: 1033.6179199219, propagate_backward_time: 1453.9019775391, weight_update_time: 0.9469425082, other_time: 20.6912841797
epoch time: 213.4160308838s, input_fired_time: 2.3248445988, output_fired_time: 0.0000000000, propagate_forward_time: 207.5374908447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5536956787
validation[          , genome  5954] predictions:    5812/  10000 (58.12%), best:    6072/10000 (60.72%), error:     11961.89551, best error:     11296.87305 on epoch:     3, epoch:    4/50, mu: 0.5889434814, learning_rate: 0.0023327081, weight_decay: 0.0003260078

epoch time: 2561.6562500000s, input_fired_time: 28.0897083282, output_fired_time: 15.8798789978, propagate_forward_time: 1035.8808593750, propagate_backward_time: 1461.1013183594, weight_update_time: 0.9213273525, other_time: 20.7045898438
epoch time: 211.7087554932s, input_fired_time: 2.3215639591, output_fired_time: 0.0000000000, propagate_forward_time: 205.8396453857, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5475463867
validation[          , genome  5954] predictions:    6120/  10000 (61.20%), best:    6120/10000 (61.20%), error:     11146.04883, best error:     11146.04883 on epoch:     5, epoch:    5/50, mu: 0.6059018373, learning_rate: 0.0021429581, weight_decay: 0.0003081311

epoch time: 2580.1745605469s, input_fired_time: 28.2080116272, output_fired_time: 16.0730037689, propagate_forward_time: 1045.4825439453, propagate_backward_time: 1469.5158691406, weight_update_time: 0.9444015026, other_time: 20.8951416016
epoch time: 213.3894195557s, input_fired_time: 2.3080663681, output_fired_time: 0.0000000000, propagate_forward_time: 207.5675964355, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5137634277
validation[          , genome  5954] predictions:    6176/  10000 (61.76%), best:    6176/10000 (61.76%), error:     11027.26367, best error:     11027.26367 on epoch:     6, epoch:    6/50, mu: 0.6221431494, learning_rate: 0.0019686429, weight_decay: 0.0002912347

epoch time: 2556.7963867188s, input_fired_time: 28.0426750183, output_fired_time: 15.9164304733, propagate_forward_time: 1038.0960693359, propagate_backward_time: 1453.9637451172, weight_update_time: 0.9140204191, other_time: 20.7773437500
epoch time: 212.9096221924s, input_fired_time: 2.3284771442, output_fired_time: 0.0000000000, propagate_forward_time: 206.9938049316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5873413086
validation[          , genome  5954] predictions:    6079/  10000 (60.79%), best:    6176/10000 (61.76%), error:     11156.30371, best error:     11027.26367 on epoch:     6, epoch:    7/50, mu: 0.6376976967, learning_rate: 0.0018085071, weight_decay: 0.0002752648

epoch time: 2564.3620605469s, input_fired_time: 28.0620098114, output_fired_time: 15.8723278046, propagate_forward_time: 1036.3258056641, propagate_backward_time: 1463.3630371094, weight_update_time: 0.9289206266, other_time: 20.7388916016
epoch time: 210.8115692139s, input_fired_time: 2.3320660591, output_fired_time: 0.0000000000, propagate_forward_time: 204.9381866455, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5413208008
validation[          , genome  5954] predictions:    6190/  10000 (61.90%), best:    6190/10000 (61.90%), error:     10953.48633, best error:     10953.48633 on epoch:     8, epoch:    8/50, mu: 0.6525945067, learning_rate: 0.0016613973, weight_decay: 0.0002601707

epoch time: 2547.3034667969s, input_fired_time: 28.0035533905, output_fired_time: 15.7629528046, propagate_forward_time: 1034.7132568359, propagate_backward_time: 1448.1783447266, weight_update_time: 0.9171875119, other_time: 20.6452636719
epoch time: 211.3795318604s, input_fired_time: 2.3031408787, output_fired_time: 0.0000000000, propagate_forward_time: 205.5441436768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5322418213
validation[          , genome  5954] predictions:    6041/  10000 (60.41%), best:    6190/10000 (61.90%), error:     11206.77734, best error:     10953.48633 on epoch:     8, epoch:    9/50, mu: 0.6668614745, learning_rate: 0.0015262539, weight_decay: 0.0002459042

epoch time: 2560.9272460938s, input_fired_time: 28.1183853149, output_fired_time: 15.8231267929, propagate_forward_time: 1037.1446533203, propagate_backward_time: 1459.1647949219, weight_update_time: 0.9155818820, other_time: 20.6761474609
epoch time: 214.8679199219s, input_fired_time: 2.3643124104, output_fired_time: 0.0000000000, propagate_forward_time: 208.9333648682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5702362061
validation[          , genome  5954] predictions:    6214/  10000 (62.14%), best:    6214/10000 (62.14%), error:     10713.83105, best error:     10713.83105 on epoch:    10, epoch:   10/50, mu: 0.6805251241, learning_rate: 0.0014021035, weight_decay: 0.0002324200

epoch time: 2561.1699218750s, input_fired_time: 28.1280574799, output_fired_time: 15.9290285110, propagate_forward_time: 1036.1920166016, propagate_backward_time: 1460.2802734375, weight_update_time: 0.9126093388, other_time: 20.6405029297
epoch time: 212.6486663818s, input_fired_time: 2.3316774368, output_fired_time: 0.0000000000, propagate_forward_time: 206.7948150635, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5221710205
validation[          , genome  5954] predictions:    6073/  10000 (60.73%), best:    6214/10000 (62.14%), error:     11155.72461, best error:     10713.83105 on epoch:    10, epoch:   11/50, mu: 0.6936110258, learning_rate: 0.0012880518, weight_decay: 0.0002196752

epoch time: 2560.4631347656s, input_fired_time: 28.2564735413, output_fired_time: 15.8632383347, propagate_forward_time: 1040.1794433594, propagate_backward_time: 1455.4520263672, weight_update_time: 0.9149563313, other_time: 20.7117919922
epoch time: 210.6025085449s, input_fired_time: 2.3329558372, output_fired_time: 0.0000000000, propagate_forward_time: 204.7359008789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5336456299
validation[          , genome  5954] predictions:    6261/  10000 (62.61%), best:    6261/10000 (62.61%), error:     10686.76953, best error:     10686.76953 on epoch:    12, epoch:   12/50, mu: 0.7061436176, learning_rate: 0.0011832776, weight_decay: 0.0002076293

epoch time: 2564.6311035156s, input_fired_time: 28.1381511688, output_fired_time: 15.8622922897, propagate_forward_time: 1037.8059082031, propagate_backward_time: 1462.1402587891, weight_update_time: 0.9276398420, other_time: 20.6844482422
epoch time: 214.1042480469s, input_fired_time: 2.3546280861, output_fired_time: 0.0000000000, propagate_forward_time: 208.1780242920, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5715942383
validation[          , genome  5954] predictions:    6320/  10000 (63.20%), best:    6320/10000 (63.20%), error:     10638.11523, best error:     10638.11523 on epoch:    13, epoch:   13/50, mu: 0.7181462646, learning_rate: 0.0010870260, weight_decay: 0.0001962439

epoch time: 2579.6240234375s, input_fired_time: 28.2994785309, output_fired_time: 15.9441671371, propagate_forward_time: 1043.4885253906, propagate_backward_time: 1470.8997802734, weight_update_time: 1.1229133606, other_time: 20.9920654297
epoch time: 216.1197204590s, input_fired_time: 2.3672614098, output_fired_time: 0.0000000000, propagate_forward_time: 210.1352844238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6171722412
validation[          , genome  5954] predictions:    6043/  10000 (60.43%), best:    6320/10000 (63.20%), error:     11247.60742, best error:     10638.11523 on epoch:    13, epoch:   14/50, mu: 0.7296413779, learning_rate: 0.0009986038, weight_decay: 0.0001854829

epoch time: 2556.5593261719s, input_fired_time: 28.3038349152, output_fired_time: 15.8393421173, propagate_forward_time: 1039.0792236328, propagate_backward_time: 1452.5288085938, weight_update_time: 0.9185267687, other_time: 20.8079833984
epoch time: 213.2894744873s, input_fired_time: 2.3644955158, output_fired_time: 0.0000000000, propagate_forward_time: 207.3901367188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5348358154
validation[          , genome  5954] predictions:    6293/  10000 (62.93%), best:    6320/10000 (63.20%), error:     10645.79785, best error:     10638.11523 on epoch:    13, epoch:   15/50, mu: 0.7406504750, learning_rate: 0.0009173741, weight_decay: 0.0001753119

epoch time: 2558.7082519531s, input_fired_time: 28.1558303833, output_fired_time: 15.8630361557, propagate_forward_time: 1036.6276855469, propagate_backward_time: 1457.3905029297, weight_update_time: 0.9170892835, other_time: 20.6712646484
epoch time: 217.6730804443s, input_fired_time: 2.4182965755, output_fired_time: 0.0000000000, propagate_forward_time: 211.6145477295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6402435303
validation[          , genome  5954] predictions:    6260/  10000 (62.60%), best:    6260/10000 (62.60%), error:     10634.54883, best error:     10634.54883 on epoch:    16, epoch:   16/50, mu: 0.7511940598, learning_rate: 0.0008427519, weight_decay: 0.0001656987

epoch time: 2549.7580566406s, input_fired_time: 28.1698684692, output_fired_time: 15.7685422897, propagate_forward_time: 1037.1602783203, propagate_backward_time: 1447.8707275391, weight_update_time: 0.9121034145, other_time: 20.7885742188
epoch time: 216.1955108643s, input_fired_time: 2.3657939434, output_fired_time: 0.0000000000, propagate_forward_time: 210.2662658691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5634460449
validation[          , genome  5954] predictions:    6396/  10000 (63.96%), best:    6396/10000 (63.96%), error:     10552.52246, best error:     10552.52246 on epoch:    17, epoch:   17/50, mu: 0.7612918019, learning_rate: 0.0007741998, weight_decay: 0.0001566126

epoch time: 2623.1865234375s, input_fired_time: 29.9955902100, output_fired_time: 16.2044200897, propagate_forward_time: 1064.9653320313, propagate_backward_time: 1490.3927001953, weight_update_time: 0.9365569949, other_time: 21.6285400391
epoch time: 214.5281219482s, input_fired_time: 2.3554444313, output_fired_time: 0.0000000000, propagate_forward_time: 208.6161956787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5564880371
validation[          , genome  5954] predictions:    6261/  10000 (62.61%), best:    6396/10000 (63.96%), error:     10596.22656, best error:     10552.52246 on epoch:    17, epoch:   18/50, mu: 0.7709625363, learning_rate: 0.0007112239, weight_decay: 0.0001480247

epoch time: 2564.0500488281s, input_fired_time: 28.2081050873, output_fired_time: 15.8885908127, propagate_forward_time: 1040.9417724609, propagate_backward_time: 1458.3508300781, weight_update_time: 0.9158531427, other_time: 20.6607666016
epoch time: 212.5567779541s, input_fired_time: 2.3557553291, output_fired_time: 0.0000000000, propagate_forward_time: 206.6794281006, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5215911865
validation[          , genome  5954] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10255.06055, best error:     10255.06055 on epoch:    19, epoch:   19/50, mu: 0.7802243829, learning_rate: 0.0006533706, weight_decay: 0.0001399078

epoch time: 2563.1979980469s, input_fired_time: 28.2730236053, output_fired_time: 15.8651618958, propagate_forward_time: 1037.8120117188, propagate_backward_time: 1460.4783935547, weight_update_time: 0.9182992578, other_time: 20.7694091797
epoch time: 215.1945343018s, input_fired_time: 2.3549373150, output_fired_time: 0.0000000000, propagate_forward_time: 209.2492065430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5903930664
validation[          , genome  5954] predictions:    6275/  10000 (62.75%), best:    6437/10000 (64.37%), error:     10663.33496, best error:     10255.06055 on epoch:    19, epoch:   20/50, mu: 0.7890945673, learning_rate: 0.0006002234, weight_decay: 0.0001322359

epoch time: 2569.2924804688s, input_fired_time: 28.6792030334, output_fired_time: 15.9571647644, propagate_forward_time: 1042.9337158203, propagate_backward_time: 1460.9067382813, weight_update_time: 0.9176774621, other_time: 20.8155517578
epoch time: 232.6210021973s, input_fired_time: 2.5970268250, output_fired_time: 0.0000000000, propagate_forward_time: 226.1112060547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9127655029
validation[          , genome  5954] predictions:    6290/  10000 (62.90%), best:    6437/10000 (64.37%), error:     10617.18750, best error:     10255.06055 on epoch:    19, epoch:   21/50, mu: 0.7975897193, learning_rate: 0.0005513993, weight_decay: 0.0001249848

epoch time: 2591.1704101563s, input_fired_time: 29.4829978943, output_fired_time: 16.1258010864, propagate_forward_time: 1047.2947998047, propagate_backward_time: 1477.3918457031, weight_update_time: 0.9249168634, other_time: 20.8751220703
epoch time: 232.0634002686s, input_fired_time: 2.5654716492, output_fired_time: 0.0000000000, propagate_forward_time: 225.5806884766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9172363281
validation[          , genome  5954] predictions:    6451/  10000 (64.51%), best:    6451/10000 (64.51%), error:     10185.99805, best error:     10185.99805 on epoch:    22, epoch:   22/50, mu: 0.8057256341, learning_rate: 0.0005065466, weight_decay: 0.0001181312

epoch time: 2679.4472656250s, input_fired_time: 30.2973728180, output_fired_time: 16.1923542023, propagate_forward_time: 1083.6756591797, propagate_backward_time: 1526.0456542969, weight_update_time: 0.9595921636, other_time: 23.2362060547
epoch time: 225.1946105957s, input_fired_time: 2.4988503456, output_fired_time: 0.0000000000, propagate_forward_time: 218.8588562012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8368988037
validation[          , genome  5954] predictions:    6351/  10000 (63.51%), best:    6451/10000 (64.51%), error:     10350.45605, best error:     10185.99805 on epoch:    22, epoch:   23/50, mu: 0.8135175109, learning_rate: 0.0004653425, weight_decay: 0.0001116535

epoch time: 2690.1811523438s, input_fired_time: 29.7200679779, output_fired_time: 16.9168624878, propagate_forward_time: 1086.7637939453, propagate_backward_time: 1534.7286376953, weight_update_time: 0.9598247409, other_time: 22.0520019531
epoch time: 214.1301879883s, input_fired_time: 2.3604235649, output_fired_time: 0.0000000000, propagate_forward_time: 208.2372283936, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5325317383
validation[          , genome  5954] predictions:    6498/  10000 (64.98%), best:    6498/10000 (64.98%), error:     10092.87500, best error:     10092.87500 on epoch:    24, epoch:   24/50, mu: 0.8209799528, learning_rate: 0.0004274900, weight_decay: 0.0001055309

epoch time: 2550.8666992188s, input_fired_time: 28.2935829163, output_fired_time: 15.8609924316, propagate_forward_time: 1034.9621582031, propagate_backward_time: 1450.9808349609, weight_update_time: 0.9246821404, other_time: 20.7689208984
epoch time: 216.1699829102s, input_fired_time: 2.3945338726, output_fired_time: 0.0000000000, propagate_forward_time: 210.2213287354, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5541229248
validation[          , genome  5954] predictions:    6361/  10000 (63.61%), best:    6498/10000 (64.98%), error:     10416.84570, best error:     10092.87500 on epoch:    24, epoch:   25/50, mu: 0.8281268477, learning_rate: 0.0003927166, weight_decay: 0.0000997441

epoch time: 2551.6367187500s, input_fired_time: 28.1988544464, output_fired_time: 15.8345117569, propagate_forward_time: 1034.2570800781, propagate_backward_time: 1452.7100830078, weight_update_time: 0.9213212729, other_time: 20.6361083984
epoch time: 213.9952392578s, input_fired_time: 2.3846905231, output_fired_time: 0.0000000000, propagate_forward_time: 208.0419311523, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5686187744
validation[          , genome  5954] predictions:    6492/  10000 (64.92%), best:    6492/10000 (64.92%), error:     10087.18652, best error:     10087.18652 on epoch:    26, epoch:   26/50, mu: 0.8349715471, learning_rate: 0.0003607718, weight_decay: 0.0000942747

epoch time: 2547.5957031250s, input_fired_time: 28.1408596039, output_fired_time: 15.8424043655, propagate_forward_time: 1033.2087402344, propagate_backward_time: 1449.7130126953, weight_update_time: 0.9193014503, other_time: 20.6907958984
epoch time: 213.9887084961s, input_fired_time: 2.3892917633, output_fired_time: 0.0000000000, propagate_forward_time: 208.0263519287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5730590820
validation[          , genome  5954] predictions:    6417/  10000 (64.17%), best:    6492/10000 (64.92%), error:     10265.05176, best error:     10087.18652 on epoch:    26, epoch:   27/50, mu: 0.8415268064, learning_rate: 0.0003314254, weight_decay: 0.0000891051

epoch time: 2752.4396972656s, input_fired_time: 30.4806842804, output_fired_time: 17.0397911072, propagate_forward_time: 1111.3646240234, propagate_backward_time: 1570.8978271484, weight_update_time: 1.0052433014, other_time: 22.6567382813
epoch time: 211.3437805176s, input_fired_time: 2.3668963909, output_fired_time: 0.0000000000, propagate_forward_time: 205.4442901611, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5325927734
validation[          , genome  5954] predictions:    6452/  10000 (64.52%), best:    6492/10000 (64.92%), error:     10102.03027, best error:     10087.18652 on epoch:    26, epoch:   28/50, mu: 0.8478049040, learning_rate: 0.0003044662, weight_decay: 0.0000842190

epoch time: 2551.6584472656s, input_fired_time: 28.3472537994, output_fired_time: 15.7950181961, propagate_forward_time: 1036.7122802734, propagate_backward_time: 1450.1002197266, weight_update_time: 0.9109845161, other_time: 20.7038574219
epoch time: 216.0648651123s, input_fired_time: 2.4281949997, output_fired_time: 0.0000000000, propagate_forward_time: 210.0566253662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5800476074
validation[          , genome  5954] predictions:    6513/  10000 (65.13%), best:    6513/10000 (65.13%), error:      9978.64453, best error:      9978.64453 on epoch:    29, epoch:   29/50, mu: 0.8538175225, learning_rate: 0.0002796999, weight_decay: 0.0000796009

epoch time: 2657.6259765625s, input_fired_time: 29.3875293732, output_fired_time: 16.4534072876, propagate_forward_time: 1076.5048828125, propagate_backward_time: 1513.5693359375, weight_update_time: 0.9590058923, other_time: 21.7109375000
epoch time: 226.3726043701s, input_fired_time: 2.5468888283, output_fired_time: 0.0000000000, propagate_forward_time: 219.9629974365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8627166748
validation[          , genome  5954] predictions:    6468/  10000 (64.68%), best:    6513/10000 (65.13%), error:     10137.15430, best error:      9978.64453 on epoch:    29, epoch:   30/50, mu: 0.8595758677, learning_rate: 0.0002569482, weight_decay: 0.0000752359

epoch time: 2741.2973632813s, input_fired_time: 30.5058231354, output_fired_time: 17.2080612183, propagate_forward_time: 1107.6513671875, propagate_backward_time: 1563.0364990234, weight_update_time: 0.9882558584, other_time: 22.8956298828
epoch time: 216.2118530273s, input_fired_time: 2.4570930004, output_fired_time: 0.0000000000, propagate_forward_time: 210.1141967773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6405639648
validation[          , genome  5954] predictions:    6478/  10000 (64.78%), best:    6513/10000 (65.13%), error:     10140.96582, best error:      9978.64453 on epoch:    29, epoch:   31/50, mu: 0.8650907278, learning_rate: 0.0002360472, weight_decay: 0.0000711104

epoch time: 2746.0839843750s, input_fired_time: 30.5428085327, output_fired_time: 17.2440052032, propagate_forward_time: 1110.2889404297, propagate_backward_time: 1565.3540039063, weight_update_time: 0.9928544760, other_time: 22.6544189453
epoch time: 211.6903228760s, input_fired_time: 2.3697764874, output_fired_time: 0.0000000000, propagate_forward_time: 205.7821502686, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5383911133
validation[          , genome  5954] predictions:    6515/  10000 (65.15%), best:    6513/10000 (65.13%), error:     10006.07910, best error:      9978.64453 on epoch:    29, epoch:   32/50, mu: 0.8703724146, learning_rate: 0.0002168464, weight_decay: 0.0000672110

epoch time: 2550.8166503906s, input_fired_time: 28.3308391571, output_fired_time: 15.7600517273, propagate_forward_time: 1038.3632812500, propagate_backward_time: 1447.6898193359, weight_update_time: 0.9215766788, other_time: 20.6727294922
epoch time: 212.7294006348s, input_fired_time: 2.4016742706, output_fired_time: 0.0000000000, propagate_forward_time: 206.7704772949, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5572509766
validation[          , genome  5954] predictions:    6539/  10000 (65.39%), best:    6513/10000 (65.13%), error:      9981.17188, best error:      9978.64453 on epoch:    29, epoch:   33/50, mu: 0.8754307628, learning_rate: 0.0001992074, weight_decay: 0.0000635255

epoch time: 2550.5390625000s, input_fired_time: 28.2139186859, output_fired_time: 15.8528795242, propagate_forward_time: 1034.0241699219, propagate_backward_time: 1451.8308105469, weight_update_time: 0.9147064686, other_time: 20.6174316406
epoch time: 211.9165954590s, input_fired_time: 2.3830053806, output_fired_time: 0.0000000000, propagate_forward_time: 206.0092010498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5243835449
validation[          , genome  5954] predictions:    6505/  10000 (65.05%), best:    6513/10000 (65.13%), error:     10045.62598, best error:      9978.64453 on epoch:    29, epoch:   34/50, mu: 0.8802752495, learning_rate: 0.0001830032, weight_decay: 0.0000600421

epoch time: 2599.2521972656s, input_fired_time: 28.9750900269, output_fired_time: 16.0361099243, propagate_forward_time: 1055.4277343750, propagate_backward_time: 1466.1878662109, weight_update_time: 0.9133596420, other_time: 32.6253662109
epoch time: 215.0439147949s, input_fired_time: 2.3971965313, output_fired_time: 0.0000000000, propagate_forward_time: 206.9102783203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7364349365
validation[          , genome  5954] predictions:    6529/  10000 (65.29%), best:    6513/10000 (65.13%), error:     10009.68066, best error:      9978.64453 on epoch:    29, epoch:   35/50, mu: 0.8849148750, learning_rate: 0.0001681172, weight_decay: 0.0000567497

epoch time: 2575.9953613281s, input_fired_time: 28.5580654144, output_fired_time: 15.9816131592, propagate_forward_time: 1043.4926757813, propagate_backward_time: 1467.1757812500, weight_update_time: 0.9254238605, other_time: 20.7871093750
epoch time: 216.0794830322s, input_fired_time: 2.4511880875, output_fired_time: 0.0000000000, propagate_forward_time: 210.0076293945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6206665039
validation[          , genome  5954] predictions:    6605/  10000 (66.05%), best:    6605/10000 (66.05%), error:      9815.50488, best error:      9815.50488 on epoch:    36, epoch:   36/50, mu: 0.8893583417, learning_rate: 0.0001544420, weight_decay: 0.0000536378

epoch time: 2559.5466308594s, input_fired_time: 28.2984275818, output_fired_time: 15.8508939743, propagate_forward_time: 1034.4741210938, propagate_backward_time: 1460.2464599609, weight_update_time: 0.9193779826, other_time: 20.6768798828
epoch time: 212.5643463135s, input_fired_time: 2.3835682869, output_fired_time: 0.0000000000, propagate_forward_time: 206.6530456543, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5277252197
validation[          , genome  5954] predictions:    6536/  10000 (65.36%), best:    6605/10000 (66.05%), error:      9931.77539, best error:      9815.50488 on epoch:    36, epoch:   37/50, mu: 0.8936138749, learning_rate: 0.0001418792, weight_decay: 0.0000506966

epoch time: 2538.8830566406s, input_fired_time: 28.1083431244, output_fired_time: 15.7063531876, propagate_forward_time: 1026.9971923828, propagate_backward_time: 1447.3684082031, weight_update_time: 0.9233710766, other_time: 20.7027587891
epoch time: 216.5551605225s, input_fired_time: 2.3825559616, output_fired_time: 0.0000000000, propagate_forward_time: 210.6394042969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5332031250
validation[          , genome  5954] predictions:    6546/  10000 (65.46%), best:    6605/10000 (66.05%), error:      9980.12402, best error:      9815.50488 on epoch:    36, epoch:   38/50, mu: 0.8976894617, learning_rate: 0.0001303382, weight_decay: 0.0000479166

epoch time: 2561.6650390625s, input_fired_time: 28.3931274414, output_fired_time: 15.9004745483, propagate_forward_time: 1037.7885742188, propagate_backward_time: 1458.7847900391, weight_update_time: 0.9122206569, other_time: 20.7982177734
epoch time: 211.2378997803s, input_fired_time: 2.4004735947, output_fired_time: 0.0000000000, propagate_forward_time: 205.2939453125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5434875488
validation[          , genome  5954] predictions:    6525/  10000 (65.25%), best:    6605/10000 (66.05%), error:      9954.94434, best error:      9815.50488 on epoch:    36, epoch:   39/50, mu: 0.9015927315, learning_rate: 0.0001197361, weight_decay: 0.0000452891

epoch time: 2547.7517089844s, input_fired_time: 28.2732963562, output_fired_time: 15.8270034790, propagate_forward_time: 1035.1486816406, propagate_backward_time: 1447.8438720703, weight_update_time: 0.9210817218, other_time: 20.6590576172
epoch time: 213.5180969238s, input_fired_time: 2.3982338905, output_fired_time: 0.0000000000, propagate_forward_time: 207.5867919922, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5330657959
validation[          , genome  5954] predictions:    6544/  10000 (65.44%), best:    6544/10000 (65.44%), error:      9807.11816, best error:      9807.11816 on epoch:    40, epoch:   40/50, mu: 0.9053309560, learning_rate: 0.0001099964, weight_decay: 0.0000428057

epoch time: 2545.1225585938s, input_fired_time: 28.3586540222, output_fired_time: 15.8157386780, propagate_forward_time: 1034.9324951172, propagate_backward_time: 1445.3856201172, weight_update_time: 0.9160317183, other_time: 20.6301269531
epoch time: 213.8966674805s, input_fired_time: 2.4068453312, output_fired_time: 0.0000000000, propagate_forward_time: 207.9429168701, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5469055176
validation[          , genome  5954] predictions:    6582/  10000 (65.82%), best:    6544/10000 (65.44%), error:      9871.07129, best error:      9807.11816 on epoch:    40, epoch:   41/50, mu: 0.9089111090, learning_rate: 0.0001010489, weight_decay: 0.0000404584

epoch time: 2531.9638671875s, input_fired_time: 28.1077632904, output_fired_time: 15.7323656082, propagate_forward_time: 1027.2496337891, propagate_backward_time: 1440.3181152344, weight_update_time: 0.9078357220, other_time: 20.5560302734
epoch time: 213.1526184082s, input_fired_time: 2.4132928848, output_fired_time: 0.0000000000, propagate_forward_time: 207.2087402344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5305786133
validation[          , genome  5954] predictions:    6537/  10000 (65.37%), best:    6544/10000 (65.44%), error:      9860.64844, best error:      9807.11816 on epoch:    40, epoch:   42/50, mu: 0.9123398662, learning_rate: 0.0000928293, weight_decay: 0.0000382399

epoch time: 2567.4765625000s, input_fired_time: 28.4384269714, output_fired_time: 15.9745016098, propagate_forward_time: 1044.2839355469, propagate_backward_time: 1457.9249267578, weight_update_time: 0.9139170051, other_time: 20.8546142578
epoch time: 213.3630065918s, input_fired_time: 2.3880822659, output_fired_time: 0.0000000000, propagate_forward_time: 207.4607238770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5142059326
validation[          , genome  5954] predictions:    6629/  10000 (66.29%), best:    6544/10000 (65.44%), error:      9816.58594, best error:      9807.11816 on epoch:    40, epoch:   43/50, mu: 0.9156236649, learning_rate: 0.0000852782, weight_decay: 0.0000361430

epoch time: 2573.1967773438s, input_fired_time: 28.5126895905, output_fired_time: 15.9717769623, propagate_forward_time: 1040.1290283203, propagate_backward_time: 1467.7784423828, weight_update_time: 0.9278175831, other_time: 20.8049316406
epoch time: 220.0656433105s, input_fired_time: 2.4785473347, output_fired_time: 0.0000000000, propagate_forward_time: 213.8822631836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7048339844
validation[          , genome  5954] predictions:    6563/  10000 (65.63%), best:    6544/10000 (65.44%), error:      9970.92676, best error:      9807.11816 on epoch:    40, epoch:   44/50, mu: 0.9187685847, learning_rate: 0.0000783414, weight_decay: 0.0000341611

epoch time: 2569.2209472656s, input_fired_time: 28.7389488220, output_fired_time: 15.9776105881, propagate_forward_time: 1040.1413574219, propagate_backward_time: 1463.4566650391, weight_update_time: 0.9181597233, other_time: 20.9063720703
epoch time: 213.8459472656s, input_fired_time: 2.4271812439, output_fired_time: 0.0000000000, propagate_forward_time: 207.8505706787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5681915283
validation[          , genome  5954] predictions:    6537/  10000 (65.37%), best:    6544/10000 (65.44%), error:      9923.19922, best error:      9807.11816 on epoch:    40, epoch:   45/50, mu: 0.9217805266, learning_rate: 0.0000719689, weight_decay: 0.0000322879

epoch time: 2562.2983398438s, input_fired_time: 28.4100761414, output_fired_time: 15.8951826096, propagate_forward_time: 1036.5770263672, propagate_backward_time: 1460.6614990234, weight_update_time: 0.9309886098, other_time: 20.7543945313
epoch time: 211.8030395508s, input_fired_time: 2.3899266720, output_fired_time: 0.0000000000, propagate_forward_time: 205.8845977783, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5285186768
validation[          , genome  5954] predictions:    6603/  10000 (66.03%), best:    6603/10000 (66.03%), error:      9769.28320, best error:      9769.28320 on epoch:    46, epoch:   46/50, mu: 0.9246651530, learning_rate: 0.0000661147, weight_decay: 0.0000305174

epoch time: 2564.3825683594s, input_fired_time: 29.1155948639, output_fired_time: 16.0132160187, propagate_forward_time: 1040.8105468750, propagate_backward_time: 1457.6323242188, weight_update_time: 0.9307579994, other_time: 20.8110351563
epoch time: 211.5643463135s, input_fired_time: 2.4056262970, output_fired_time: 0.0000000000, propagate_forward_time: 205.6248168945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5339050293
validation[          , genome  5954] predictions:    6675/  10000 (66.75%), best:    6675/10000 (66.75%), error:      9648.76563, best error:      9648.76563 on epoch:    47, epoch:   47/50, mu: 0.9274277687, learning_rate: 0.0000607367, weight_decay: 0.0000288439

epoch time: 2596.6254882813s, input_fired_time: 28.6622486115, output_fired_time: 16.1549663544, propagate_forward_time: 1054.6837158203, propagate_backward_time: 1475.8267822266, weight_update_time: 0.9461064935, other_time: 21.2976074219
epoch time: 215.7955627441s, input_fired_time: 2.4385595322, output_fired_time: 0.0000000000, propagate_forward_time: 209.7315521240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6254577637
validation[          , genome  5954] predictions:    6684/  10000 (66.84%), best:    6684/10000 (66.84%), error:      9619.06055, best error:      9619.06055 on epoch:    48, epoch:   48/50, mu: 0.9300735593, learning_rate: 0.0000557962, weight_decay: 0.0000272623

epoch time: 2688.5920410156s, input_fired_time: 30.0607109070, output_fired_time: 16.7920551300, propagate_forward_time: 1084.8521728516, propagate_backward_time: 1533.2077636719, weight_update_time: 1.2669857740, other_time: 23.6793212891
epoch time: 230.3187103271s, input_fired_time: 2.6640374660, output_fired_time: 0.0000000000, propagate_forward_time: 223.7496795654, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9049987793
validation[          , genome  5954] predictions:    6625/  10000 (66.25%), best:    6625/10000 (66.25%), error:      9616.89160, best error:      9616.89160 on epoch:    49, epoch:   49/50, mu: 0.9326075315, learning_rate: 0.0000512576, weight_decay: 0.0000257673

epoch time: 2748.1557617188s, input_fired_time: 30.9502258301, output_fired_time: 17.4460468292, propagate_forward_time: 1108.5258789063, propagate_backward_time: 1568.5107421875, weight_update_time: 0.9968085289, other_time: 22.7229003906
epoch time: 232.8249511719s, input_fired_time: 2.6313624382, output_fired_time: 0.0000000000, propagate_forward_time: 226.1484069824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0451812744
validation[          , genome  5954] predictions:    6650/  10000 (66.50%), best:    6625/10000 (66.25%), error:      9726.76367, best error:      9616.89160 on epoch:    49, epoch:   50/50, mu: 0.9350343347, learning_rate: 0.0000470881, weight_decay: 0.0000243544

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1118.7558593750s, input_fired_time: 12.6215238571, output_fired_time: 0.0000000000, propagate_forward_time: 1087.2292480469, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 18.9050292969
best training[          , genome  5954] predictions:   37275/  50000 (74.55%), best:    6625/10000 (66.25%), error:     38737.86719, best error:      9616.89160 on epoch:    49, epoch:   51/50, mu: 0.9373584986, learning_rate: 0.0000432578, weight_decay: 0.0000230189
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 219.7074432373s, input_fired_time: 2.4428377151, output_fired_time: 0.0000000000, propagate_forward_time: 210.1535644531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1110382080
   testing[          , genome  5954] predictions:    6625/  10000 (66.25%), best:    6625/10000 (66.25%), error:      9616.89160, best error:      9616.89160 on epoch:    49, epoch:   51/50, mu: 0.9373584986, learning_rate: 0.0000432578, weight_decay: 0.0000230189
backpropagation finished successfully!
14:15:36 (7336): called boinc_finish(0)

</stderr_txt>
]]>