Nameexact_genome_1542692703_178_6370_0
Workunit4100303
Created20 Nov 2018, 6:33:40 UTC
Sent20 Nov 2018, 7:07:37 UTC
Report deadline21 Jan 2019, 8:25:49 UTC
Received24 Nov 2018, 6:57:46 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID83973
Run time2 days 5 hours 7 min 46 sec
CPU time2 days 4 hours 52 min 23 sec
Validate stateValid
Credit15,513.58
Device peak FLOPS2.50 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size561.95 MB
Peak swap size581.87 MB
Peak disk usage14.61 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370_0_r609381456_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542692703_178_6370.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.486052
read mu: 0.486052
read mu_delta: 0.946414
read initial_learning_rate: 0.000636067
read learning_rate: 0.000636067
read learning_rate_delta: 0.913922
read initial_weight_decay: 0.000627951
read weight_decay: 0.000627951
read weight_decay_delta: 0.923629
read batch_size: 110
read epsilon: 1e-07
read alpha: 0.0704542
read input_dropout_probability: 0.00118852
read hidden_dropout_probability: 0.113796
read velocity_reset: 2264
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6370
read normal distribution: '1 0 0'
generator_str: '2007346138'
read generator: 2007346138
read generated_by_map:
1 enable_edge 1
reading 101 nodes.
reading 530 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 498446996
post shuffle 1: 1503383327
epoch time: 3880.85375976563s, input_fired_time: 34.2880592346191, output_fired_time: 23.2471313476563, propagate_forward_time: 1504.52783203125, propagate_backward_time: 2283.04345703125, weight_update_time: 1.8714382648468, other_time: 35.747314453125
epoch time: 306.483367919922s, input_fired_time: 2.62985992431641, output_fired_time: 0, propagate_forward_time: 298.595367431641, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.25814819335938
validation[          , genome  6370] predictions:    5495/  10000 (54.95%), best:    5495/10000 (54.95%), error:     13651.16699, best error:     13651.16699 on epoch:     0, epoch:    0/50, mu: 0.4860522151, learning_rate: 0.0006360675, weight_decay: 0.0006279508

epoch time: 3821.5690917969s, input_fired_time: 35.8508453369, output_fired_time: 21.9282073975, propagate_forward_time: 1492.7877197266, propagate_backward_time: 2237.4367675781, weight_update_time: 0.8656474352, other_time: 33.5656738281
epoch time: 302.6117858887s, input_fired_time: 2.7078831196, output_fired_time: 0.0000000000, propagate_forward_time: 293.8634948730, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0404052734
validation[          , genome  6370] predictions:    5876/  10000 (58.76%), best:    5876/10000 (58.76%), error:     12183.70117, best error:     12183.70117 on epoch:     1, epoch:    1/50, mu: 0.5130566359, learning_rate: 0.0005813160, weight_decay: 0.0005799936

epoch time: 3833.2807617188s, input_fired_time: 35.7136993408, output_fired_time: 19.6202945709, propagate_forward_time: 1490.3035888672, propagate_backward_time: 2254.8952636719, weight_update_time: 0.8456788659, other_time: 32.7478027344
epoch time: 308.1619873047s, input_fired_time: 2.6620326042, output_fired_time: 0.0000000000, propagate_forward_time: 298.1144409180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3855285645
validation[          , genome  6370] predictions:    5825/  10000 (58.25%), best:    5825/10000 (58.25%), error:     12061.86426, best error:     12061.86426 on epoch:     2, epoch:    2/50, mu: 0.5386139750, learning_rate: 0.0005312774, weight_decay: 0.0005356990

epoch time: 3825.6914062500s, input_fired_time: 38.6416091919, output_fired_time: 20.7592811584, propagate_forward_time: 1490.2154541016, propagate_backward_time: 2240.3854980469, weight_update_time: 0.8685867786, other_time: 35.6896972656
epoch time: 304.1891784668s, input_fired_time: 2.6036684513, output_fired_time: 0.0000000000, propagate_forward_time: 295.2100830078, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3754272461
validation[          , genome  6370] predictions:    5380/  10000 (53.80%), best:    5825/10000 (58.25%), error:     13176.24805, best error:     12061.86426 on epoch:     2, epoch:    3/50, mu: 0.5628018379, learning_rate: 0.0004855461, weight_decay: 0.0004947871

epoch time: 3808.0063476563s, input_fired_time: 37.6380081177, output_fired_time: 19.6265773773, propagate_forward_time: 1480.3244628906, propagate_backward_time: 2237.8234863281, weight_update_time: 0.8589506149, other_time: 32.5939941406
epoch time: 305.5462036133s, input_fired_time: 2.5770337582, output_fired_time: 0.0000000000, propagate_forward_time: 294.9989624023, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9702148438
validation[          , genome  6370] predictions:    5878/  10000 (58.78%), best:    5878/10000 (58.78%), error:     11958.82813, best error:     11958.82813 on epoch:     4, epoch:    4/50, mu: 0.5856935382, learning_rate: 0.0004437512, weight_decay: 0.0004569998

epoch time: 3808.1545410156s, input_fired_time: 34.6856651306, output_fired_time: 19.2529830933, propagate_forward_time: 1480.9217529297, propagate_backward_time: 2234.1408691406, weight_update_time: 0.8556445241, other_time: 39.1535644531
epoch time: 305.8260192871s, input_fired_time: 2.6494767666, output_fired_time: 0.0000000000, propagate_forward_time: 297.7132873535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4632568359
validation[          , genome  6370] predictions:    5478/  10000 (54.78%), best:    5878/10000 (58.78%), error:     13092.09766, best error:     11958.82813 on epoch:     4, epoch:    5/50, mu: 0.6073585749, learning_rate: 0.0004055540, weight_decay: 0.0004220983

epoch time: 3816.1979980469s, input_fired_time: 36.8282890320, output_fired_time: 23.2623443604, propagate_forward_time: 1491.8742675781, propagate_backward_time: 2230.9323730469, weight_update_time: 0.8709602356, other_time: 33.3005371094
epoch time: 305.1397705078s, input_fired_time: 3.6318047047, output_fired_time: 0.0000000000, propagate_forward_time: 294.7973937988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7105712891
validation[          , genome  6370] predictions:    5586/  10000 (55.86%), best:    5878/10000 (58.78%), error:     12740.59766, best error:     11958.82813 on epoch:     4, epoch:    6/50, mu: 0.6278626919, learning_rate: 0.0003706446, weight_decay: 0.0003898623

epoch time: 3809.5832519531s, input_fired_time: 36.7168388367, output_fired_time: 22.6130962372, propagate_forward_time: 1480.2614746094, propagate_backward_time: 2236.2026367188, weight_update_time: 0.8550888896, other_time: 33.7893066406
epoch time: 309.9433288574s, input_fired_time: 2.6136851311, output_fired_time: 0.0000000000, propagate_forward_time: 301.0936889648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2359619141
validation[          , genome  6370] predictions:    5509/  10000 (55.09%), best:    5878/10000 (58.78%), error:     13057.38281, best error:     11958.82813 on epoch:     4, epoch:    7/50, mu: 0.6472680569, learning_rate: 0.0003387403, weight_decay: 0.0003600882

epoch time: 3828.1635742188s, input_fired_time: 36.6206588745, output_fired_time: 18.9210662842, propagate_forward_time: 1492.5179443359, propagate_backward_time: 2244.4323730469, weight_update_time: 0.8713140488, other_time: 35.6716308594
epoch time: 303.6380310059s, input_fired_time: 2.5295319557, output_fired_time: 0.0000000000, propagate_forward_time: 296.3383789063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7701110840
validation[          , genome  6370] predictions:    5839/  10000 (58.39%), best:    5878/10000 (58.78%), error:     12263.33691, best error:     11958.82813 on epoch:     4, epoch:    8/50, mu: 0.6656336188, learning_rate: 0.0003095822, weight_decay: 0.0003325879

epoch time: 3813.1564941406s, input_fired_time: 41.0654830933, output_fired_time: 22.0598983765, propagate_forward_time: 1480.1485595703, propagate_backward_time: 2238.8430175781, weight_update_time: 0.8635321856, other_time: 31.0397949219
epoch time: 306.1237792969s, input_fired_time: 3.5833415985, output_fired_time: 0.0000000000, propagate_forward_time: 296.3666687012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1737670898
validation[          , genome  6370] predictions:    5856/  10000 (58.56%), best:    5878/10000 (58.78%), error:     12033.66309, best error:     11958.82813 on epoch:     4, epoch:    9/50, mu: 0.6830150485, learning_rate: 0.0002829339, weight_decay: 0.0003071878

epoch time: 3822.2875976563s, input_fired_time: 34.8245429993, output_fired_time: 20.7012729645, propagate_forward_time: 1487.2009277344, propagate_backward_time: 2247.0268554688, weight_update_time: 0.8659381270, other_time: 32.5341796875
epoch time: 308.4310607910s, input_fired_time: 3.6151854992, output_fired_time: 0.0000000000, propagate_forward_time: 298.0813903809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7344970703
validation[          , genome  6370] predictions:    5756/  10000 (57.56%), best:    5878/10000 (58.78%), error:     12326.87109, best error:     11958.82813 on epoch:     4, epoch:   10/50, mu: 0.6994650364, learning_rate: 0.0002585795, weight_decay: 0.0002837276

epoch time: 3813.5888671875s, input_fired_time: 37.4607124329, output_fired_time: 21.9275741577, propagate_forward_time: 1477.0417480469, propagate_backward_time: 2243.9240722656, weight_update_time: 0.8701874614, other_time: 33.2348632813
epoch time: 303.9703674316s, input_fired_time: 2.5596332550, output_fired_time: 0.0000000000, propagate_forward_time: 296.3945007324, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0162353516
validation[          , genome  6370] predictions:    5870/  10000 (58.70%), best:    5878/10000 (58.78%), error:     12015.41602, best error:     11958.82813 on epoch:     4, epoch:   11/50, mu: 0.7150335312, learning_rate: 0.0002363215, weight_decay: 0.0002620591

epoch time: 3834.7465820313s, input_fired_time: 37.0819778442, output_fired_time: 19.0241489410, propagate_forward_time: 1496.5943603516, propagate_backward_time: 2241.5241699219, weight_update_time: 0.8662904501, other_time: 40.5217285156
epoch time: 306.9062500000s, input_fired_time: 2.6590402126, output_fired_time: 0.0000000000, propagate_forward_time: 298.7661437988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4810791016
validation[          , genome  6370] predictions:    6031/  10000 (60.31%), best:    6031/10000 (60.31%), error:     11608.99512, best error:     11608.99512 on epoch:    12, epoch:   12/50, mu: 0.7297677994, learning_rate: 0.0002159794, weight_decay: 0.0002420454

epoch time: 3822.4768066406s, input_fired_time: 38.6137580872, output_fired_time: 22.1223392487, propagate_forward_time: 1484.7346191406, propagate_backward_time: 2240.7841796875, weight_update_time: 0.8648230433, other_time: 36.2219238281
epoch time: 303.7348327637s, input_fired_time: 2.6256425381, output_fired_time: 0.0000000000, propagate_forward_time: 295.8833007813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2258911133
validation[          , genome  6370] predictions:    5951/  10000 (59.51%), best:    6031/10000 (60.31%), error:     11860.06836, best error:     11608.99512 on epoch:    12, epoch:   13/50, mu: 0.7437125444, learning_rate: 0.0001973883, weight_decay: 0.0002235602

epoch time: 3825.5742187500s, input_fired_time: 37.6833419800, output_fired_time: 20.8938560486, propagate_forward_time: 1489.9594726563, propagate_backward_time: 2244.8203125000, weight_update_time: 0.8744488955, other_time: 32.2172851563
epoch time: 303.3482055664s, input_fired_time: 2.5667657852, output_fired_time: 0.0000000000, propagate_forward_time: 295.8472900391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9341430664
validation[          , genome  6370] predictions:    5657/  10000 (56.57%), best:    6031/10000 (60.31%), error:     12482.42578, best error:     11608.99512 on epoch:    12, epoch:   14/50, mu: 0.7569100261, learning_rate: 0.0001803975, weight_decay: 0.0002064867

epoch time: 3832.9548339844s, input_fired_time: 39.5819778442, output_fired_time: 21.1623611450, propagate_forward_time: 1482.7952880859, propagate_backward_time: 2259.0461425781, weight_update_time: 0.8710951805, other_time: 30.3688964844
epoch time: 307.6088256836s, input_fired_time: 3.5827720165, output_fired_time: 0.0000000000, propagate_forward_time: 296.9577026367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0683593750
validation[          , genome  6370] predictions:    5958/  10000 (59.58%), best:    6031/10000 (60.31%), error:     11751.99902, best error:     11608.99512 on epoch:    12, epoch:   15/50, mu: 0.7694002986, learning_rate: 0.0001648692, weight_decay: 0.0001907171

epoch time: 3821.3234863281s, input_fired_time: 35.4853897095, output_fired_time: 20.0350608826, propagate_forward_time: 1497.9100341797, propagate_backward_time: 2236.9243164063, weight_update_time: 0.8645229936, other_time: 30.9687500000
epoch time: 303.0644836426s, input_fired_time: 2.6134157181, output_fired_time: 0.0000000000, propagate_forward_time: 295.1217346191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3293457031
validation[          , genome  6370] predictions:    5968/  10000 (59.68%), best:    5968/10000 (59.68%), error:     11499.74609, best error:     11499.74609 on epoch:    16, epoch:   16/50, mu: 0.7812212706, learning_rate: 0.0001506776, weight_decay: 0.0001761519

epoch time: 3866.8581542969s, input_fired_time: 34.6268577576, output_fired_time: 23.3533020020, propagate_forward_time: 1503.5262451172, propagate_backward_time: 2273.1513671875, weight_update_time: 0.8462427258, other_time: 32.2001953125
epoch time: 306.6648254395s, input_fired_time: 3.6959259510, output_fired_time: 0.0000000000, propagate_forward_time: 296.4429931641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5259094238
validation[          , genome  6370] predictions:    6020/  10000 (60.20%), best:    5968/10000 (59.68%), error:     11557.54688, best error:     11499.74609 on epoch:    16, epoch:   17/50, mu: 0.7924088240, learning_rate: 0.0001377075, weight_decay: 0.0001626990

epoch time: 3827.5327148438s, input_fired_time: 37.4439201355, output_fired_time: 18.7398872375, propagate_forward_time: 1484.7100830078, propagate_backward_time: 2255.4848632813, weight_update_time: 0.8720517755, other_time: 31.1538085938
epoch time: 304.6353454590s, input_fired_time: 3.5964553356, output_fired_time: 0.0000000000, propagate_forward_time: 295.5650329590, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4738464355
validation[          , genome  6370] predictions:    5886/  10000 (58.86%), best:    5968/10000 (59.68%), error:     12046.28027, best error:     11499.74609 on epoch:    16, epoch:   18/50, mu: 0.8029968739, learning_rate: 0.0001258539, weight_decay: 0.0001502735

epoch time: 3826.8061523438s, input_fired_time: 38.7913017273, output_fired_time: 23.8376235962, propagate_forward_time: 1480.1166992188, propagate_backward_time: 2249.6965332031, weight_update_time: 0.8714833260, other_time: 34.3640136719
epoch time: 304.0852661133s, input_fired_time: 2.5611970425, output_fired_time: 0.0000000000, propagate_forward_time: 294.4032897949, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1207885742
validation[          , genome  6370] predictions:    5973/  10000 (59.73%), best:    5968/10000 (59.68%), error:     11699.58008, best error:     11499.74609 on epoch:    16, epoch:   19/50, mu: 0.8130175471, learning_rate: 0.0001150207, weight_decay: 0.0001387970

epoch time: 3818.8095703125s, input_fired_time: 34.6007270813, output_fired_time: 19.7019233704, propagate_forward_time: 1491.0743408203, propagate_backward_time: 2242.6232910156, weight_update_time: 0.8460644484, other_time: 30.8093261719
epoch time: 304.7370605469s, input_fired_time: 3.6069350243, output_fired_time: 0.0000000000, propagate_forward_time: 295.5482177734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5819091797
validation[          , genome  6370] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11338.86621, best error:     11338.86621 on epoch:    20, epoch:   20/50, mu: 0.8225013018, learning_rate: 0.0001051199, weight_decay: 0.0001281970

epoch time: 3824.1833496094s, input_fired_time: 36.7759284973, output_fired_time: 18.9519405365, propagate_forward_time: 1491.3911132813, propagate_backward_time: 2244.8337402344, weight_update_time: 0.8648720980, other_time: 32.2307128906
epoch time: 305.3121948242s, input_fired_time: 2.6014993191, output_fired_time: 0.0000000000, propagate_forward_time: 297.6510314941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0596618652
validation[          , genome  6370] predictions:    6146/  10000 (61.46%), best:    6146/10000 (61.46%), error:     11094.59961, best error:     11094.59961 on epoch:    21, epoch:   21/50, mu: 0.8314768672, learning_rate: 0.0000960714, weight_decay: 0.0001184064

epoch time: 3826.5563964844s, input_fired_time: 40.8399276733, output_fired_time: 20.0722236633, propagate_forward_time: 1488.1394042969, propagate_backward_time: 2246.5214843750, weight_update_time: 0.8622334599, other_time: 30.9833984375
epoch time: 306.0115966797s, input_fired_time: 3.6041595936, output_fired_time: 0.0000000000, propagate_forward_time: 293.8158264160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5916137695
validation[          , genome  6370] predictions:    6119/  10000 (61.19%), best:    6146/10000 (61.46%), error:     11298.76465, best error:     11094.59961 on epoch:    21, epoch:   22/50, mu: 0.8399714231, learning_rate: 0.0000878017, weight_decay: 0.0001093636

epoch time: 3818.1364746094s, input_fired_time: 38.7931594849, output_fired_time: 21.7157039642, propagate_forward_time: 1486.3690185547, propagate_backward_time: 2237.5527343750, weight_update_time: 1.8289623260, other_time: 33.7055664063
epoch time: 307.5068664551s, input_fired_time: 2.6195969582, output_fired_time: 0.0000000000, propagate_forward_time: 298.7301025391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1571655273
validation[          , genome  6370] predictions:    6064/  10000 (60.64%), best:    6146/10000 (61.46%), error:     11349.50391, best error:     11094.59961 on epoch:    21, epoch:   23/50, mu: 0.8480108380, learning_rate: 0.0000802439, weight_decay: 0.0001010114

epoch time: 3840.8481445313s, input_fired_time: 35.0309791565, output_fired_time: 20.8729000092, propagate_forward_time: 1489.1666259766, propagate_backward_time: 2262.4770507813, weight_update_time: 0.8749120831, other_time: 33.3007812500
epoch time: 306.2731933594s, input_fired_time: 2.6746549606, output_fired_time: 0.0000000000, propagate_forward_time: 298.1323547363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4661865234
validation[          , genome  6370] predictions:    6164/  10000 (61.64%), best:    6146/10000 (61.46%), error:     11102.73926, best error:     11094.59961 on epoch:    21, epoch:   24/50, mu: 0.8556194305, learning_rate: 0.0000733367, weight_decay: 0.0000932971

epoch time: 3829.1430664063s, input_fired_time: 36.8069343567, output_fired_time: 23.7488956451, propagate_forward_time: 1493.3406982422, propagate_backward_time: 2243.2441406250, weight_update_time: 0.8624054193, other_time: 32.0024414063
epoch time: 307.2648620605s, input_fired_time: 2.6370437145, output_fired_time: 0.0000000000, propagate_forward_time: 299.3024597168, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3253479004
validation[          , genome  6370] predictions:    6194/  10000 (61.94%), best:    6194/10000 (61.94%), error:     11048.55664, best error:     11048.55664 on epoch:    25, epoch:   25/50, mu: 0.8628203273, learning_rate: 0.0000670240, weight_decay: 0.0000861719

epoch time: 3826.7868652344s, input_fired_time: 38.4249801636, output_fired_time: 22.0462913513, propagate_forward_time: 1485.4270019531, propagate_backward_time: 2244.6962890625, weight_update_time: 0.8576048017, other_time: 36.1921386719
epoch time: 307.1954345703s, input_fired_time: 2.6391673088, output_fired_time: 0.0000000000, propagate_forward_time: 297.0956726074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4606018066
validation[          , genome  6370] predictions:    6128/  10000 (61.28%), best:    6194/10000 (61.94%), error:     11242.00977, best error:     11048.55664 on epoch:    25, epoch:   26/50, mu: 0.8696353436, learning_rate: 0.0000612547, weight_decay: 0.0000795909

epoch time: 3877.0175781250s, input_fired_time: 35.9306564331, output_fired_time: 19.8288230896, propagate_forward_time: 1511.4752197266, propagate_backward_time: 2277.3571777344, weight_update_time: 1.8541904688, other_time: 32.4255371094
epoch time: 335.1068115234s, input_fired_time: 4.7488193512, output_fired_time: 0.0000000000, propagate_forward_time: 324.7820739746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5759277344
validation[          , genome  6370] predictions:    6249/  10000 (62.49%), best:    6249/10000 (62.49%), error:     10933.91895, best error:     10933.91895 on epoch:    27, epoch:   27/50, mu: 0.8760851622, learning_rate: 0.0000559820, weight_decay: 0.0000735125

epoch time: 3847.2197265625s, input_fired_time: 37.0550003052, output_fired_time: 23.8301277161, propagate_forward_time: 1487.8146972656, propagate_backward_time: 2261.2080078125, weight_update_time: 0.8728471398, other_time: 37.3120117188
epoch time: 307.0114440918s, input_fired_time: 3.6534318924, output_fired_time: 0.0000000000, propagate_forward_time: 296.8866271973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4713745117
validation[          , genome  6370] predictions:    6247/  10000 (62.47%), best:    6247/10000 (62.47%), error:     10818.81836, best error:     10818.81836 on epoch:    28, epoch:   28/50, mu: 0.8821893930, learning_rate: 0.0000511632, weight_decay: 0.0000678983

epoch time: 3848.3872070313s, input_fired_time: 40.0398979187, output_fired_time: 20.9901046753, propagate_forward_time: 1495.0723876953, propagate_backward_time: 2261.0354003906, weight_update_time: 0.8664017916, other_time: 31.2497558594
epoch time: 307.3847045898s, input_fired_time: 3.6732773781, output_fired_time: 0.0000000000, propagate_forward_time: 298.5338134766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1776123047
validation[          , genome  6370] predictions:    6246/  10000 (62.46%), best:    6247/10000 (62.47%), error:     10832.40430, best error:     10818.81836 on epoch:    28, epoch:   29/50, mu: 0.8879665136, learning_rate: 0.0000467592, weight_decay: 0.0000627128

epoch time: 3839.1657714844s, input_fired_time: 38.0758590698, output_fired_time: 23.0226058960, propagate_forward_time: 1490.6788330078, propagate_backward_time: 2253.0759277344, weight_update_time: 0.8688870668, other_time: 34.3122558594
epoch time: 306.7458496094s, input_fired_time: 2.6602525711, output_fired_time: 0.0000000000, propagate_forward_time: 296.5751342773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5104675293
validation[          , genome  6370] predictions:    6302/  10000 (63.02%), best:    6302/10000 (63.02%), error:     10750.92480, best error:     10750.92480 on epoch:    30, epoch:   30/50, mu: 0.8934340477, learning_rate: 0.0000427342, weight_decay: 0.0000579234

epoch time: 3845.4291992188s, input_fired_time: 36.1246643066, output_fired_time: 20.7730369568, propagate_forward_time: 1492.3049316406, propagate_backward_time: 2262.7121582031, weight_update_time: 0.8752360344, other_time: 33.5144042969
epoch time: 307.0859375000s, input_fired_time: 2.6773684025, output_fired_time: 0.0000000000, propagate_forward_time: 298.0952453613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3133239746
validation[          , genome  6370] predictions:    6387/  10000 (63.87%), best:    6387/10000 (63.87%), error:     10551.79980, best error:     10551.79980 on epoch:    31, epoch:   31/50, mu: 0.8986086249, learning_rate: 0.0000390557, weight_decay: 0.0000534997

epoch time: 3847.2670898438s, input_fired_time: 37.2166557312, output_fired_time: 19.2223758698, propagate_forward_time: 1500.8937988281, propagate_backward_time: 2254.4804687500, weight_update_time: 0.8711076379, other_time: 35.4538574219
epoch time: 307.9560241699s, input_fired_time: 2.7070400715, output_fired_time: 0.0000000000, propagate_forward_time: 299.8789672852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3700256348
validation[          , genome  6370] predictions:    6185/  10000 (61.85%), best:    6387/10000 (63.87%), error:     10892.11230, best error:     10551.79980 on epoch:    31, epoch:   32/50, mu: 0.9035059214, learning_rate: 0.0000356939, weight_decay: 0.0000494139

epoch time: 3839.4611816406s, input_fired_time: 36.0067405701, output_fired_time: 20.0215911865, propagate_forward_time: 1494.3525390625, propagate_backward_time: 2253.4846191406, weight_update_time: 0.8791801929, other_time: 35.5957031250
epoch time: 306.0887451172s, input_fired_time: 3.6783938408, output_fired_time: 0.0000000000, propagate_forward_time: 296.2348022461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1755371094
validation[          , genome  6370] predictions:    6306/  10000 (63.06%), best:    6387/10000 (63.87%), error:     10682.11328, best error:     10551.79980 on epoch:    31, epoch:   33/50, mu: 0.9081407785, learning_rate: 0.0000326214, weight_decay: 0.0000456401

epoch time: 3839.0290527344s, input_fired_time: 36.0510902405, output_fired_time: 20.7966575623, propagate_forward_time: 1491.3575439453, propagate_backward_time: 2257.2150878906, weight_update_time: 1.8770240545, other_time: 33.6086425781
epoch time: 305.7008972168s, input_fired_time: 2.6366298199, output_fired_time: 0.0000000000, propagate_forward_time: 297.8414611816, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2228088379
validation[          , genome  6370] predictions:    6272/  10000 (62.72%), best:    6387/10000 (63.87%), error:     10764.71289, best error:     10551.79980 on epoch:    31, epoch:   34/50, mu: 0.9125272632, learning_rate: 0.0000298134, weight_decay: 0.0000421545

epoch time: 3843.6213378906s, input_fired_time: 38.1715087891, output_fired_time: 22.9551277161, propagate_forward_time: 1493.5661621094, propagate_backward_time: 2252.9851074219, weight_update_time: 0.8687413335, other_time: 35.9433593750
epoch time: 307.5609436035s, input_fired_time: 2.6782705784, output_fired_time: 0.0000000000, propagate_forward_time: 297.4158325195, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4668273926
validation[          , genome  6370] predictions:    6338/  10000 (63.38%), best:    6387/10000 (63.87%), error:     10576.46289, best error:     10551.79980 on epoch:    31, epoch:   35/50, mu: 0.9166786671, learning_rate: 0.0000272471, weight_decay: 0.0000389352

epoch time: 3839.2114257813s, input_fired_time: 37.1992149353, output_fired_time: 22.9656162262, propagate_forward_time: 1497.0123291016, propagate_backward_time: 2249.6789550781, weight_update_time: 0.8589189649, other_time: 32.3552246094
epoch time: 300.7907104492s, input_fired_time: 3.6654243469, output_fired_time: 0.0000000000, propagate_forward_time: 291.8810424805, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2442321777
validation[          , genome  6370] predictions:    6385/  10000 (63.85%), best:    6385/10000 (63.85%), error:     10506.85938, best error:     10506.85938 on epoch:    36, epoch:   36/50, mu: 0.9206076264, learning_rate: 0.0000249018, weight_decay: 0.0000359616

epoch time: 3841.2727050781s, input_fired_time: 43.1633300781, output_fired_time: 21.2477092743, propagate_forward_time: 1487.0732421875, propagate_backward_time: 2256.4643554688, weight_update_time: 0.8592279553, other_time: 33.3239746094
epoch time: 299.3091430664s, input_fired_time: 2.6814949512, output_fired_time: 0.0000000000, propagate_forward_time: 289.9645996094, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6630554199
validation[          , genome  6370] predictions:    6370/  10000 (63.70%), best:    6370/10000 (63.70%), error:     10476.96777, best error:     10476.96777 on epoch:    37, epoch:   37/50, mu: 0.9243260622, learning_rate: 0.0000227583, weight_decay: 0.0000332152

epoch time: 3824.6457519531s, input_fired_time: 38.1241073608, output_fired_time: 18.8877010345, propagate_forward_time: 1482.4150390625, propagate_backward_time: 2249.5029296875, weight_update_time: 0.8871968389, other_time: 35.7160644531
epoch time: 305.5517272949s, input_fired_time: 2.7615253925, output_fired_time: 0.0000000000, propagate_forward_time: 296.7789306641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0112609863
validation[          , genome  6370] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10390.01074, best error:     10390.01074 on epoch:    38, epoch:   38/50, mu: 0.9278452396, learning_rate: 0.0000207993, weight_decay: 0.0000306785

epoch time: 3836.3359375000s, input_fired_time: 33.9823608398, output_fired_time: 19.6874275208, propagate_forward_time: 1487.9931640625, propagate_backward_time: 2261.0522460938, weight_update_time: 0.8781681061, other_time: 33.6206054688
epoch time: 303.8999938965s, input_fired_time: 2.6049108505, output_fired_time: 0.0000000000, propagate_forward_time: 296.3932495117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9018249512
validation[          , genome  6370] predictions:    6403/  10000 (64.03%), best:    6388/10000 (63.88%), error:     10398.60156, best error:     10390.01074 on epoch:    38, epoch:   39/50, mu: 0.9311758280, learning_rate: 0.0000190089, weight_decay: 0.0000283356

epoch time: 3865.1030273438s, input_fired_time: 37.7869071960, output_fired_time: 20.9115238190, propagate_forward_time: 1488.1627197266, propagate_backward_time: 2286.4973144531, weight_update_time: 0.8809528351, other_time: 31.7443847656
epoch time: 306.5552062988s, input_fired_time: 3.6634681225, output_fired_time: 0.0000000000, propagate_forward_time: 293.3713684082, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.5203552246
validation[          , genome  6370] predictions:    6424/  10000 (64.24%), best:    6388/10000 (63.88%), error:     10393.96094, best error:     10390.01074 on epoch:    38, epoch:   40/50, mu: 0.9343279600, learning_rate: 0.0000173727, weight_decay: 0.0000261716

epoch time: 3817.2395019531s, input_fired_time: 37.9518432617, output_fired_time: 19.9266395569, propagate_forward_time: 1473.7758789063, propagate_backward_time: 2247.9445800781, weight_update_time: 1.8866831064, other_time: 37.6403808594
epoch time: 303.7373352051s, input_fired_time: 3.6735732555, output_fired_time: 0.0000000000, propagate_forward_time: 291.5274353027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5363159180
validation[          , genome  6370] predictions:    6410/  10000 (64.10%), best:    6388/10000 (63.88%), error:     10409.46289, best error:     10390.01074 on epoch:    38, epoch:   41/50, mu: 0.9373111725, learning_rate: 0.0000158773, weight_decay: 0.0000241728

epoch time: 3760.7575683594s, input_fired_time: 38.0312728882, output_fired_time: 21.6935214996, propagate_forward_time: 1449.9389648438, propagate_backward_time: 2219.2927246094, weight_update_time: 0.8823239803, other_time: 31.8010253906
epoch time: 301.7899169922s, input_fired_time: 2.7450587749, output_fired_time: 0.0000000000, propagate_forward_time: 292.8032531738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2416076660
validation[          , genome  6370] predictions:    6404/  10000 (64.04%), best:    6404/10000 (64.04%), error:     10353.64160, best error:     10353.64160 on epoch:    42, epoch:   42/50, mu: 0.9401345253, learning_rate: 0.0000145106, weight_decay: 0.0000223267

epoch time: 3990.2841796875s, input_fired_time: 47.7196922302, output_fired_time: 21.0518112183, propagate_forward_time: 1529.0091552734, propagate_backward_time: 2357.5415039063, weight_update_time: 0.9066821337, other_time: 34.9619140625
epoch time: 309.0170288086s, input_fired_time: 2.7290117741, output_fired_time: 0.0000000000, propagate_forward_time: 300.0527343750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2352905273
validation[          , genome  6370] predictions:    6446/  10000 (64.46%), best:    6446/10000 (64.46%), error:     10291.49805, best error:     10291.49805 on epoch:    43, epoch:   43/50, mu: 0.9428066015, learning_rate: 0.0000132615, weight_decay: 0.0000206216

epoch time: 3799.8947753906s, input_fired_time: 38.0846290588, output_fired_time: 19.9020309448, propagate_forward_time: 1468.4423828125, propagate_backward_time: 2234.2202148438, weight_update_time: 0.8779128790, other_time: 39.2453613281
epoch time: 303.7164306641s, input_fired_time: 2.7309269905, output_fired_time: 0.0000000000, propagate_forward_time: 295.0847778320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9007263184
validation[          , genome  6370] predictions:    6394/  10000 (63.94%), best:    6446/10000 (64.46%), error:     10394.59863, best error:     10291.49805 on epoch:    43, epoch:   44/50, mu: 0.9453355074, learning_rate: 0.0000121200, weight_decay: 0.0000190467

epoch time: 3783.9641113281s, input_fired_time: 39.0691604614, output_fired_time: 20.8242149353, propagate_forward_time: 1460.7744140625, propagate_backward_time: 2229.5402832031, weight_update_time: 0.8892030716, other_time: 33.7561035156
epoch time: 308.1307067871s, input_fired_time: 2.6481986046, output_fired_time: 0.0000000000, propagate_forward_time: 300.3399047852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1426086426
validation[          , genome  6370] predictions:    6421/  10000 (64.21%), best:    6446/10000 (64.46%), error:     10296.09961, best error:     10291.49805 on epoch:    43, epoch:   45/50, mu: 0.9477288723, learning_rate: 0.0000110767, weight_decay: 0.0000175921

epoch time: 3798.0241699219s, input_fired_time: 38.1200942993, output_fired_time: 20.8529186249, propagate_forward_time: 1473.9581298828, propagate_backward_time: 2232.4157714844, weight_update_time: 0.8741240501, other_time: 32.6770019531
epoch time: 302.7535095215s, input_fired_time: 2.6662724018, output_fired_time: 0.0000000000, propagate_forward_time: 294.7325439453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3547058105
validation[          , genome  6370] predictions:    6464/  10000 (64.64%), best:    6464/10000 (64.64%), error:     10260.19531, best error:     10260.19531 on epoch:    46, epoch:   46/50, mu: 0.9499940276, learning_rate: 0.0000101233, weight_decay: 0.0000162486

epoch time: 3785.6003417969s, input_fired_time: 37.9043807983, output_fired_time: 21.7451400757, propagate_forward_time: 1465.8123779297, propagate_backward_time: 2222.5886230469, weight_update_time: 0.8713173866, other_time: 37.5500488281
epoch time: 290.6297302246s, input_fired_time: 2.5849902630, output_fired_time: 0.0000000000, propagate_forward_time: 282.7761535645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2685852051
validation[          , genome  6370] predictions:    6417/  10000 (64.17%), best:    6464/10000 (64.64%), error:     10292.41113, best error:     10260.19531 on epoch:    46, epoch:   47/50, mu: 0.9521377683, learning_rate: 0.0000092519, weight_decay: 0.0000150077

epoch time: 3731.1047363281s, input_fired_time: 36.8108634949, output_fired_time: 19.4393901825, propagate_forward_time: 1433.8941650391, propagate_backward_time: 2207.1269531250, weight_update_time: 0.9041599631, other_time: 33.8334960938
epoch time: 286.4326171875s, input_fired_time: 2.5781972408, output_fired_time: 0.0000000000, propagate_forward_time: 278.1400451660, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7143859863
validation[          , genome  6370] predictions:    6447/  10000 (64.47%), best:    6464/10000 (64.64%), error:     10365.42871, best error:     10260.19531 on epoch:    46, epoch:   48/50, mu: 0.9541666508, learning_rate: 0.0000084555, weight_decay: 0.0000138615

epoch time: 3737.4162597656s, input_fired_time: 38.6982307434, output_fired_time: 23.5513172150, propagate_forward_time: 1439.0361328125, propagate_backward_time: 2199.4396972656, weight_update_time: 0.8922307491, other_time: 36.6909179688
epoch time: 301.8112792969s, input_fired_time: 3.7347271442, output_fired_time: 0.0000000000, propagate_forward_time: 292.5910339355, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4855041504
validation[          , genome  6370] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10180.34473, best error:     10180.34473 on epoch:    49, epoch:   49/50, mu: 0.9560868144, learning_rate: 0.0000077277, weight_decay: 0.0000128029

epoch time: 3746.9272460938s, input_fired_time: 34.9561576843, output_fired_time: 18.7595386505, propagate_forward_time: 1449.5001220703, propagate_backward_time: 2210.8374023438, weight_update_time: 0.8825392127, other_time: 32.8740234375
epoch time: 294.7561340332s, input_fired_time: 3.6927826405, output_fired_time: 0.0000000000, propagate_forward_time: 285.2113647461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8519897461
validation[          , genome  6370] predictions:    6502/  10000 (65.02%), best:    6485/10000 (64.85%), error:     10215.21875, best error:     10180.34473 on epoch:    49, epoch:   50/50, mu: 0.9579041004, learning_rate: 0.0000070625, weight_decay: 0.0000118251

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1507.2047119141s, input_fired_time: 14.2508335114, output_fired_time: 0.0000000000, propagate_forward_time: 1462.9697265625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 29.9841308594
best training[          , genome  6370] predictions:   34765/  50000 (69.53%), best:    6485/10000 (64.85%), error:     46037.83594, best error:     10180.34473 on epoch:    49, epoch:   51/50, mu: 0.9596239924, learning_rate: 0.0000064546, weight_decay: 0.0000109220
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 304.8210144043s, input_fired_time: 3.7641522884, output_fired_time: 0.0000000000, propagate_forward_time: 292.0061645508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.0506896973
   testing[          , genome  6370] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10180.34473, best error:     10180.34473 on epoch:    49, epoch:   51/50, mu: 0.9596239924, learning_rate: 0.0000064546, weight_decay: 0.0000109220
backpropagation finished successfully!
00:57:11 (3372): called boinc_finish(0)

</stderr_txt>
]]>