Nameexact_genome_1542277804_179_5966_0
Workunit4090441
Created15 Nov 2018, 13:48:51 UTC
Sent15 Nov 2018, 13:49:10 UTC
Report deadline20 Jan 2019, 18:54:01 UTC
Received20 Nov 2018, 12:12:44 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID83973
Run time2 days 9 hours 51 min 32 sec
CPU time2 days 9 hours 39 min 16 sec
Validate stateValid
Credit16,552.93
Device peak FLOPS2.50 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size576.43 MB
Peak swap size596.45 MB
Peak disk usage16.02 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_179_5966.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_179_5966_0_r1154627228_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542277804_179_5966.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.511884
read mu: 0.511884
read mu_delta: 0.9555
read initial_learning_rate: 0.000587637
read learning_rate: 0.000587637
read learning_rate_delta: 0.914149
read initial_weight_decay: 0.00045852
read weight_decay: 0.00045852
read weight_decay_delta: 0.936517
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.109247
read input_dropout_probability: 0.00128562
read hidden_dropout_probability: 0.0974779
read velocity_reset: 1742
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5966
read normal distribution: '1 0 0'
generator_str: '1375049947'
read generator: 1375049947
read generated_by_map:
1 disable_edge 1
reading 112 nodes.
reading 578 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1392933862
post shuffle 1: 1236575691
epoch time: 4159.6240234375s, input_fired_time: 45.2066192626953, output_fired_time: 26.9698524475098, propagate_forward_time: 1610.14575195313, propagate_backward_time: 2439.71826171875, weight_update_time: 1.0355144739151, other_time: 37.58349609375
epoch time: 335.235900878906s, input_fired_time: 3.3872504234314, output_fired_time: 0, propagate_forward_time: 324.449615478516, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.3990478515625
validation[          , genome  5966] predictions:    5496/  10000 (54.96%), best:    5496/10000 (54.96%), error:     13277.65234, best error:     13277.65234 on epoch:     0, epoch:    0/50, mu: 0.5118840933, learning_rate: 0.0005876372, weight_decay: 0.0004585199

epoch time: 4166.0605468750s, input_fired_time: 47.3899841309, output_fired_time: 28.0247650146, propagate_forward_time: 1612.6816406250, propagate_backward_time: 2436.2500000000, weight_update_time: 1.0099619627, other_time: 41.7141113281
epoch time: 335.6231079102s, input_fired_time: 4.3312444687, output_fired_time: 0.0000000000, propagate_forward_time: 323.9442443848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3476257324
validation[          , genome  5966] predictions:    6127/  10000 (61.27%), best:    6127/10000 (61.27%), error:     11248.12012, best error:     11248.12012 on epoch:     1, epoch:    1/50, mu: 0.5331600308, learning_rate: 0.0005371879, weight_decay: 0.0004294118

epoch time: 4160.5278320313s, input_fired_time: 47.3231010437, output_fired_time: 26.7250804901, propagate_forward_time: 1612.8244628906, propagate_backward_time: 2434.6828613281, weight_update_time: 1.0298708677, other_time: 38.9721679688
epoch time: 335.3109741211s, input_fired_time: 3.3338570595, output_fired_time: 0.0000000000, propagate_forward_time: 325.6404113770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3367004395
validation[          , genome  5966] predictions:    5913/  10000 (59.13%), best:    6127/10000 (61.27%), error:     11825.78906, best error:     11248.12012 on epoch:     1, epoch:    2/50, mu: 0.5534892082, learning_rate: 0.0004910697, weight_decay: 0.0004021515

epoch time: 4154.3369140625s, input_fired_time: 45.1421585083, output_fired_time: 28.0779819489, propagate_forward_time: 1617.2259521484, propagate_backward_time: 2428.3083496094, weight_update_time: 1.0376099348, other_time: 35.5827636719
epoch time: 335.2869873047s, input_fired_time: 3.3388378620, output_fired_time: 0.0000000000, propagate_forward_time: 325.5806274414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3675231934
validation[          , genome  5966] predictions:    6104/  10000 (61.04%), best:    6127/10000 (61.27%), error:     11330.00488, best error:     11248.12012 on epoch:     1, epoch:    3/50, mu: 0.5729137063, learning_rate: 0.0004489108, weight_decay: 0.0003766218

epoch time: 4149.4086914063s, input_fired_time: 47.9306144714, output_fired_time: 25.8811340332, propagate_forward_time: 1604.2944335938, propagate_backward_time: 2431.9904785156, weight_update_time: 1.0098049641, other_time: 39.3120117188
epoch time: 334.1587524414s, input_fired_time: 3.3310728073, output_fired_time: 0.0000000000, propagate_forward_time: 324.4063415527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4213256836
validation[          , genome  5966] predictions:    5974/  10000 (59.74%), best:    6127/10000 (61.27%), error:     11936.53027, best error:     11248.12012 on epoch:     1, epoch:    4/50, mu: 0.5914738178, learning_rate: 0.0004103713, weight_decay: 0.0003527128

epoch time: 4138.6406250000s, input_fired_time: 45.9044609070, output_fired_time: 26.7661209106, propagate_forward_time: 1601.0834960938, propagate_backward_time: 2427.2653808594, weight_update_time: 1.0198559761, other_time: 37.6210937500
epoch time: 332.2170104980s, input_fired_time: 3.3333270550, output_fired_time: 0.0000000000, propagate_forward_time: 321.6597290039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2239685059
validation[          , genome  5966] predictions:    5965/  10000 (59.65%), best:    6127/10000 (61.27%), error:     11707.81250, best error:     11248.12012 on epoch:     1, epoch:    5/50, mu: 0.6092080474, learning_rate: 0.0003751405, weight_decay: 0.0003303217

epoch time: 4145.8427734375s, input_fired_time: 45.8147811890, output_fired_time: 25.8339328766, propagate_forward_time: 1609.9815673828, propagate_backward_time: 2425.2749023438, weight_update_time: 1.0332539082, other_time: 38.9375000000
epoch time: 332.6634521484s, input_fired_time: 3.3000676632, output_fired_time: 0.0000000000, propagate_forward_time: 322.0648803711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2984924316
validation[          , genome  5966] predictions:    5621/  10000 (56.21%), best:    6127/10000 (61.27%), error:     12686.41309, best error:     11248.12012 on epoch:     1, epoch:    6/50, mu: 0.6261531115, learning_rate: 0.0003429342, weight_decay: 0.0003093519

epoch time: 4150.4111328125s, input_fired_time: 43.0596961975, output_fired_time: 26.8858070374, propagate_forward_time: 1612.3891601563, propagate_backward_time: 2426.8491210938, weight_update_time: 1.0321887732, other_time: 41.2275390625
epoch time: 333.9372558594s, input_fired_time: 3.3333308697, output_fired_time: 0.0000000000, propagate_forward_time: 324.2648315430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3390808105
validation[          , genome  5966] predictions:    6172/  10000 (61.72%), best:    6172/10000 (61.72%), error:     11118.64551, best error:     11118.64551 on epoch:     7, epoch:    7/50, mu: 0.6423441172, learning_rate: 0.0003134929, weight_decay: 0.0002897134

epoch time: 4315.3657226563s, input_fired_time: 49.1438407898, output_fired_time: 27.3433628082, propagate_forward_time: 1622.5930175781, propagate_backward_time: 2580.2258300781, weight_update_time: 1.0394357443, other_time: 36.0595703125
epoch time: 335.4762268066s, input_fired_time: 3.2715964317, output_fired_time: 0.0000000000, propagate_forward_time: 324.9612426758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2433776855
validation[          , genome  5966] predictions:    5854/  10000 (58.54%), best:    6172/10000 (61.72%), error:     12064.09570, best error:     11118.64551 on epoch:     7, epoch:    8/50, mu: 0.6578146219, learning_rate: 0.0002865792, weight_decay: 0.0002713216

epoch time: 4155.5332031250s, input_fired_time: 47.8365554810, output_fired_time: 26.6819114685, propagate_forward_time: 1607.8309326172, propagate_backward_time: 2435.3229980469, weight_update_time: 1.0689923763, other_time: 37.8610839844
epoch time: 335.4011535645s, input_fired_time: 3.3583540916, output_fired_time: 0.0000000000, propagate_forward_time: 325.6565246582, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3862609863
validation[          , genome  5966] predictions:    6236/  10000 (62.36%), best:    6236/10000 (62.36%), error:     10954.69629, best error:     10954.69629 on epoch:     9, epoch:    9/50, mu: 0.6725966930, learning_rate: 0.0002619761, weight_decay: 0.0002540974

epoch time: 4181.7207031250s, input_fired_time: 46.9635620117, output_fired_time: 23.7873954773, propagate_forward_time: 1613.2983398438, propagate_backward_time: 2458.8840332031, weight_update_time: 1.0295438766, other_time: 38.7873535156
epoch time: 336.1599121094s, input_fired_time: 3.3348500729, output_fired_time: 0.0000000000, propagate_forward_time: 326.5729675293, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2521057129
validation[          , genome  5966] predictions:    6146/  10000 (61.46%), best:    6236/10000 (62.36%), error:     11197.86914, best error:     10954.69629 on epoch:     9, epoch:   10/50, mu: 0.6867209673, learning_rate: 0.0002394851, weight_decay: 0.0002379666

epoch time: 4175.0283203125s, input_fired_time: 43.0293350220, output_fired_time: 29.0110664368, propagate_forward_time: 1618.1927490234, propagate_backward_time: 2445.8208007813, weight_update_time: 1.0409038067, other_time: 38.9741210938
epoch time: 335.2640686035s, input_fired_time: 3.3278470039, output_fired_time: 0.0000000000, propagate_forward_time: 325.8272399902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1089782715
validation[          , genome  5966] predictions:    6237/  10000 (62.37%), best:    6237/10000 (62.37%), error:     10927.57031, best error:     10927.57031 on epoch:    11, epoch:   11/50, mu: 0.7002167106, learning_rate: 0.0002189251, weight_decay: 0.0002228598

epoch time: 4174.7314453125s, input_fired_time: 47.1235122681, output_fired_time: 25.9659385681, propagate_forward_time: 1633.5405273438, propagate_backward_time: 2432.5207519531, weight_update_time: 1.0244607925, other_time: 35.5808105469
epoch time: 337.9419860840s, input_fired_time: 4.3509306908, output_fired_time: 0.0000000000, propagate_forward_time: 325.1835937500, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4074707031
validation[          , genome  5966] predictions:    5922/  10000 (59.22%), best:    6237/10000 (62.37%), error:     11638.34863, best error:     10927.57031 on epoch:    11, epoch:   12/50, mu: 0.7131119370, learning_rate: 0.0002001301, weight_decay: 0.0002087120

epoch time: 4165.2768554688s, input_fired_time: 45.3799209595, output_fired_time: 33.0348739624, propagate_forward_time: 1617.2642822266, propagate_backward_time: 2427.7263183594, weight_update_time: 2.0053420067, other_time: 41.8713378906
epoch time: 335.0330505371s, input_fired_time: 4.3072876930, output_fired_time: 0.0000000000, propagate_forward_time: 323.5401306152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1856384277
validation[          , genome  5966] predictions:    6106/  10000 (61.06%), best:    6237/10000 (62.37%), error:     11273.11133, best error:     10927.57031 on epoch:    11, epoch:   13/50, mu: 0.7254333496, learning_rate: 0.0001829487, weight_decay: 0.0001954624

epoch time: 4185.9423828125s, input_fired_time: 47.0134162903, output_fired_time: 28.8405246735, propagate_forward_time: 1617.1796875000, propagate_backward_time: 2455.9877929688, weight_update_time: 1.0267798901, other_time: 36.9213867188
epoch time: 346.5675659180s, input_fired_time: 3.4068183899, output_fired_time: 0.0000000000, propagate_forward_time: 336.5575256348, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6032104492
validation[          , genome  5966] predictions:    6267/  10000 (62.67%), best:    6267/10000 (62.67%), error:     10895.38574, best error:     10895.38574 on epoch:    14, epoch:   14/50, mu: 0.7372064590, learning_rate: 0.0001672424, weight_decay: 0.0001830539

epoch time: 4167.2509765625s, input_fired_time: 45.1521415710, output_fired_time: 26.6174793243, propagate_forward_time: 1612.7530517578, propagate_backward_time: 2441.6579589844, weight_update_time: 2.0607345104, other_time: 41.0700683594
epoch time: 333.7543029785s, input_fired_time: 3.3498458862, output_fired_time: 0.0000000000, propagate_forward_time: 324.2092285156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1952209473
validation[          , genome  5966] predictions:    6161/  10000 (61.61%), best:    6267/10000 (62.67%), error:     11115.66016, best error:     10895.38574 on epoch:    14, epoch:   15/50, mu: 0.7484556437, learning_rate: 0.0001528844, weight_decay: 0.0001714332

epoch time: 4075.0305175781s, input_fired_time: 48.7557830811, output_fired_time: 24.7294254303, propagate_forward_time: 1583.7788085938, propagate_backward_time: 2377.9460449219, weight_update_time: 1.0192278624, other_time: 39.8203125000
epoch time: 335.6914978027s, input_fired_time: 3.3571276665, output_fired_time: 0.0000000000, propagate_forward_time: 325.0808105469, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2535705566
validation[          , genome  5966] predictions:    6270/  10000 (62.70%), best:    6270/10000 (62.70%), error:     10851.67773, best error:     10851.67773 on epoch:    16, epoch:   16/50, mu: 0.7592042685, learning_rate: 0.0001397591, weight_decay: 0.0001605501

epoch time: 4159.2465820313s, input_fired_time: 43.9788131714, output_fired_time: 25.6611614227, propagate_forward_time: 1610.1621093750, propagate_backward_time: 2438.7199707031, weight_update_time: 1.0063204765, other_time: 40.7243652344
epoch time: 336.6840209961s, input_fired_time: 4.3578276634, output_fired_time: 0.0000000000, propagate_forward_time: 326.0581359863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2680664063
validation[          , genome  5966] predictions:    6247/  10000 (62.47%), best:    6270/10000 (62.70%), error:     10901.76855, best error:     10851.67773 on epoch:    16, epoch:   17/50, mu: 0.7694745660, learning_rate: 0.0001277606, weight_decay: 0.0001503579

epoch time: 4146.7500000000s, input_fired_time: 46.0356025696, output_fired_time: 27.5599060059, propagate_forward_time: 1601.9338378906, propagate_backward_time: 2428.1901855469, weight_update_time: 2.0354764462, other_time: 43.0305175781
epoch time: 332.6322021484s, input_fired_time: 4.3452005386, output_fired_time: 0.0000000000, propagate_forward_time: 320.0942382813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1927490234
validation[          , genome  5966] predictions:    6295/  10000 (62.95%), best:    6295/10000 (62.95%), error:     10733.27344, best error:     10733.27344 on epoch:    18, epoch:   18/50, mu: 0.7792878151, learning_rate: 0.0001167922, weight_decay: 0.0001408128

epoch time: 4129.9985351563s, input_fired_time: 50.9872016907, output_fired_time: 24.5839653015, propagate_forward_time: 1600.7053222656, propagate_backward_time: 2413.8100585938, weight_update_time: 1.0204889774, other_time: 39.9118652344
epoch time: 336.7412109375s, input_fired_time: 3.3684875965, output_fired_time: 0.0000000000, propagate_forward_time: 327.0338745117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3388366699
validation[          , genome  5966] predictions:    6468/  10000 (64.68%), best:    6468/10000 (64.68%), error:     10298.67773, best error:     10298.67773 on epoch:    19, epoch:   19/50, mu: 0.7886644006, learning_rate: 0.0001067655, weight_decay: 0.0001318736

epoch time: 4139.3930664063s, input_fired_time: 46.0086479187, output_fired_time: 26.4649085999, propagate_forward_time: 1596.8237304688, propagate_backward_time: 2429.3320312500, weight_update_time: 1.0172283649, other_time: 40.7639160156
epoch time: 333.3842163086s, input_fired_time: 4.3487977982, output_fired_time: 0.0000000000, propagate_forward_time: 322.9295654297, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1058654785
validation[          , genome  5966] predictions:    6150/  10000 (61.50%), best:    6468/10000 (64.68%), error:     11599.08887, best error:     10298.67773 on epoch:    19, epoch:   20/50, mu: 0.7976237535, learning_rate: 0.0000975995, weight_decay: 0.0001235019

epoch time: 4038.9858398438s, input_fired_time: 45.2815017700, output_fired_time: 26.9687728882, propagate_forward_time: 1557.1390380859, propagate_backward_time: 2371.0678710938, weight_update_time: 2.0035173893, other_time: 38.5288085938
epoch time: 331.8972167969s, input_fired_time: 4.3097524643, output_fired_time: 0.0000000000, propagate_forward_time: 321.5328979492, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0545654297
validation[          , genome  5966] predictions:    6254/  10000 (62.54%), best:    6468/10000 (64.68%), error:     10800.80762, best error:     10298.67773 on epoch:    19, epoch:   21/50, mu: 0.8061844110, learning_rate: 0.0000892205, weight_decay: 0.0001156617

epoch time: 4103.9047851563s, input_fired_time: 41.7575378418, output_fired_time: 24.1965923309, propagate_forward_time: 1584.6729736328, propagate_backward_time: 2414.5036621094, weight_update_time: 2.0120017529, other_time: 38.7741699219
epoch time: 317.4861450195s, input_fired_time: 5.0537052155, output_fired_time: 0.0000000000, propagate_forward_time: 306.5459899902, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.8864440918
validation[          , genome  5966] predictions:    6354/  10000 (63.54%), best:    6468/10000 (64.68%), error:     10485.32910, best error:     10298.67773 on epoch:    19, epoch:   22/50, mu: 0.8143641353, learning_rate: 0.0000815608, weight_decay: 0.0001083191

epoch time: 4142.0981445313s, input_fired_time: 43.9736709595, output_fired_time: 27.6708793640, propagate_forward_time: 1609.5653076172, propagate_backward_time: 2419.1398925781, weight_update_time: 1.0158072710, other_time: 41.7482910156
epoch time: 335.4105834961s, input_fired_time: 3.3627588749, output_fired_time: 0.0000000000, propagate_forward_time: 324.7501220703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2976989746
validation[          , genome  5966] predictions:    6239/  10000 (62.39%), best:    6468/10000 (64.68%), error:     10936.08398, best error:     10298.67773 on epoch:    19, epoch:   23/50, mu: 0.8221798539, learning_rate: 0.0000745587, weight_decay: 0.0001014427

epoch time: 4142.6967773438s, input_fired_time: 44.9947891235, output_fired_time: 24.4880981445, propagate_forward_time: 1610.9906005859, propagate_backward_time: 2423.0385742188, weight_update_time: 0.9996759892, other_time: 39.1850585938
epoch time: 337.1140136719s, input_fired_time: 3.3931016922, output_fired_time: 0.0000000000, propagate_forward_time: 326.3206481934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4002685547
validation[          , genome  5966] predictions:    6435/  10000 (64.35%), best:    6468/10000 (64.68%), error:     10413.57617, best error:     10298.67773 on epoch:    19, epoch:   24/50, mu: 0.8296477795, learning_rate: 0.0000681578, weight_decay: 0.0000950029

epoch time: 4162.7631835938s, input_fired_time: 49.9583587646, output_fired_time: 25.7508316040, propagate_forward_time: 1613.2272949219, propagate_backward_time: 2436.6188964844, weight_update_time: 1.0335637331, other_time: 37.2077636719
epoch time: 329.1760559082s, input_fired_time: 3.2719819546, output_fired_time: 0.0000000000, propagate_forward_time: 317.6892700195, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2148132324
validation[          , genome  5966] predictions:    6389/  10000 (63.89%), best:    6468/10000 (64.68%), error:     10526.65723, best error:     10298.67773 on epoch:    19, epoch:   25/50, mu: 0.8367833495, learning_rate: 0.0000623064, weight_decay: 0.0000889718

epoch time: 4165.7177734375s, input_fired_time: 53.2087516785, output_fired_time: 23.9716205597, propagate_forward_time: 1612.5600585938, propagate_backward_time: 2441.1545410156, weight_update_time: 1.0383597612, other_time: 34.8225097656
epoch time: 335.5086059570s, input_fired_time: 3.3806502819, output_fired_time: 0.0000000000, propagate_forward_time: 325.9960021973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1319580078
validation[          , genome  5966] predictions:    6381/  10000 (63.81%), best:    6468/10000 (64.68%), error:     10490.54297, best error:     10298.67773 on epoch:    19, epoch:   26/50, mu: 0.8436014056, learning_rate: 0.0000569573, weight_decay: 0.0000833237

epoch time: 4148.3359375000s, input_fired_time: 47.9879035950, output_fired_time: 25.6067943573, propagate_forward_time: 1609.9603271484, propagate_backward_time: 2424.2016601563, weight_update_time: 1.0021876097, other_time: 40.5795898438
epoch time: 332.3086242676s, input_fired_time: 4.3515367508, output_fired_time: 0.0000000000, propagate_forward_time: 320.9182128906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0388793945
validation[          , genome  5966] predictions:    6401/  10000 (64.01%), best:    6468/10000 (64.68%), error:     10418.74902, best error:     10298.67773 on epoch:    19, epoch:   27/50, mu: 0.8501160741, learning_rate: 0.0000520674, weight_decay: 0.0000780340

epoch time: 4185.3242187500s, input_fired_time: 47.2082481384, output_fired_time: 24.8216285706, propagate_forward_time: 1611.3037109375, propagate_backward_time: 2459.7343750000, weight_update_time: 1.0303863287, other_time: 42.2563476563
epoch time: 337.5677490234s, input_fired_time: 3.3917689323, output_fired_time: 0.0000000000, propagate_forward_time: 326.8254089355, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3505859375
validation[          , genome  5966] predictions:    6530/  10000 (65.30%), best:    6530/10000 (65.30%), error:     10146.86914, best error:     10146.86914 on epoch:    28, epoch:   28/50, mu: 0.8563408256, learning_rate: 0.0000475974, weight_decay: 0.0000730802

epoch time: 4141.0937500000s, input_fired_time: 50.0599212646, output_fired_time: 26.5316638947, propagate_forward_time: 1596.2147216797, propagate_backward_time: 2428.4023437500, weight_update_time: 1.0276881456, other_time: 39.8852539063
epoch time: 335.4993286133s, input_fired_time: 5.4195499420, output_fired_time: 0.0000000000, propagate_forward_time: 320.6390075684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.4407653809
validation[          , genome  5966] predictions:    6475/  10000 (64.75%), best:    6530/10000 (65.30%), error:     10356.02734, best error:     10146.86914 on epoch:    28, epoch:   29/50, mu: 0.8622885942, learning_rate: 0.0000435111, weight_decay: 0.0000684409

epoch time: 4133.1333007813s, input_fired_time: 44.0498809814, output_fired_time: 24.5435333252, propagate_forward_time: 1603.6597900391, propagate_backward_time: 2420.6550292969, weight_update_time: 0.9998840690, other_time: 40.2253417969
epoch time: 333.3331298828s, input_fired_time: 4.3872437477, output_fired_time: 0.0000000000, propagate_forward_time: 322.8080444336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1378479004
validation[          , genome  5966] predictions:    6420/  10000 (64.20%), best:    6530/10000 (65.30%), error:     10285.95117, best error:     10146.86914 on epoch:    28, epoch:   30/50, mu: 0.8679717183, learning_rate: 0.0000397756, weight_decay: 0.0000640961

epoch time: 4144.3032226563s, input_fired_time: 49.3614921570, output_fired_time: 26.5553054810, propagate_forward_time: 1609.1916503906, propagate_backward_time: 2422.3417968750, weight_update_time: 1.0337316990, other_time: 36.8527832031
epoch time: 335.7251586914s, input_fired_time: 3.3734457493, output_fired_time: 0.0000000000, propagate_forward_time: 326.0425720215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3091430664
validation[          , genome  5966] predictions:    6489/  10000 (64.89%), best:    6489/10000 (64.89%), error:     10088.74512, best error:     10088.74512 on epoch:    31, epoch:   31/50, mu: 0.8734019399, learning_rate: 0.0000363608, weight_decay: 0.0000600271

epoch time: 4133.0488281250s, input_fired_time: 51.9219856262, output_fired_time: 27.3441238403, propagate_forward_time: 1592.6171875000, propagate_backward_time: 2424.4155273438, weight_update_time: 1.0287750959, other_time: 36.7500000000
epoch time: 336.4819946289s, input_fired_time: 3.3852427006, output_fired_time: 0.0000000000, propagate_forward_time: 326.8872985840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2094421387
validation[          , genome  5966] predictions:    6537/  10000 (65.37%), best:    6489/10000 (64.89%), error:     10264.89453, best error:     10088.74512 on epoch:    31, epoch:   32/50, mu: 0.8785905242, learning_rate: 0.0000332392, weight_decay: 0.0000562164

epoch time: 4127.2426757813s, input_fired_time: 51.9524383545, output_fired_time: 27.7210960388, propagate_forward_time: 1602.8687744141, propagate_backward_time: 2401.6745605469, weight_update_time: 0.9802246690, other_time: 43.0256347656
epoch time: 330.7503967285s, input_fired_time: 3.3539192677, output_fired_time: 0.0000000000, propagate_forward_time: 321.0691223145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3273620605
validation[          , genome  5966] predictions:    6436/  10000 (64.36%), best:    6489/10000 (64.89%), error:     10309.39355, best error:     10088.74512 on epoch:    31, epoch:   33/50, mu: 0.8835482001, learning_rate: 0.0000303856, weight_decay: 0.0000526476

epoch time: 4414.5043945313s, input_fired_time: 47.2960777283, output_fired_time: 25.0842266083, propagate_forward_time: 1697.1936035156, propagate_backward_time: 2606.4987792969, weight_update_time: 1.0462332964, other_time: 38.4321289063
epoch time: 403.8585510254s, input_fired_time: 3.5875501633, output_fired_time: 0.0000000000, propagate_forward_time: 392.5371704102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.7338256836
validation[          , genome  5966] predictions:    6594/  10000 (65.94%), best:    6594/10000 (65.94%), error:     10022.74316, best error:     10022.74316 on epoch:    34, epoch:   34/50, mu: 0.8882852793, learning_rate: 0.0000277770, weight_decay: 0.0000493054

epoch time: 4197.9619140625s, input_fired_time: 48.4452819824, output_fired_time: 28.1425628662, propagate_forward_time: 1621.0533447266, propagate_backward_time: 2461.2175292969, weight_update_time: 1.0373591185, other_time: 39.1032714844
epoch time: 335.2919311523s, input_fired_time: 4.4896464348, output_fired_time: 0.0000000000, propagate_forward_time: 323.5371093750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2651672363
validation[          , genome  5966] predictions:    6590/  10000 (65.90%), best:    6590/10000 (65.90%), error:      9960.63086, best error:      9960.63086 on epoch:    35, epoch:   35/50, mu: 0.8928115368, learning_rate: 0.0000253923, weight_decay: 0.0000461754

epoch time: 4175.8457031250s, input_fired_time: 50.2481727600, output_fired_time: 26.7354621887, propagate_forward_time: 1614.1553955078, propagate_backward_time: 2443.9206542969, weight_update_time: 1.0219490528, other_time: 40.7863769531
epoch time: 334.5482177734s, input_fired_time: 4.3767647743, output_fired_time: 0.0000000000, propagate_forward_time: 323.9214477539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2500000000
validation[          , genome  5966] predictions:    6493/  10000 (64.93%), best:    6590/10000 (65.90%), error:     10208.25781, best error:      9960.63086 on epoch:    35, epoch:   36/50, mu: 0.8971363902, learning_rate: 0.0000232123, weight_decay: 0.0000432440

epoch time: 4174.2651367188s, input_fired_time: 47.4316024780, output_fired_time: 27.1431694031, propagate_forward_time: 1613.6699218750, propagate_backward_time: 2446.2016601563, weight_update_time: 1.0071189404, other_time: 39.8188476563
epoch time: 330.0762023926s, input_fired_time: 3.3077456951, output_fired_time: 0.0000000000, propagate_forward_time: 318.5364379883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2320251465
validation[          , genome  5966] predictions:    6579/  10000 (65.79%), best:    6590/10000 (65.90%), error:     10010.51563, best error:      9960.63086 on epoch:    35, epoch:   37/50, mu: 0.9012687802, learning_rate: 0.0000212195, weight_decay: 0.0000404988

epoch time: 4173.7846679688s, input_fired_time: 48.4727096558, output_fired_time: 24.7625598907, propagate_forward_time: 1619.1386718750, propagate_backward_time: 2443.5688476563, weight_update_time: 1.0250995159, other_time: 37.8417968750
epoch time: 335.0225219727s, input_fired_time: 3.3837871552, output_fired_time: 0.0000000000, propagate_forward_time: 323.3789978027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2597351074
validation[          , genome  5966] predictions:    6524/  10000 (65.24%), best:    6590/10000 (65.90%), error:     10138.94141, best error:      9960.63086 on epoch:    35, epoch:   38/50, mu: 0.9052172899, learning_rate: 0.0000193978, weight_decay: 0.0000379278

epoch time: 4178.9580078125s, input_fired_time: 48.4014091492, output_fired_time: 26.8646030426, propagate_forward_time: 1620.5993652344, propagate_backward_time: 2444.8642578125, weight_update_time: 1.0228326321, other_time: 38.2282714844
epoch time: 337.0385742188s, input_fired_time: 5.3746194839, output_fired_time: 0.0000000000, propagate_forward_time: 324.1554565430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5084838867
validation[          , genome  5966] predictions:    6681/  10000 (66.81%), best:    6681/10000 (66.81%), error:      9798.54883, best error:      9798.54883 on epoch:    39, epoch:   39/50, mu: 0.9089900851, learning_rate: 0.0000177325, weight_decay: 0.0000355200

epoch time: 4173.3212890625s, input_fired_time: 48.4047966003, output_fired_time: 23.8497848511, propagate_forward_time: 1614.2576904297, propagate_backward_time: 2446.0837402344, weight_update_time: 1.0289506912, other_time: 40.7253417969
epoch time: 333.5669250488s, input_fired_time: 4.4783778191, output_fired_time: 0.0000000000, propagate_forward_time: 322.9029541016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1856079102
validation[          , genome  5966] predictions:    6517/  10000 (65.17%), best:    6681/10000 (66.81%), error:     10139.00879, best error:      9798.54883 on epoch:    39, epoch:   40/50, mu: 0.9125949740, learning_rate: 0.0000162101, weight_decay: 0.0000332651

epoch time: 4176.6181640625s, input_fired_time: 47.3296470642, output_fired_time: 24.6931800842, propagate_forward_time: 1618.6274414063, propagate_backward_time: 2447.9362792969, weight_update_time: 1.0310927629, other_time: 38.0314941406
epoch time: 334.5147705078s, input_fired_time: 4.3590073586, output_fired_time: 0.0000000000, propagate_forward_time: 322.0176391602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1381225586
validation[          , genome  5966] predictions:    6584/  10000 (65.84%), best:    6681/10000 (66.81%), error:     10054.97949, best error:      9798.54883 on epoch:    39, epoch:   41/50, mu: 0.9160394669, learning_rate: 0.0000148185, weight_decay: 0.0000311534

epoch time: 4168.6479492188s, input_fired_time: 43.3633880615, output_fired_time: 31.0724468231, propagate_forward_time: 1624.7423095703, propagate_backward_time: 2430.1545410156, weight_update_time: 0.9981048107, other_time: 39.3151855469
epoch time: 337.0308532715s, input_fired_time: 3.4056925774, output_fired_time: 0.0000000000, propagate_forward_time: 327.2651977539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3599548340
validation[          , genome  5966] predictions:    6601/  10000 (66.01%), best:    6681/10000 (66.81%), error:      9917.66406, best error:      9798.54883 on epoch:    39, epoch:   42/50, mu: 0.9193306565, learning_rate: 0.0000135463, weight_decay: 0.0000291757

epoch time: 4170.9575195313s, input_fired_time: 47.8605575562, output_fired_time: 23.7156658173, propagate_forward_time: 1616.7855224609, propagate_backward_time: 2443.3352050781, weight_update_time: 1.0237420797, other_time: 39.2604980469
epoch time: 333.7894592285s, input_fired_time: 3.4028394222, output_fired_time: 0.0000000000, propagate_forward_time: 323.1864013672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2002258301
validation[          , genome  5966] predictions:    6602/  10000 (66.02%), best:    6681/10000 (66.81%), error:      9900.64453, best error:      9798.54883 on epoch:    39, epoch:   43/50, mu: 0.9224753976, learning_rate: 0.0000123833, weight_decay: 0.0000273235

epoch time: 4167.3266601563s, input_fired_time: 45.2027206421, output_fired_time: 26.9543876648, propagate_forward_time: 1608.1293945313, propagate_backward_time: 2442.7761230469, weight_update_time: 1.0376836061, other_time: 44.2641601563
epoch time: 331.6269226074s, input_fired_time: 3.3640015125, output_fired_time: 0.0000000000, propagate_forward_time: 321.1651306152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0977783203
validation[          , genome  5966] predictions:    6564/  10000 (65.64%), best:    6681/10000 (66.81%), error:      9957.99902, best error:      9798.54883 on epoch:    39, epoch:   44/50, mu: 0.9254801869, learning_rate: 0.0000113202, weight_decay: 0.0000255889

epoch time: 4167.9648437500s, input_fired_time: 48.4376983643, output_fired_time: 26.8131332397, propagate_forward_time: 1617.0755615234, propagate_backward_time: 2436.7031250000, weight_update_time: 2.0240507126, other_time: 38.9355468750
epoch time: 335.0895385742s, input_fired_time: 3.4171822071, output_fired_time: 0.0000000000, propagate_forward_time: 325.2984619141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3739013672
validation[          , genome  5966] predictions:    6649/  10000 (66.49%), best:    6681/10000 (66.81%), error:      9835.48340, best error:      9798.54883 on epoch:    39, epoch:   45/50, mu: 0.9283512831, learning_rate: 0.0000103483, weight_decay: 0.0000239645

epoch time: 4163.9169921875s, input_fired_time: 45.3775291443, output_fired_time: 23.5798149109, propagate_forward_time: 1618.7097167969, propagate_backward_time: 2438.5234375000, weight_update_time: 1.0307773352, other_time: 37.7265625000
epoch time: 338.5287170410s, input_fired_time: 3.4428594112, output_fired_time: 0.0000000000, propagate_forward_time: 328.9305419922, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1553039551
validation[          , genome  5966] predictions:    6576/  10000 (65.76%), best:    6681/10000 (66.81%), error:     10003.55176, best error:      9798.54883 on epoch:    39, epoch:   46/50, mu: 0.9310946465, learning_rate: 0.0000094599, weight_decay: 0.0000224432

epoch time: 4163.3530273438s, input_fired_time: 46.3533744812, output_fired_time: 24.9949684143, propagate_forward_time: 1617.3698730469, propagate_backward_time: 2434.0056152344, weight_update_time: 1.0191944838, other_time: 40.6291503906
epoch time: 337.3878784180s, input_fired_time: 3.4186501503, output_fired_time: 0.0000000000, propagate_forward_time: 326.6918640137, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2773742676
validation[          , genome  5966] predictions:    6620/  10000 (66.20%), best:    6681/10000 (66.81%), error:      9905.62012, best error:      9798.54883 on epoch:    39, epoch:   47/50, mu: 0.9337158799, learning_rate: 0.0000086478, weight_decay: 0.0000210184

epoch time: 4174.6582031250s, input_fired_time: 48.5593299866, output_fired_time: 25.6730823517, propagate_forward_time: 1608.1805419922, propagate_backward_time: 2450.8422851563, weight_update_time: 1.0476223230, other_time: 41.4028320313
epoch time: 337.0837707520s, input_fired_time: 3.4183781147, output_fired_time: 0.0000000000, propagate_forward_time: 326.4243774414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2410278320
validation[          , genome  5966] predictions:    6692/  10000 (66.92%), best:    6692/10000 (66.92%), error:      9666.37988, best error:      9666.37988 on epoch:    48, epoch:   48/50, mu: 0.9362204671, learning_rate: 0.0000079054, weight_decay: 0.0000196841

epoch time: 4157.0683593750s, input_fired_time: 45.3307533264, output_fired_time: 27.8802757263, propagate_forward_time: 1619.3333740234, propagate_backward_time: 2421.6862792969, weight_update_time: 1.0130810738, other_time: 42.8376464844
epoch time: 335.3215332031s, input_fired_time: 3.4280405045, output_fired_time: 0.0000000000, propagate_forward_time: 325.4234313965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4700622559
validation[          , genome  5966] predictions:    6693/  10000 (66.93%), best:    6692/10000 (66.92%), error:      9715.59473, best error:      9666.37988 on epoch:    48, epoch:   49/50, mu: 0.9386136532, learning_rate: 0.0000072267, weight_decay: 0.0000184345

epoch time: 4170.7910156250s, input_fired_time: 52.5045890808, output_fired_time: 25.9517879486, propagate_forward_time: 1615.5854492188, propagate_backward_time: 2442.0336914063, weight_update_time: 1.0188634396, other_time: 34.7155761719
epoch time: 335.8044433594s, input_fired_time: 3.4439172745, output_fired_time: 0.0000000000, propagate_forward_time: 326.0982971191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2622375488
validation[          , genome  5966] predictions:    6658/  10000 (66.58%), best:    6692/10000 (66.92%), error:      9774.34570, best error:      9666.37988 on epoch:    48, epoch:   50/50, mu: 0.9409003258, learning_rate: 0.0000066063, weight_decay: 0.0000172642

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1661.4864501953s, input_fired_time: 17.8706626892, output_fired_time: 0.0000000000, propagate_forward_time: 1609.3393554688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 34.2764892578
best training[          , genome  5966] predictions:   35506/  50000 (71.01%), best:    6692/10000 (66.92%), error:     43769.08984, best error:      9666.37988 on epoch:    48, epoch:   51/50, mu: 0.9430852532, learning_rate: 0.0000060391, weight_decay: 0.0000161682
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 341.5882263184s, input_fired_time: 3.4480495453, output_fired_time: 0.0000000000, propagate_forward_time: 324.4645385742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.6756286621
   testing[          , genome  5966] predictions:    6692/  10000 (66.92%), best:    6692/10000 (66.92%), error:      9666.37988, best error:      9666.37988 on epoch:    48, epoch:   51/50, mu: 0.9430852532, learning_rate: 0.0000060391, weight_decay: 0.0000161682
backpropagation finished successfully!
05:17:13 (4168): called boinc_finish(0)

</stderr_txt>
]]>