Nameexact_genome_1542649503_177_6360_2
Workunit4099951
Created20 Nov 2018, 4:56:17 UTC
Sent20 Nov 2018, 5:07:52 UTC
Report deadline13 Jan 2019, 16:23:36 UTC
Received23 Nov 2018, 22:22:17 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID31505
Run time20 hours 11 min 43 sec
CPU time20 hours 1 min 19 sec
Validate stateValid
Credit13,617.32
Device peak FLOPS4.20 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size527.58 MB
Peak swap size525.30 MB
Peak disk usage12.74 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_177_6360.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_177_6360_2_r1532363570_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_177_6360.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.5
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0125
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.0005
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6360
read normal distribution: '1 0 0'
generator_str: '1652287907'
read generator: 1652287907
read generated_by_map:
1 disable_edge 1
reading 104 nodes.
reading 420 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 891813592
post shuffle 1: 241453981
epoch time: 1302.02575683594s, input_fired_time: 10.6214466094971, output_fired_time: 10.0492849349976, propagate_forward_time: 491.682525634766, propagate_backward_time: 777.098999023438, weight_update_time: 1.19950926303864, other_time: 12.573486328125
epoch time: 101.888572692871s, input_fired_time: 1.28978729248047, output_fired_time: 0, propagate_forward_time: 98.3706817626953, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.22810363769531
validation[          , genome  6360] predictions:    4652/  10000 (46.52%), best:    4652/10000 (46.52%), error:     20228.91602, best error:     20228.91602 on epoch:     0, epoch:    0/50, mu: 0.5000000000, learning_rate: 0.0125000002, weight_decay: 0.0005000000

epoch time: 1303.5506591797s, input_fired_time: 10.5430192947, output_fired_time: 10.0787506104, propagate_forward_time: 492.8442993164, propagate_backward_time: 777.4712524414, weight_update_time: 1.1879858971, other_time: 12.6134033203
epoch time: 102.0497665405s, input_fired_time: 1.2932827473, output_fired_time: 0.0000000000, propagate_forward_time: 98.5245361328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2319488525
validation[          , genome  6360] predictions:    5065/  10000 (50.65%), best:    5065/10000 (50.65%), error:     17131.96680, best error:     17131.96680 on epoch:     1, epoch:    1/50, mu: 0.5245000124, learning_rate: 0.0118749999, weight_decay: 0.0004750000

epoch time: 1404.9865722656s, input_fired_time: 11.1462221146, output_fired_time: 11.1424026489, propagate_forward_time: 532.2688598633, propagate_backward_time: 836.8815917969, weight_update_time: 1.2697455883, other_time: 13.5474243164
epoch time: 102.1164932251s, input_fired_time: 1.2826167345, output_fired_time: 0.0000000000, propagate_forward_time: 98.6116867065, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2221908569
validation[          , genome  6360] predictions:    5151/  10000 (51.51%), best:    5151/10000 (51.51%), error:     16166.14746, best error:     16166.14746 on epoch:     2, epoch:    2/50, mu: 0.5477750301, learning_rate: 0.0112812500, weight_decay: 0.0004512500

epoch time: 1302.8721923828s, input_fired_time: 10.4272203445, output_fired_time: 10.0836992264, propagate_forward_time: 492.4264831543, propagate_backward_time: 777.3206176758, weight_update_time: 1.1903097630, other_time: 12.6140747070
epoch time: 101.8698730469s, input_fired_time: 1.2725805044, output_fired_time: 0.0000000000, propagate_forward_time: 98.3701400757, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2271499634
validation[          , genome  6360] predictions:    5193/  10000 (51.93%), best:    5193/10000 (51.93%), error:     15028.86719, best error:     15028.86719 on epoch:     3, epoch:    3/50, mu: 0.5698862672, learning_rate: 0.0107171871, weight_decay: 0.0004286875

epoch time: 1307.2800292969s, input_fired_time: 10.4507236481, output_fired_time: 10.1009483337, propagate_forward_time: 494.3397521973, propagate_backward_time: 779.7294921875, weight_update_time: 1.1961908340, other_time: 12.6591796875
epoch time: 102.1675643921s, input_fired_time: 1.2745403051, output_fired_time: 0.0000000000, propagate_forward_time: 98.6650466919, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2279739380
validation[          , genome  6360] predictions:    5226/  10000 (52.26%), best:    5193/10000 (51.93%), error:     15376.65625, best error:     15028.86719 on epoch:     3, epoch:    4/50, mu: 0.5908919573, learning_rate: 0.0101813274, weight_decay: 0.0004072531

epoch time: 1303.1464843750s, input_fired_time: 10.4013834000, output_fired_time: 10.0972185135, propagate_forward_time: 492.8219604492, propagate_backward_time: 777.1681518555, weight_update_time: 1.1881471872, other_time: 12.6578369141
epoch time: 101.9832534790s, input_fired_time: 1.2664759159, output_fired_time: 0.0000000000, propagate_forward_time: 98.5026931763, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2140808105
validation[          , genome  6360] predictions:    5310/  10000 (53.10%), best:    5310/10000 (53.10%), error:     14613.31641, best error:     14613.31641 on epoch:     5, epoch:    5/50, mu: 0.6108473539, learning_rate: 0.0096722608, weight_decay: 0.0003868904

epoch time: 1305.2849121094s, input_fired_time: 10.3975734711, output_fired_time: 10.0717592239, propagate_forward_time: 493.4976501465, propagate_backward_time: 778.7138671875, weight_update_time: 1.1850862503, other_time: 12.6040039063
epoch time: 101.9090576172s, input_fired_time: 1.2613296509, output_fired_time: 0.0000000000, propagate_forward_time: 98.4340209961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2137069702
validation[          , genome  6360] predictions:    5049/  10000 (50.49%), best:    5310/10000 (53.10%), error:     16048.44629, best error:     14613.31641 on epoch:     5, epoch:    6/50, mu: 0.6298049688, learning_rate: 0.0091886474, weight_decay: 0.0003675459

epoch time: 1304.0255126953s, input_fired_time: 10.3940954208, output_fired_time: 10.0669946671, propagate_forward_time: 492.8239135742, propagate_backward_time: 778.1338500977, weight_update_time: 1.1890265942, other_time: 12.6066894531
epoch time: 102.2574768066s, input_fired_time: 1.2687890530, output_fired_time: 0.0000000000, propagate_forward_time: 98.7669296265, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2217559814
validation[          , genome  6360] predictions:    5239/  10000 (52.39%), best:    5310/10000 (53.10%), error:     15925.42773, best error:     14613.31641 on epoch:     5, epoch:    7/50, mu: 0.6478147507, learning_rate: 0.0087292148, weight_decay: 0.0003491686

epoch time: 1305.5277099609s, input_fired_time: 10.4077529907, output_fired_time: 10.0810241699, propagate_forward_time: 493.1694946289, propagate_backward_time: 779.2428588867, weight_update_time: 1.1926088333, other_time: 12.6265869141
epoch time: 102.1512832642s, input_fired_time: 1.2685236931, output_fired_time: 0.0000000000, propagate_forward_time: 98.6558074951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2269515991
validation[          , genome  6360] predictions:    5171/  10000 (51.71%), best:    5310/10000 (53.10%), error:     15192.29980, best error:     14613.31641 on epoch:     5, epoch:    8/50, mu: 0.6649240255, learning_rate: 0.0082927542, weight_decay: 0.0003317102

epoch time: 1305.2960205078s, input_fired_time: 10.4290876389, output_fired_time: 10.0787935257, propagate_forward_time: 493.6365661621, propagate_backward_time: 778.4498291016, weight_update_time: 1.1999855042, other_time: 12.7017822266
epoch time: 101.9816436768s, input_fired_time: 1.2804784775, output_fired_time: 0.0000000000, propagate_forward_time: 98.4709396362, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2302246094
validation[          , genome  6360] predictions:    5349/  10000 (53.49%), best:    5310/10000 (53.10%), error:     14847.42578, best error:     14613.31641 on epoch:     5, epoch:    9/50, mu: 0.6811778545, learning_rate: 0.0078781163, weight_decay: 0.0003151247

epoch time: 1306.1683349609s, input_fired_time: 10.3973588943, output_fired_time: 10.0835762024, propagate_forward_time: 493.3922119141, propagate_backward_time: 779.6513061523, weight_update_time: 1.1909810305, other_time: 12.6438598633
epoch time: 102.0816268921s, input_fired_time: 1.2704578638, output_fired_time: 0.0000000000, propagate_forward_time: 98.5820693970, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2291030884
validation[          , genome  6360] predictions:    5121/  10000 (51.21%), best:    5310/10000 (53.10%), error:     16139.59961, best error:     14613.31641 on epoch:     5, epoch:   10/50, mu: 0.6966189742, learning_rate: 0.0074842102, weight_decay: 0.0002993684

epoch time: 1331.2307128906s, input_fired_time: 10.5021705627, output_fired_time: 10.3228244781, propagate_forward_time: 502.4315185547, propagate_backward_time: 795.0819702148, weight_update_time: 1.2189668417, other_time: 12.8921508789
epoch time: 102.3821182251s, input_fired_time: 1.2664910555, output_fired_time: 0.0000000000, propagate_forward_time: 98.8794555664, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2361679077
validation[          , genome  6360] predictions:    5394/  10000 (53.94%), best:    5394/10000 (53.94%), error:     14588.62305, best error:     14588.62305 on epoch:    11, epoch:   11/50, mu: 0.7112880349, learning_rate: 0.0071099997, weight_decay: 0.0002844000

epoch time: 1306.8511962891s, input_fired_time: 10.4443216324, output_fired_time: 10.0459709167, propagate_forward_time: 494.2394104004, propagate_backward_time: 779.4943237305, weight_update_time: 1.1948801279, other_time: 12.6271362305
epoch time: 102.0109939575s, input_fired_time: 1.2736740112, output_fired_time: 0.0000000000, propagate_forward_time: 98.5105743408, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2267456055
validation[          , genome  6360] predictions:    5386/  10000 (53.86%), best:    5386/10000 (53.86%), error:     14374.08105, best error:     14374.08105 on epoch:    12, epoch:   12/50, mu: 0.7252236605, learning_rate: 0.0067544994, weight_decay: 0.0002701800

epoch time: 1306.8553466797s, input_fired_time: 10.4439649582, output_fired_time: 10.0777740479, propagate_forward_time: 494.0200195313, propagate_backward_time: 779.6702880859, weight_update_time: 1.1913270950, other_time: 12.6433105469
epoch time: 102.6253662109s, input_fired_time: 1.2793369293, output_fired_time: 0.0000000000, propagate_forward_time: 99.1155700684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2304611206
validation[          , genome  6360] predictions:    5490/  10000 (54.90%), best:    5490/10000 (54.90%), error:     13754.34082, best error:     13754.34082 on epoch:    13, epoch:   13/50, mu: 0.7384625077, learning_rate: 0.0064167744, weight_decay: 0.0002566710

epoch time: 1305.7668457031s, input_fired_time: 10.4503517151, output_fired_time: 10.0495367050, propagate_forward_time: 493.5398864746, propagate_backward_time: 779.0971679688, weight_update_time: 1.1911149025, other_time: 12.6298828125
epoch time: 101.9239807129s, input_fired_time: 1.2751373053, output_fired_time: 0.0000000000, propagate_forward_time: 98.4398498535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2089920044
validation[          , genome  6360] predictions:    5407/  10000 (54.07%), best:    5490/10000 (54.90%), error:     14531.12402, best error:     13754.34082 on epoch:    13, epoch:   14/50, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 1307.5848388672s, input_fired_time: 10.4650964737, output_fired_time: 10.0984754562, propagate_forward_time: 494.1943359375, propagate_backward_time: 780.1553344727, weight_update_time: 1.1975953579, other_time: 12.6715698242
epoch time: 102.2315979004s, input_fired_time: 1.2751703262, output_fired_time: 0.0000000000, propagate_forward_time: 98.7276840210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2287445068
validation[          , genome  6360] predictions:    5512/  10000 (55.12%), best:    5512/10000 (55.12%), error:     13581.68164, best error:     13581.68164 on epoch:    15, epoch:   15/50, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 1306.7886962891s, input_fired_time: 10.4877605438, output_fired_time: 10.0996789932, propagate_forward_time: 494.2189941406, propagate_backward_time: 779.3204956055, weight_update_time: 1.1941967010, other_time: 12.6616821289
epoch time: 102.0887908936s, input_fired_time: 1.2786283493, output_fired_time: 0.0000000000, propagate_forward_time: 98.5969390869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2132263184
validation[          , genome  6360] predictions:    5521/  10000 (55.21%), best:    5512/10000 (55.12%), error:     14458.58398, best error:     13581.68164 on epoch:    15, epoch:   16/50, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 1304.4406738281s, input_fired_time: 10.4492530823, output_fired_time: 10.0589714050, propagate_forward_time: 493.2588195801, propagate_backward_time: 778.0510253906, weight_update_time: 1.1913015842, other_time: 12.6226806641
epoch time: 102.3815231323s, input_fired_time: 1.2856653929, output_fired_time: 0.0000000000, propagate_forward_time: 98.8745346069, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2213211060
validation[          , genome  6360] predictions:    5721/  10000 (57.21%), best:    5721/10000 (57.21%), error:     12985.94629, best error:     12985.94629 on epoch:    17, epoch:   17/50, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 1306.0687255859s, input_fired_time: 10.4941282272, output_fired_time: 10.1005039215, propagate_forward_time: 493.8522033691, propagate_backward_time: 778.9246826172, weight_update_time: 1.1920387745, other_time: 12.6972656250
epoch time: 101.7828903198s, input_fired_time: 1.2706495523, output_fired_time: 0.0000000000, propagate_forward_time: 98.2927093506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2195281982
validation[          , genome  6360] predictions:    5387/  10000 (53.87%), best:    5721/10000 (57.21%), error:     15525.49902, best error:     12985.94629 on epoch:    17, epoch:   18/50, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 1310.7177734375s, input_fired_time: 10.5252799988, output_fired_time: 10.0894842148, propagate_forward_time: 495.7170104980, propagate_backward_time: 781.7290649414, weight_update_time: 1.1915059090, other_time: 12.6569213867
epoch time: 102.2285995483s, input_fired_time: 1.2849338055, output_fired_time: 0.0000000000, propagate_forward_time: 98.7271041870, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2165603638
validation[          , genome  6360] predictions:    5675/  10000 (56.75%), best:    5721/10000 (57.21%), error:     13447.14941, best error:     12985.94629 on epoch:    17, epoch:   19/50, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

epoch time: 1305.7352294922s, input_fired_time: 10.4811840057, output_fired_time: 10.0833892822, propagate_forward_time: 493.5736389160, propagate_backward_time: 778.9510498047, weight_update_time: 1.1934939623, other_time: 12.6459960938
epoch time: 102.3224182129s, input_fired_time: 1.2959672213, output_fired_time: 0.0000000000, propagate_forward_time: 98.8027343750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2237167358
validation[          , genome  6360] predictions:    5586/  10000 (55.86%), best:    5721/10000 (57.21%), error:     13916.75293, best error:     12985.94629 on epoch:    17, epoch:   20/50, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 1306.1875000000s, input_fired_time: 10.5232429504, output_fired_time: 10.0674304962, propagate_forward_time: 493.6678161621, propagate_backward_time: 779.2739257813, weight_update_time: 1.1935956478, other_time: 12.6551513672
epoch time: 102.1423645020s, input_fired_time: 1.2967420816, output_fired_time: 0.0000000000, propagate_forward_time: 98.6211166382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2245025635
validation[          , genome  6360] predictions:    5606/  10000 (56.06%), best:    5721/10000 (57.21%), error:     13794.24512, best error:     12985.94629 on epoch:    17, epoch:   21/50, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 1306.9134521484s, input_fired_time: 10.5387153625, output_fired_time: 10.0621652603, propagate_forward_time: 493.8628234863, propagate_backward_time: 779.7762451172, weight_update_time: 1.1914879084, other_time: 12.6735839844
epoch time: 101.7690277100s, input_fired_time: 1.2955495119, output_fired_time: 0.0000000000, propagate_forward_time: 98.2647933960, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2086868286
validation[          , genome  6360] predictions:    5766/  10000 (57.66%), best:    5721/10000 (57.21%), error:     13107.57422, best error:     12985.94629 on epoch:    17, epoch:   22/50, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 1306.1008300781s, input_fired_time: 10.5394458771, output_fired_time: 10.0680828094, propagate_forward_time: 493.5220336914, propagate_backward_time: 779.3143310547, weight_update_time: 1.1855734587, other_time: 12.6569213867
epoch time: 103.0284957886s, input_fired_time: 1.3037712574, output_fired_time: 0.0000000000, propagate_forward_time: 99.4768905640, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2478332520
validation[          , genome  6360] predictions:    5791/  10000 (57.91%), best:    5721/10000 (57.21%), error:     13216.89453, best error:     12985.94629 on epoch:    17, epoch:   23/50, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 1305.5850830078s, input_fired_time: 10.5522775650, output_fired_time: 10.1008110046, propagate_forward_time: 493.5761108398, propagate_backward_time: 778.7457275391, weight_update_time: 1.1892654896, other_time: 12.6101684570
epoch time: 102.0783233643s, input_fired_time: 1.2956423759, output_fired_time: 0.0000000000, propagate_forward_time: 98.5628051758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2198791504
validation[          , genome  6360] predictions:    5623/  10000 (56.23%), best:    5721/10000 (57.21%), error:     13785.58594, best error:     12985.94629 on epoch:    17, epoch:   24/50, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 1306.6218261719s, input_fired_time: 10.5599632263, output_fired_time: 10.0754404068, propagate_forward_time: 493.8831787109, propagate_backward_time: 779.4710083008, weight_update_time: 1.1909656525, other_time: 12.6322631836
epoch time: 102.0765609741s, input_fired_time: 1.3023520708, output_fired_time: 0.0000000000, propagate_forward_time: 98.5547409058, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2194671631
validation[          , genome  6360] predictions:    5702/  10000 (57.02%), best:    5721/10000 (57.21%), error:     13355.41504, best error:     12985.94629 on epoch:    17, epoch:   25/50, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 1306.9077148438s, input_fired_time: 10.5831937790, output_fired_time: 10.0823135376, propagate_forward_time: 493.9842224121, propagate_backward_time: 779.6236572266, weight_update_time: 1.1869862080, other_time: 12.6342773438
epoch time: 102.0371475220s, input_fired_time: 1.2938400507, output_fired_time: 0.0000000000, propagate_forward_time: 98.5145263672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2287826538
validation[          , genome  6360] predictions:    5830/  10000 (58.30%), best:    5721/10000 (57.21%), error:     13043.15039, best error:     12985.94629 on epoch:    17, epoch:   26/50, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 1311.8145751953s, input_fired_time: 10.5910177231, output_fired_time: 10.0915803909, propagate_forward_time: 495.6897888184, propagate_backward_time: 782.7906494141, weight_update_time: 1.1890028715, other_time: 12.6514892578
epoch time: 102.1596145630s, input_fired_time: 1.3121048212, output_fired_time: 0.0000000000, propagate_forward_time: 98.6192245483, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2282867432
validation[          , genome  6360] predictions:    5854/  10000 (58.54%), best:    5721/10000 (57.21%), error:     13159.89453, best error:     12985.94629 on epoch:    17, epoch:   27/50, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 1305.1888427734s, input_fired_time: 10.5880241394, output_fired_time: 10.0848503113, propagate_forward_time: 493.5269775391, propagate_backward_time: 778.3802490234, weight_update_time: 1.1872111559, other_time: 12.6087646484
epoch time: 101.9214935303s, input_fired_time: 1.3008016348, output_fired_time: 0.0000000000, propagate_forward_time: 98.4173965454, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2032928467
validation[          , genome  6360] predictions:    5644/  10000 (56.44%), best:    5721/10000 (57.21%), error:     14033.79492, best error:     12985.94629 on epoch:    17, epoch:   28/50, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1307.2427978516s, input_fired_time: 10.6051845551, output_fired_time: 10.0761451721, propagate_forward_time: 494.3768615723, propagate_backward_time: 779.5738525391, weight_update_time: 1.1953700781, other_time: 12.6107177734
epoch time: 102.0310668945s, input_fired_time: 1.3052146435, output_fired_time: 0.0000000000, propagate_forward_time: 98.4885940552, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2372589111
validation[          , genome  6360] predictions:    5769/  10000 (57.69%), best:    5721/10000 (57.21%), error:     13040.66309, best error:     12985.94629 on epoch:    17, epoch:   29/50, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 1304.3541259766s, input_fired_time: 10.5953311920, output_fired_time: 10.0477533340, propagate_forward_time: 493.1539611816, propagate_backward_time: 777.9407958984, weight_update_time: 1.1881185770, other_time: 12.6163330078
epoch time: 102.1889877319s, input_fired_time: 1.3033082485, output_fired_time: 0.0000000000, propagate_forward_time: 98.6561584473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2295227051
validation[          , genome  6360] predictions:    5792/  10000 (57.92%), best:    5721/10000 (57.21%), error:     13532.82910, best error:     12985.94629 on epoch:    17, epoch:   30/50, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 1312.7668457031s, input_fired_time: 10.6815052032, output_fired_time: 10.1183090210, propagate_forward_time: 496.7910766602, propagate_backward_time: 782.4916381836, weight_update_time: 1.1974648237, other_time: 12.6843261719
epoch time: 102.0853347778s, input_fired_time: 1.3130221367, output_fired_time: 0.0000000000, propagate_forward_time: 98.5280303955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2442855835
validation[          , genome  6360] predictions:    5871/  10000 (58.71%), best:    5871/10000 (58.71%), error:     12763.05469, best error:     12763.05469 on epoch:    31, epoch:   31/50, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 1307.5517578125s, input_fired_time: 10.6455478668, output_fired_time: 10.1201915741, propagate_forward_time: 493.7696838379, propagate_backward_time: 780.2973022461, weight_update_time: 1.1901142597, other_time: 12.7190551758
epoch time: 102.3920288086s, input_fired_time: 1.3205844164, output_fired_time: 0.0000000000, propagate_forward_time: 98.8461456299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2252960205
validation[          , genome  6360] predictions:    5924/  10000 (59.24%), best:    5871/10000 (58.71%), error:     12978.53906, best error:     12763.05469 on epoch:    31, epoch:   32/50, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 1308.2139892578s, input_fired_time: 10.6591138840, output_fired_time: 10.0905971527, propagate_forward_time: 494.1847839355, propagate_backward_time: 780.6515502930, weight_update_time: 1.1911000013, other_time: 12.6279907227
epoch time: 102.2991485596s, input_fired_time: 1.3138058186, output_fired_time: 0.0000000000, propagate_forward_time: 98.7736434937, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2117004395
validation[          , genome  6360] predictions:    5944/  10000 (59.44%), best:    5944/10000 (59.44%), error:     12621.46484, best error:     12621.46484 on epoch:    33, epoch:   33/50, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 1311.9897460938s, input_fired_time: 10.6953449249, output_fired_time: 10.1442670822, propagate_forward_time: 496.1520996094, propagate_backward_time: 782.3187255859, weight_update_time: 1.1916750669, other_time: 12.6793212891
epoch time: 102.0805740356s, input_fired_time: 1.3188695908, output_fired_time: 0.0000000000, propagate_forward_time: 98.5207443237, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2409591675
validation[          , genome  6360] predictions:    5934/  10000 (59.34%), best:    5944/10000 (59.44%), error:     12707.94824, best error:     12621.46484 on epoch:    33, epoch:   34/50, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 1307.8242187500s, input_fired_time: 10.6953649521, output_fired_time: 10.1090478897, propagate_forward_time: 494.2306823730, propagate_backward_time: 780.1511840820, weight_update_time: 1.1922190189, other_time: 12.6380004883
epoch time: 102.1791687012s, input_fired_time: 1.3235125542, output_fired_time: 0.0000000000, propagate_forward_time: 98.6421432495, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2135162354
validation[          , genome  6360] predictions:    5854/  10000 (58.54%), best:    5944/10000 (59.44%), error:     13301.09180, best error:     12621.46484 on epoch:    33, epoch:   35/50, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 1310.4415283203s, input_fired_time: 10.6996107101, output_fired_time: 10.1160917282, propagate_forward_time: 494.8776245117, propagate_backward_time: 781.9724121094, weight_update_time: 1.2074066401, other_time: 12.7758178711
epoch time: 102.4817276001s, input_fired_time: 1.3269397020, output_fired_time: 0.0000000000, propagate_forward_time: 98.9261627197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2286224365
validation[          , genome  6360] predictions:    5937/  10000 (59.37%), best:    5944/10000 (59.44%), error:     12653.94824, best error:     12621.46484 on epoch:    33, epoch:   36/50, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 1307.9169921875s, input_fired_time: 10.7028989792, output_fired_time: 10.1281394958, propagate_forward_time: 494.0348510742, propagate_backward_time: 780.4159545898, weight_update_time: 1.1892210245, other_time: 12.6351318359
epoch time: 102.3720169067s, input_fired_time: 1.3229684830, output_fired_time: 0.0000000000, propagate_forward_time: 98.8340835571, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2149658203
validation[          , genome  6360] predictions:    5932/  10000 (59.32%), best:    5944/10000 (59.44%), error:     12682.54590, best error:     12621.46484 on epoch:    33, epoch:   37/50, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 1308.5830078125s, input_fired_time: 10.7187719345, output_fired_time: 10.1031913757, propagate_forward_time: 494.3792419434, propagate_backward_time: 780.6523437500, weight_update_time: 1.1990422010, other_time: 12.7294921875
epoch time: 102.2761611938s, input_fired_time: 1.3345850706, output_fired_time: 0.0000000000, propagate_forward_time: 98.7009735107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2406005859
validation[          , genome  6360] predictions:    5856/  10000 (58.56%), best:    5944/10000 (59.44%), error:     12888.94336, best error:     12621.46484 on epoch:    33, epoch:   38/50, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 1310.6783447266s, input_fired_time: 10.7377290726, output_fired_time: 10.1585378647, propagate_forward_time: 495.1316223145, propagate_backward_time: 781.9676513672, weight_update_time: 1.1903231144, other_time: 12.6828613281
epoch time: 102.3004226685s, input_fired_time: 1.3340574503, output_fired_time: 0.0000000000, propagate_forward_time: 98.7271041870, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2392578125
validation[          , genome  6360] predictions:    5850/  10000 (58.50%), best:    5944/10000 (59.44%), error:     12776.46973, best error:     12621.46484 on epoch:    33, epoch:   39/50, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 1309.1502685547s, input_fired_time: 10.7395257950, output_fired_time: 10.1002416611, propagate_forward_time: 494.8596496582, propagate_backward_time: 780.7554931641, weight_update_time: 1.1982972622, other_time: 12.6954345703
epoch time: 102.5129013062s, input_fired_time: 1.3281549215, output_fired_time: 0.0000000000, propagate_forward_time: 98.9491729736, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2355728149
validation[          , genome  6360] predictions:    5877/  10000 (58.77%), best:    5944/10000 (59.44%), error:     12941.71191, best error:     12621.46484 on epoch:    33, epoch:   40/50, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 1315.3697509766s, input_fired_time: 10.7858304977, output_fired_time: 10.2050800323, propagate_forward_time: 496.8134155273, propagate_backward_time: 784.8635253906, weight_update_time: 1.1962319613, other_time: 12.7018432617
epoch time: 102.6301727295s, input_fired_time: 1.3392138481, output_fired_time: 0.0000000000, propagate_forward_time: 99.0483932495, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2425689697
validation[          , genome  6360] predictions:    5983/  10000 (59.83%), best:    5944/10000 (59.44%), error:     12649.37305, best error:     12621.46484 on epoch:    33, epoch:   41/50, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 1311.7587890625s, input_fired_time: 10.7885847092, output_fired_time: 10.1427383423, propagate_forward_time: 495.3682861328, propagate_backward_time: 782.7659912109, weight_update_time: 1.1967326403, other_time: 12.6932373047
epoch time: 103.1209869385s, input_fired_time: 1.3426643610, output_fired_time: 0.0000000000, propagate_forward_time: 99.5272674561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2510528564
validation[          , genome  6360] predictions:    5887/  10000 (58.87%), best:    5887/10000 (58.87%), error:     12572.97363, best error:     12572.97363 on epoch:    42, epoch:   42/50, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 1314.1635742188s, input_fired_time: 10.7967720032, output_fired_time: 10.2130317688, propagate_forward_time: 495.9672546387, propagate_backward_time: 784.4613037109, weight_update_time: 1.1979348660, other_time: 12.7252197266
epoch time: 102.8527984619s, input_fired_time: 1.3474938869, output_fired_time: 0.0000000000, propagate_forward_time: 99.2333755493, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2719268799
validation[          , genome  6360] predictions:    5972/  10000 (59.72%), best:    5972/10000 (59.72%), error:     12497.36328, best error:     12497.36328 on epoch:    43, epoch:   43/50, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 1312.1135253906s, input_fired_time: 10.7981719971, output_fired_time: 10.1587009430, propagate_forward_time: 495.3508300781, propagate_backward_time: 783.0517578125, weight_update_time: 1.2047618628, other_time: 12.7540283203
epoch time: 102.7840423584s, input_fired_time: 1.3573868275, output_fired_time: 0.0000000000, propagate_forward_time: 99.1829528809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2437057495
validation[          , genome  6360] predictions:    6104/  10000 (61.04%), best:    6104/10000 (61.04%), error:     12139.26758, best error:     12139.26758 on epoch:    44, epoch:   44/50, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 1311.8999023438s, input_fired_time: 10.8213586807, output_fired_time: 10.1892528534, propagate_forward_time: 495.5743713379, propagate_backward_time: 782.6181640625, weight_update_time: 1.1955673695, other_time: 12.6967773438
epoch time: 102.8715744019s, input_fired_time: 1.3481228352, output_fired_time: 0.0000000000, propagate_forward_time: 99.2799606323, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2434921265
validation[          , genome  6360] predictions:    6116/  10000 (61.16%), best:    6104/10000 (61.04%), error:     12155.97754, best error:     12139.26758 on epoch:    44, epoch:   45/50, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 1312.4559326172s, input_fired_time: 10.8173885345, output_fired_time: 10.2406167984, propagate_forward_time: 495.4990844727, propagate_backward_time: 783.1797485352, weight_update_time: 1.1894721985, other_time: 12.7191162109
epoch time: 102.5880432129s, input_fired_time: 1.3526456356, output_fired_time: 0.0000000000, propagate_forward_time: 98.9838180542, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2515792847
validation[          , genome  6360] predictions:    6137/  10000 (61.37%), best:    6104/10000 (61.04%), error:     12226.49414, best error:     12139.26758 on epoch:    44, epoch:   46/50, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 1312.2490234375s, input_fired_time: 10.8366708755, output_fired_time: 10.1742372513, propagate_forward_time: 495.1913452148, propagate_backward_time: 783.2591552734, weight_update_time: 1.1937494278, other_time: 12.7876586914
epoch time: 102.7810974121s, input_fired_time: 1.3546053171, output_fired_time: 0.0000000000, propagate_forward_time: 99.1777954102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2486953735
validation[          , genome  6360] predictions:    5791/  10000 (57.91%), best:    6104/10000 (61.04%), error:     13310.63574, best error:     12139.26758 on epoch:    44, epoch:   47/50, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 1312.2233886719s, input_fired_time: 10.8613367081, output_fired_time: 10.1739931107, propagate_forward_time: 495.7504577637, propagate_backward_time: 782.7125854492, weight_update_time: 1.1966887712, other_time: 12.7250366211
epoch time: 102.7081832886s, input_fired_time: 1.3619863987, output_fired_time: 0.0000000000, propagate_forward_time: 99.1278610229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2183380127
validation[          , genome  6360] predictions:    6197/  10000 (61.97%), best:    6197/10000 (61.97%), error:     11901.14941, best error:     11901.14941 on epoch:    48, epoch:   48/50, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 1312.6781005859s, input_fired_time: 10.8424720764, output_fired_time: 10.1806879044, propagate_forward_time: 495.6513366699, propagate_backward_time: 783.2973022461, weight_update_time: 1.2011586428, other_time: 12.7062377930
epoch time: 102.7481079102s, input_fired_time: 1.3597429991, output_fired_time: 0.0000000000, propagate_forward_time: 99.1606216431, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2277450562
validation[          , genome  6360] predictions:    6076/  10000 (60.76%), best:    6197/10000 (61.97%), error:     12146.19824, best error:     11901.14941 on epoch:    48, epoch:   49/50, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 1314.8486328125s, input_fired_time: 10.8939580917, output_fired_time: 10.2046365738, propagate_forward_time: 496.4504089355, propagate_backward_time: 784.5643920898, weight_update_time: 1.1959177256, other_time: 12.7352905273
epoch time: 102.7997436523s, input_fired_time: 1.3562446833, output_fired_time: 0.0000000000, propagate_forward_time: 99.2026443481, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2408523560
validation[          , genome  6360] predictions:    6023/  10000 (60.23%), best:    6197/10000 (61.97%), error:     12611.28418, best error:     11901.14941 on epoch:    48, epoch:   50/50, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 513.8615112305s, input_fired_time: 6.8224201202, output_fired_time: 0.0000000000, propagate_forward_time: 495.7922973633, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.2467956543
best training[          , genome  6360] predictions:   37783/  50000 (75.57%), best:    6197/10000 (61.97%), error:     35144.34375, best error:     11901.14941 on epoch:    48, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 102.7050399780s, input_fired_time: 1.3625050783, output_fired_time: 0.0000000000, propagate_forward_time: 99.1136779785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2288589478
   testing[          , genome  6360] predictions:    6197/  10000 (61.97%), best:    6197/10000 (61.97%), error:     11901.14941, best error:     11901.14941 on epoch:    48, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
backpropagation finished successfully!
07:21:16 (7440): called boinc_finish(0)

</stderr_txt>
]]>