Nameexact_genome_1542605141_178_6297_1
Workunit4098482
Created19 Nov 2018, 6:41:20 UTC
Sent19 Nov 2018, 7:05:33 UTC
Report deadline23 Jan 2019, 11:29:07 UTC
Received20 Nov 2018, 11:26:59 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84860
Run time1 days 3 hours 20 min 45 sec
CPU time1 days 3 hours 3 min 8 sec
Validate stateValid
Credit16,295.76
Device peak FLOPS4.41 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size546.54 MB
Peak swap size549.28 MB
Peak disk usage10.10 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6297.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6297_1_r190773997_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542605141_178_6297.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.494862
read mu: 0.494862
read mu_delta: 0.952503
read initial_learning_rate: 0.00157872
read learning_rate: 0.00157872
read learning_rate_delta: 0.921764
read initial_weight_decay: 0.00052458
read weight_decay: 0.00052458
read weight_decay_delta: 0.932195
read batch_size: 102
read epsilon: 1e-07
read alpha: 0.0854529
read input_dropout_probability: 0.00103773
read hidden_dropout_probability: 0.106457
read velocity_reset: 2261
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6297
read normal distribution: '0 -0.0915169343352318 -1.29089105129242'
generator_str: '1586422662'
read generator: 1586422662
read generated_by_map:
1 split_edge 1
reading 107 nodes.
reading 578 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1996202729
post shuffle 1: 230836572
epoch time: 1694.58959960938s, input_fired_time: 17.1289672851562, output_fired_time: 5.85455846786499, propagate_forward_time: 671.311889648438, propagate_backward_time: 987.502868652344, weight_update_time: 0.519583642482758, other_time: 12.7913208007812
epoch time: 140.327117919922s, input_fired_time: 0.846691310405731, output_fired_time: 0, propagate_forward_time: 137.2744140625, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.20600891113281
validation[          , genome  6297] predictions:    5708/  10000 (57.08%), best:    5708/10000 (57.08%), error:     12254.89258, best error:     12254.89258 on epoch:     0, epoch:    0/50, mu: 0.4948624074, learning_rate: 0.0015787220, weight_decay: 0.0005245801

epoch time: 1729.9763183594s, input_fired_time: 17.0006561279, output_fired_time: 6.0689229965, propagate_forward_time: 678.8466796875, propagate_backward_time: 1015.2128295898, weight_update_time: 0.5087064505, other_time: 12.8472290039
epoch time: 133.2006225586s, input_fired_time: 0.8635243177, output_fired_time: 0.0000000000, propagate_forward_time: 130.2289428711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1081542969
validation[          , genome  6297] predictions:    5448/  10000 (54.48%), best:    5708/10000 (57.08%), error:     13179.84961, best error:     12254.89258 on epoch:     0, epoch:    1/50, mu: 0.5183799267, learning_rate: 0.0014552096, weight_decay: 0.0004890112

epoch time: 1722.7017822266s, input_fired_time: 16.9804782867, output_fired_time: 6.0200119019, propagate_forward_time: 675.4514770508, propagate_backward_time: 1011.5946655273, weight_update_time: 0.5006516576, other_time: 12.6550903320
epoch time: 142.6699829102s, input_fired_time: 0.8821305037, output_fired_time: 0.0000000000, propagate_forward_time: 139.7819976807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0058593750
validation[          , genome  6297] predictions:    5599/  10000 (55.99%), best:    5708/10000 (57.08%), error:     12629.62891, best error:     12254.89258 on epoch:     0, epoch:    2/50, mu: 0.5407804251, learning_rate: 0.0013413602, weight_decay: 0.0004558541

epoch time: 1753.7264404297s, input_fired_time: 16.9857025146, output_fired_time: 6.0591540337, propagate_forward_time: 688.5953979492, propagate_backward_time: 1029.3995361328, weight_update_time: 0.4938474000, other_time: 12.6866455078
epoch time: 141.9184722900s, input_fired_time: 0.8360705972, output_fired_time: 0.0000000000, propagate_forward_time: 138.9321289062, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1502685547
validation[          , genome  6297] predictions:    5892/  10000 (58.92%), best:    5892/10000 (58.92%), error:     11856.57617, best error:     11856.57617 on epoch:     3, epoch:    3/50, mu: 0.5621169806, learning_rate: 0.0012364178, weight_decay: 0.0004249451

epoch time: 1731.4128417969s, input_fired_time: 17.3366680145, output_fired_time: 5.9322853088, propagate_forward_time: 690.5292358398, propagate_backward_time: 1004.9598388672, weight_update_time: 0.5037977099, other_time: 12.6548461914
epoch time: 141.6526794434s, input_fired_time: 0.9028250575, output_fired_time: 0.0000000000, propagate_forward_time: 138.6954040527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0544433594
validation[          , genome  6297] predictions:    5726/  10000 (57.26%), best:    5892/10000 (58.92%), error:     12494.06445, best error:     11856.57617 on epoch:     3, epoch:    4/50, mu: 0.5824401379, learning_rate: 0.0011396857, weight_decay: 0.0003961319

epoch time: 1738.0500488281s, input_fired_time: 16.9370307922, output_fired_time: 5.9848532677, propagate_forward_time: 685.2365112305, propagate_backward_time: 1017.3687744141, weight_update_time: 0.4911012053, other_time: 12.5228271484
epoch time: 133.9385528564s, input_fired_time: 0.8981891274, output_fired_time: 0.0000000000, propagate_forward_time: 130.9222564697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1181030273
validation[          , genome  6297] predictions:    6022/  10000 (60.22%), best:    6022/10000 (60.22%), error:     11466.52637, best error:     11466.52637 on epoch:     5, epoch:    5/50, mu: 0.6017979980, learning_rate: 0.0010505216, weight_decay: 0.0003692724

epoch time: 1757.2607421875s, input_fired_time: 17.6109046936, output_fired_time: 6.0257024765, propagate_forward_time: 699.3108520508, propagate_backward_time: 1021.5164184570, weight_update_time: 0.5215572119, other_time: 12.7968139648
epoch time: 141.5364074707s, input_fired_time: 0.8515607715, output_fired_time: 0.0000000000, propagate_forward_time: 138.5410461426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1437988281
validation[          , genome  6297] predictions:    5497/  10000 (54.97%), best:    6022/10000 (60.22%), error:     13138.14551, best error:     11466.52637 on epoch:     5, epoch:    6/50, mu: 0.6202363968, learning_rate: 0.0009683333, weight_decay: 0.0003442340

epoch time: 1763.5760498047s, input_fired_time: 17.1016616821, output_fired_time: 6.1378617287, propagate_forward_time: 693.0203857422, propagate_backward_time: 1034.7628173828, weight_update_time: 0.4906671047, other_time: 12.5533447266
epoch time: 131.6452331543s, input_fired_time: 0.8700998425, output_fired_time: 0.0000000000, propagate_forward_time: 128.7812194824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9939117432
validation[          , genome  6297] predictions:    5758/  10000 (57.58%), best:    6022/10000 (60.22%), error:     12140.96387, best error:     11466.52637 on epoch:     5, epoch:    7/50, mu: 0.6377990246, learning_rate: 0.0008925750, weight_decay: 0.0003208934

epoch time: 1765.4821777344s, input_fired_time: 17.0449504852, output_fired_time: 6.0687112808, propagate_forward_time: 694.0697021484, propagate_backward_time: 1035.5515136719, weight_update_time: 0.4894970655, other_time: 12.7473144531
epoch time: 137.2796020508s, input_fired_time: 0.8674720526, output_fired_time: 0.0000000000, propagate_forward_time: 134.4275665283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9845581055
validation[          , genome  6297] predictions:    5486/  10000 (54.86%), best:    6022/10000 (60.22%), error:     12893.17578, best error:     11466.52637 on epoch:     5, epoch:    8/50, mu: 0.6545274854, learning_rate: 0.0008227438, weight_decay: 0.0002991354

epoch time: 1762.3383789062s, input_fired_time: 16.8594474792, output_fired_time: 6.0972776413, propagate_forward_time: 687.6537475586, propagate_backward_time: 1038.8774414062, weight_update_time: 0.4893456101, other_time: 12.8503417969
epoch time: 145.9424591064s, input_fired_time: 0.9068982005, output_fired_time: 0.0000000000, propagate_forward_time: 142.9221649170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1134033203
validation[          , genome  6297] predictions:    5676/  10000 (56.76%), best:    6022/10000 (60.22%), error:     12786.76758, best error:     11466.52637 on epoch:     5, epoch:    9/50, mu: 0.6704614162, learning_rate: 0.0007583758, weight_decay: 0.0002788526

epoch time: 1723.4008789062s, input_fired_time: 17.1477851868, output_fired_time: 5.9649882317, propagate_forward_time: 682.7637329102, propagate_backward_time: 1004.5695190430, weight_update_time: 0.5094146729, other_time: 12.9548339844
epoch time: 137.6990356445s, input_fired_time: 0.8306246400, output_fired_time: 0.0000000000, propagate_forward_time: 134.6765594482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1918487549
validation[          , genome  6297] predictions:    5348/  10000 (53.48%), best:    6022/10000 (60.22%), error:     14512.83105, best error:     11466.52637 on epoch:     5, epoch:   10/50, mu: 0.6856385469, learning_rate: 0.0006990437, weight_decay: 0.0002599452

epoch time: 1771.5034179688s, input_fired_time: 17.0878906250, output_fired_time: 6.0874600410, propagate_forward_time: 697.4013671875, propagate_backward_time: 1037.8859863281, weight_update_time: 0.4982090592, other_time: 13.0407714844
epoch time: 156.5061950684s, input_fired_time: 0.9255231023, output_fired_time: 0.0000000000, propagate_forward_time: 153.4165649414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1641082764
validation[          , genome  6297] predictions:    5651/  10000 (56.51%), best:    6022/10000 (60.22%), error:     12411.40918, best error:     11466.52637 on epoch:     5, epoch:   11/50, mu: 0.7000948191, learning_rate: 0.0006443535, weight_decay: 0.0002423197

epoch time: 1804.8164062500s, input_fired_time: 18.3101215363, output_fired_time: 6.0807123184, propagate_forward_time: 735.3521118164, propagate_backward_time: 1032.3898925781, weight_update_time: 0.4970608354, other_time: 12.6835937500
epoch time: 132.5193481445s, input_fired_time: 0.9087899327, output_fired_time: 0.0000000000, propagate_forward_time: 129.6172943115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9932708740
validation[          , genome  6297] predictions:    6157/  10000 (61.57%), best:    6157/10000 (61.57%), error:     11067.56445, best error:     11067.56445 on epoch:    12, epoch:   12/50, mu: 0.7138644457, learning_rate: 0.0005939419, weight_decay: 0.0002258894

epoch time: 1793.0919189453s, input_fired_time: 17.9296226501, output_fired_time: 6.0732693672, propagate_forward_time: 726.5217285156, propagate_backward_time: 1029.7100830078, weight_update_time: 0.5139635801, other_time: 12.8572998047
epoch time: 174.9281311035s, input_fired_time: 0.9979572296, output_fired_time: 0.0000000000, propagate_forward_time: 171.6850280762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2451477051
validation[          , genome  6297] predictions:    6144/  10000 (61.44%), best:    6157/10000 (61.57%), error:     11078.33887, best error:     11067.56445 on epoch:    12, epoch:   13/50, mu: 0.7269800305, learning_rate: 0.0005474745, weight_decay: 0.0002105730

epoch time: 1766.3839111328s, input_fired_time: 17.2533187866, output_fired_time: 6.0927333832, propagate_forward_time: 696.0150756836, propagate_backward_time: 1033.8872070312, weight_update_time: 0.5114135742, other_time: 13.1354980469
epoch time: 140.7728118896s, input_fired_time: 0.8579891324, output_fired_time: 0.0000000000, propagate_forward_time: 137.7864074707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1284179688
validation[          , genome  6297] predictions:    6150/  10000 (61.50%), best:    6157/10000 (61.57%), error:     11257.39941, best error:     11067.56445 on epoch:    12, epoch:   14/50, mu: 0.7394726872, learning_rate: 0.0005046424, weight_decay: 0.0001962952

epoch time: 1743.2158203125s, input_fired_time: 17.3651180267, output_fired_time: 5.9764304161, propagate_forward_time: 697.1042480469, propagate_backward_time: 1009.8410034180, weight_update_time: 0.5162298679, other_time: 12.9290161133
epoch time: 155.5331115723s, input_fired_time: 0.9475477934, output_fired_time: 0.0000000000, propagate_forward_time: 152.2738189697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3117523193
validation[          , genome  6297] predictions:    5979/  10000 (59.79%), best:    6157/10000 (61.57%), error:     11534.89258, best error:     11067.56445 on epoch:    12, epoch:   15/50, mu: 0.7513719797, learning_rate: 0.0004651614, weight_decay: 0.0001829855

epoch time: 1845.8479003906s, input_fired_time: 18.1981372833, output_fired_time: 6.1743330956, propagate_forward_time: 753.3604125977, propagate_backward_time: 1054.5523681641, weight_update_time: 0.5350375772, other_time: 13.5626220703
epoch time: 143.9887084961s, input_fired_time: 0.9276123643, output_fired_time: 0.0000000000, propagate_forward_time: 140.9807586670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0803375244
validation[          , genome  6297] predictions:    6129/  10000 (61.29%), best:    6157/10000 (61.57%), error:     11082.23145, best error:     11067.56445 on epoch:    12, epoch:   16/50, mu: 0.7627061009, learning_rate: 0.0004287691, weight_decay: 0.0001705783

epoch time: 1778.9620361328s, input_fired_time: 17.7711238861, output_fired_time: 6.0376577377, propagate_forward_time: 719.3645019531, propagate_backward_time: 1022.6987304688, weight_update_time: 0.5243481398, other_time: 13.0900878906
epoch time: 144.8054656982s, input_fired_time: 0.9262918830, output_fired_time: 0.0000000000, propagate_forward_time: 141.6915283203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1876525879
validation[          , genome  6297] predictions:    6183/  10000 (61.83%), best:    6183/10000 (61.83%), error:     10872.24512, best error:     10872.24512 on epoch:    17, epoch:   17/50, mu: 0.7735018730, learning_rate: 0.0003952241, weight_decay: 0.0001590123

epoch time: 1767.9248046875s, input_fired_time: 17.2996253967, output_fired_time: 6.0812706947, propagate_forward_time: 701.1370849609, propagate_backward_time: 1030.1239013672, weight_update_time: 0.5151463151, other_time: 13.2828369141
epoch time: 146.6492919922s, input_fired_time: 0.9193116426, output_fired_time: 0.0000000000, propagate_forward_time: 143.5905456543, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1394348145
validation[          , genome  6297] predictions:    6224/  10000 (62.24%), best:    6224/10000 (62.24%), error:     10847.26562, best error:     10847.26562 on epoch:    18, epoch:   18/50, mu: 0.7837848663, learning_rate: 0.0003643034, weight_decay: 0.0001482306

epoch time: 1707.6519775391s, input_fired_time: 16.8557701111, output_fired_time: 6.1131949425, propagate_forward_time: 664.2574462891, propagate_backward_time: 1007.4277954102, weight_update_time: 0.5258769989, other_time: 12.9978637695
epoch time: 139.0626525879s, input_fired_time: 0.8646577001, output_fired_time: 0.0000000000, propagate_forward_time: 135.9725646973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2254333496
validation[          , genome  6297] predictions:    6214/  10000 (62.14%), best:    6224/10000 (62.24%), error:     10926.48926, best error:     10847.26562 on epoch:    18, epoch:   19/50, mu: 0.7935794592, learning_rate: 0.0003358019, weight_decay: 0.0001381799

epoch time: 1764.7448730469s, input_fired_time: 17.7449264526, output_fired_time: 6.1686382294, propagate_forward_time: 704.6356811523, propagate_backward_time: 1022.9589843750, weight_update_time: 0.5348087549, other_time: 13.2368164062
epoch time: 150.2075347900s, input_fired_time: 0.9320683479, output_fired_time: 0.0000000000, propagate_forward_time: 147.1535949707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1218719482
validation[          , genome  6297] predictions:    6064/  10000 (60.64%), best:    6224/10000 (62.24%), error:     11542.83887, best error:     10847.26562 on epoch:    18, epoch:   20/50, mu: 0.8029088378, learning_rate: 0.0003095302, weight_decay: 0.0001288106

epoch time: 1783.6973876953s, input_fired_time: 17.7873916626, output_fired_time: 6.0620017052, propagate_forward_time: 717.7295532227, propagate_backward_time: 1028.6790771484, weight_update_time: 0.5340332985, other_time: 13.4393310547
epoch time: 138.6687164307s, input_fired_time: 0.9040565491, output_fired_time: 0.0000000000, propagate_forward_time: 135.7088165283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0558471680
validation[          , genome  6297] predictions:    6282/  10000 (62.82%), best:    6282/10000 (62.82%), error:     10679.38770, best error:     10679.38770 on epoch:    21, epoch:   21/50, mu: 0.8117951155, learning_rate: 0.0002853138, weight_decay: 0.0001200767

epoch time: 1724.2728271484s, input_fired_time: 17.4447879791, output_fired_time: 5.9758739471, propagate_forward_time: 688.3091430664, propagate_backward_time: 999.5809326172, weight_update_time: 0.5231909752, other_time: 12.9620971680
epoch time: 146.3918762207s, input_fired_time: 0.9339835048, output_fired_time: 0.0000000000, propagate_forward_time: 143.2096252441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2482604980
validation[          , genome  6297] predictions:    6056/  10000 (60.56%), best:    6282/10000 (62.82%), error:     11184.88965, best error:     10679.38770 on epoch:    21, epoch:   22/50, mu: 0.8202593327, learning_rate: 0.0002629921, weight_decay: 0.0001119350

epoch time: 1764.8543701172s, input_fired_time: 17.8678512573, output_fired_time: 6.0663599968, propagate_forward_time: 710.6364746094, propagate_backward_time: 1016.8892211914, weight_update_time: 0.5386466980, other_time: 13.3944702148
epoch time: 144.4695281982s, input_fired_time: 0.9354577661, output_fired_time: 0.0000000000, propagate_forward_time: 141.4813690186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0527038574
validation[          , genome  6297] predictions:    6252/  10000 (62.52%), best:    6282/10000 (62.82%), error:     10768.55762, best error:     10679.38770 on epoch:    21, epoch:   23/50, mu: 0.8283215165, learning_rate: 0.0002424167, weight_decay: 0.0001043453

epoch time: 1766.3345947266s, input_fired_time: 17.6324768066, output_fired_time: 5.9999742508, propagate_forward_time: 706.8824462891, propagate_backward_time: 1022.6173095703, weight_update_time: 0.5118577480, other_time: 13.2023925781
epoch time: 147.1952972412s, input_fired_time: 0.9222654104, output_fired_time: 0.0000000000, propagate_forward_time: 144.1497955322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1232299805
validation[          , genome  6297] predictions:    6374/  10000 (63.74%), best:    6374/10000 (63.74%), error:     10492.72949, best error:     10492.72949 on epoch:    24, epoch:   24/50, mu: 0.8360007405, learning_rate: 0.0002234511, weight_decay: 0.0000972702

epoch time: 1745.2808837891s, input_fired_time: 17.5413417816, output_fired_time: 6.0137877464, propagate_forward_time: 695.9061279297, propagate_backward_time: 1012.6903686523, weight_update_time: 0.5175586343, other_time: 13.1292114258
epoch time: 145.1535644531s, input_fired_time: 0.9355064631, output_fired_time: 0.0000000000, propagate_forward_time: 141.9703674316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2476959229
validation[          , genome  6297] predictions:    6334/  10000 (63.34%), best:    6374/10000 (63.74%), error:     10524.44531, best error:     10492.72949 on epoch:    24, epoch:   25/50, mu: 0.8433152437, learning_rate: 0.0002059692, weight_decay: 0.0000906748

epoch time: 1749.6643066406s, input_fired_time: 17.4420547485, output_fired_time: 6.0556740761, propagate_forward_time: 691.7636718750, propagate_backward_time: 1021.1690063477, weight_update_time: 0.5162940621, other_time: 13.2339477539
epoch time: 140.7295837402s, input_fired_time: 0.8632175326, output_fired_time: 0.0000000000, propagate_forward_time: 137.6273040771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2390594482
validation[          , genome  6297] predictions:    6405/  10000 (64.05%), best:    6405/10000 (64.05%), error:     10410.69727, best error:     10410.69727 on epoch:    26, epoch:   26/50, mu: 0.8502823114, learning_rate: 0.0001898551, weight_decay: 0.0000845267

epoch time: 1750.1353759766s, input_fired_time: 17.3030529022, output_fired_time: 6.0103201866, propagate_forward_time: 695.4668579102, propagate_backward_time: 1018.1072998047, weight_update_time: 0.5221408010, other_time: 13.2476806641
epoch time: 157.6899261475s, input_fired_time: 0.9687242508, output_fired_time: 0.0000000000, propagate_forward_time: 154.3744506836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3467559814
validation[          , genome  6297] predictions:    6324/  10000 (63.24%), best:    6405/10000 (64.05%), error:     10552.11523, best error:     10410.69727 on epoch:    26, epoch:   27/50, mu: 0.8569184542, learning_rate: 0.0001750016, weight_decay: 0.0000787954

epoch time: 1820.8265380859s, input_fired_time: 18.4590587616, output_fired_time: 6.3058977127, propagate_forward_time: 734.4922485352, propagate_backward_time: 1047.5782470703, weight_update_time: 0.5588908792, other_time: 13.9910888672
epoch time: 149.4121093750s, input_fired_time: 0.9682717323, output_fired_time: 0.0000000000, propagate_forward_time: 146.1424102783, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3014221191
validation[          , genome  6297] predictions:    6396/  10000 (63.96%), best:    6396/10000 (63.96%), error:     10410.03613, best error:     10410.03613 on epoch:    28, epoch:   28/50, mu: 0.8632394075, learning_rate: 0.0001613102, weight_decay: 0.0000734527

epoch time: 1827.3737792969s, input_fired_time: 18.5766124725, output_fired_time: 6.2585587502, propagate_forward_time: 737.6348266602, propagate_backward_time: 1051.1875000000, weight_update_time: 0.5563461185, other_time: 13.7163085938
epoch time: 147.2529754639s, input_fired_time: 0.9613581896, output_fired_time: 0.0000000000, propagate_forward_time: 144.0292816162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2623291016
validation[          , genome  6297] predictions:    6395/  10000 (63.95%), best:    6395/10000 (63.95%), error:     10338.04492, best error:     10338.04492 on epoch:    29, epoch:   29/50, mu: 0.8692601323, learning_rate: 0.0001486900, weight_decay: 0.0000684723

epoch time: 1785.1661376953s, input_fired_time: 18.2081584930, output_fired_time: 6.1636929512, propagate_forward_time: 715.0256347656, propagate_backward_time: 1032.6141357422, weight_update_time: 0.5282039642, other_time: 13.1545410156
epoch time: 157.8128051758s, input_fired_time: 1.0350635052, output_fired_time: 0.0000000000, propagate_forward_time: 154.5321197510, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2456207275
validation[          , genome  6297] predictions:    6399/  10000 (63.99%), best:    6395/10000 (63.95%), error:     10461.20020, best error:     10338.04492 on epoch:    29, epoch:   30/50, mu: 0.8749949336, learning_rate: 0.0001370571, weight_decay: 0.0000638295

epoch time: 1860.0904541016s, input_fired_time: 19.0456523895, output_fired_time: 6.3667092323, propagate_forward_time: 747.6420288086, propagate_backward_time: 1073.1037597656, weight_update_time: 0.5876070261, other_time: 13.9323730469
epoch time: 149.2121429443s, input_fired_time: 0.9631856680, output_fired_time: 0.0000000000, propagate_forward_time: 146.0830078125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1659545898
validation[          , genome  6297] predictions:    6331/  10000 (63.31%), best:    6395/10000 (63.95%), error:     10681.23730, best error:     10338.04492 on epoch:    29, epoch:   31/50, mu: 0.8804573417, learning_rate: 0.0001263344, weight_decay: 0.0000595016

epoch time: 1786.9881591797s, input_fired_time: 18.1250019073, output_fired_time: 6.1864266396, propagate_forward_time: 715.0598144531, propagate_backward_time: 1034.1877441406, weight_update_time: 0.5356619358, other_time: 13.4291992188
epoch time: 148.9756164551s, input_fired_time: 0.9939405322, output_fired_time: 0.0000000000, propagate_forward_time: 145.7578125000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2238616943
validation[          , genome  6297] predictions:    6416/  10000 (64.16%), best:    6416/10000 (64.16%), error:     10322.70215, best error:     10322.70215 on epoch:    32, epoch:   32/50, mu: 0.8856602907, learning_rate: 0.0001164505, weight_decay: 0.0000554671

epoch time: 1765.4707031250s, input_fired_time: 17.9104976654, output_fired_time: 6.0724759102, propagate_forward_time: 706.9526367188, propagate_backward_time: 1021.3494873047, weight_update_time: 0.5240502954, other_time: 13.1855468750
epoch time: 146.5252380371s, input_fired_time: 0.9425623417, output_fired_time: 0.0000000000, propagate_forward_time: 143.4982604980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0844116211
validation[          , genome  6297] predictions:    6161/  10000 (61.61%), best:    6416/10000 (64.16%), error:     11086.25488, best error:     10322.70215 on epoch:    32, epoch:   33/50, mu: 0.8906161189, learning_rate: 0.0001073399, weight_decay: 0.0000517062

epoch time: 1754.8530273438s, input_fired_time: 17.7486438751, output_fired_time: 5.9772429466, propagate_forward_time: 705.1481933594, propagate_backward_time: 1013.1624145508, weight_update_time: 0.5239567757, other_time: 12.8164672852
epoch time: 145.0190277100s, input_fired_time: 0.9120779634, output_fired_time: 0.0000000000, propagate_forward_time: 141.9791259766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1278228760
validation[          , genome  6297] predictions:    6480/  10000 (64.80%), best:    6480/10000 (64.80%), error:     10199.56250, best error:     10199.56250 on epoch:    34, epoch:   34/50, mu: 0.8953365684, learning_rate: 0.0000989421, weight_decay: 0.0000482003

epoch time: 1745.4990234375s, input_fired_time: 17.7578639984, output_fired_time: 6.0786948204, propagate_forward_time: 696.8561401367, propagate_backward_time: 1012.1166381836, weight_update_time: 0.5235128999, other_time: 12.6897583008
epoch time: 150.5018768311s, input_fired_time: 0.9828560948, output_fired_time: 0.0000000000, propagate_forward_time: 147.3035736084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2154541016
validation[          , genome  6297] predictions:    6370/  10000 (63.70%), best:    6480/10000 (64.80%), error:     10419.92871, best error:     10199.56250 on epoch:    34, epoch:   35/50, mu: 0.8998327851, learning_rate: 0.0000912013, weight_decay: 0.0000449321

epoch time: 1775.3156738281s, input_fired_time: 17.7884101868, output_fired_time: 6.1284394264, propagate_forward_time: 707.6358642578, propagate_backward_time: 1030.5472412109, weight_update_time: 0.5226776004, other_time: 13.2156982422
epoch time: 154.2895050049s, input_fired_time: 0.9697073102, output_fired_time: 0.0000000000, propagate_forward_time: 151.0965576172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2232360840
validation[          , genome  6297] predictions:    6519/  10000 (65.19%), best:    6519/10000 (65.19%), error:     10070.53809, best error:     10070.53809 on epoch:    36, epoch:   36/50, mu: 0.9041154385, learning_rate: 0.0000840661, weight_decay: 0.0000418855

epoch time: 1786.9482421875s, input_fired_time: 17.8205184937, output_fired_time: 6.0797662735, propagate_forward_time: 716.3279418945, propagate_backward_time: 1033.3945312500, weight_update_time: 0.5281586647, other_time: 13.3254394531
epoch time: 149.1669464111s, input_fired_time: 0.9612985253, output_fired_time: 0.0000000000, propagate_forward_time: 146.0922851562, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1133575439
validation[          , genome  6297] predictions:    6505/  10000 (65.05%), best:    6519/10000 (65.19%), error:     10164.02637, best error:     10070.53809 on epoch:    36, epoch:   37/50, mu: 0.9081947207, learning_rate: 0.0000774891, weight_decay: 0.0000390455

epoch time: 1775.4228515625s, input_fired_time: 17.4573802948, output_fired_time: 6.0754547119, propagate_forward_time: 702.2280273438, propagate_backward_time: 1037.3618164062, weight_update_time: 0.4880692363, other_time: 12.3001708984
epoch time: 142.8529968262s, input_fired_time: 0.8919586539, output_fired_time: 0.0000000000, propagate_forward_time: 140.0311126709, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9299316406
validation[          , genome  6297] predictions:    6494/  10000 (64.94%), best:    6519/10000 (65.19%), error:     10093.46289, best error:     10070.53809 on epoch:    36, epoch:   38/50, mu: 0.9120802283, learning_rate: 0.0000714267, weight_decay: 0.0000363980

epoch time: 1763.3447265625s, input_fired_time: 17.3176517487, output_fired_time: 6.0567822456, propagate_forward_time: 695.9228515625, propagate_backward_time: 1031.8491210938, weight_update_time: 0.4773251712, other_time: 12.1983642578
epoch time: 143.7761230469s, input_fired_time: 0.8757757545, output_fired_time: 0.0000000000, propagate_forward_time: 140.9294433594, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9709014893
validation[          , genome  6297] predictions:    6459/  10000 (64.59%), best:    6519/10000 (65.19%), error:     10276.54785, best error:     10070.53809 on epoch:    36, epoch:   39/50, mu: 0.9157811999, learning_rate: 0.0000658386, weight_decay: 0.0000339301

epoch time: 1762.2275390625s, input_fired_time: 17.1456127167, output_fired_time: 6.0348868370, propagate_forward_time: 688.7731323242, propagate_backward_time: 1038.5314941406, weight_update_time: 0.4596902728, other_time: 11.7424316406
epoch time: 140.8429107666s, input_fired_time: 0.8548825979, output_fired_time: 0.0000000000, propagate_forward_time: 138.1639404297, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8240814209
validation[          , genome  6297] predictions:    6438/  10000 (64.38%), best:    6519/10000 (65.19%), error:     10255.29004, best error:     10070.53809 on epoch:    36, epoch:   40/50, mu: 0.9193063974, learning_rate: 0.0000606876, weight_decay: 0.0000316294

epoch time: 1766.5354003906s, input_fired_time: 17.1113510132, output_fired_time: 6.0383315086, propagate_forward_time: 692.4985351562, propagate_backward_time: 1039.2845458984, weight_update_time: 0.4524133801, other_time: 11.6026611328
epoch time: 142.4974060059s, input_fired_time: 0.9138802886, output_fired_time: 0.0000000000, propagate_forward_time: 139.6166534424, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9668731689
validation[          , genome  6297] predictions:    6557/  10000 (65.57%), best:    6557/10000 (65.57%), error:     10024.07910, best error:     10024.07910 on epoch:    41, epoch:   41/50, mu: 0.9226641059, learning_rate: 0.0000559397, weight_decay: 0.0000294848

epoch time: 1747.5021972656s, input_fired_time: 17.1415252686, output_fired_time: 6.0623478889, propagate_forward_time: 681.5482177734, propagate_backward_time: 1031.4327392578, weight_update_time: 0.4572031200, other_time: 11.3173828125
epoch time: 137.6197967529s, input_fired_time: 0.8943957686, output_fired_time: 0.0000000000, propagate_forward_time: 134.8850708008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8403320312
validation[          , genome  6297] predictions:    6516/  10000 (65.16%), best:    6557/10000 (65.57%), error:     10127.19238, best error:     10024.07910 on epoch:    41, epoch:   42/50, mu: 0.9258623719, learning_rate: 0.0000515632, weight_decay: 0.0000274856

epoch time: 1752.9378662109s, input_fired_time: 17.0733356476, output_fired_time: 5.9670605659, propagate_forward_time: 688.7175903320, propagate_backward_time: 1029.7320556641, weight_update_time: 0.4513259232, other_time: 11.4478759766
epoch time: 136.4852447510s, input_fired_time: 0.9190420508, output_fired_time: 0.0000000000, propagate_forward_time: 133.6847534180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8814544678
validation[          , genome  6297] predictions:    6523/  10000 (65.23%), best:    6557/10000 (65.57%), error:     10086.37402, best error:     10024.07910 on epoch:    41, epoch:   43/50, mu: 0.9289087057, learning_rate: 0.0000475291, weight_decay: 0.0000256220

epoch time: 1763.3684082031s, input_fired_time: 17.1439876556, output_fired_time: 6.0951957703, propagate_forward_time: 688.0270385742, propagate_backward_time: 1040.0830078125, weight_update_time: 0.4758666754, other_time: 12.0190429688
epoch time: 139.4645843506s, input_fired_time: 0.8803953528, output_fired_time: 0.0000000000, propagate_forward_time: 136.5810852051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0030975342
validation[          , genome  6297] predictions:    6492/  10000 (64.92%), best:    6557/10000 (65.57%), error:     10105.30078, best error:     10024.07910 on epoch:    41, epoch:   44/50, mu: 0.9318103790, learning_rate: 0.0000438107, weight_decay: 0.0000238847

epoch time: 1787.3060302734s, input_fired_time: 17.3863601685, output_fired_time: 6.1039004326, propagate_forward_time: 703.6767578125, propagate_backward_time: 1047.7054443359, weight_update_time: 0.4808171690, other_time: 12.4335937500
epoch time: 148.9459228516s, input_fired_time: 0.9536733627, output_fired_time: 0.0000000000, propagate_forward_time: 145.8980865479, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0941619873
validation[          , genome  6297] predictions:    6540/  10000 (65.40%), best:    6540/10000 (65.40%), error:      9963.61035, best error:      9963.61035 on epoch:    45, epoch:   45/50, mu: 0.9345741868, learning_rate: 0.0000403831, weight_decay: 0.0000222652

epoch time: 1795.9522705078s, input_fired_time: 17.7618618011, output_fired_time: 6.1167154312, propagate_forward_time: 714.8795166016, propagate_backward_time: 1044.6728515625, weight_update_time: 0.4897968769, other_time: 12.5213623047
epoch time: 161.6386871338s, input_fired_time: 0.9927000403, output_fired_time: 0.0000000000, propagate_forward_time: 158.5581359863, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0878448486
validation[          , genome  6297] predictions:    6544/  10000 (65.44%), best:    6540/10000 (65.40%), error:     10010.58203, best error:      9963.61035 on epoch:    45, epoch:   46/50, mu: 0.9372067451, learning_rate: 0.0000372237, weight_decay: 0.0000207555

epoch time: 1801.5728759766s, input_fired_time: 17.5043087006, output_fired_time: 6.1474661827, propagate_forward_time: 709.4309082031, propagate_backward_time: 1055.8596191406, weight_update_time: 0.4855593741, other_time: 12.6306152344
epoch time: 143.9558868408s, input_fired_time: 0.9099932909, output_fired_time: 0.0000000000, propagate_forward_time: 140.9674530029, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0784454346
validation[          , genome  6297] predictions:    6520/  10000 (65.20%), best:    6540/10000 (65.40%), error:      9966.27344, best error:      9963.61035 on epoch:    45, epoch:   47/50, mu: 0.9397142529, learning_rate: 0.0000343115, weight_decay: 0.0000193482

epoch time: 1798.6125488281s, input_fired_time: 17.5753421783, output_fired_time: 6.1939473152, propagate_forward_time: 707.8567504883, propagate_backward_time: 1054.4168701172, weight_update_time: 0.4805144668, other_time: 12.5697021484
epoch time: 147.9517822266s, input_fired_time: 0.9601232409, output_fired_time: 0.0000000000, propagate_forward_time: 144.8790588379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1125946045
validation[          , genome  6297] predictions:    6573/  10000 (65.73%), best:    6573/10000 (65.73%), error:      9882.30176, best error:      9882.30176 on epoch:    48, epoch:   48/50, mu: 0.9421026707, learning_rate: 0.0000316271, weight_decay: 0.0000180363

epoch time: 1767.2048339844s, input_fired_time: 17.5508155823, output_fired_time: 6.0708093643, propagate_forward_time: 697.9436035156, propagate_backward_time: 1033.2459716797, weight_update_time: 0.4877147675, other_time: 12.3936767578
epoch time: 147.4775543213s, input_fired_time: 0.9733043909, output_fired_time: 0.0000000000, propagate_forward_time: 144.4483337402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0559234619
validation[          , genome  6297] predictions:    6585/  10000 (65.85%), best:    6585/10000 (65.85%), error:      9867.68164, best error:      9867.68164 on epoch:    49, epoch:   49/50, mu: 0.9443776608, learning_rate: 0.0000291527, weight_decay: 0.0000168134

epoch time: 1792.7813720703s, input_fired_time: 17.8521308899, output_fired_time: 6.1367683411, propagate_forward_time: 710.4464721680, propagate_backward_time: 1046.0722656250, weight_update_time: 0.4902470112, other_time: 12.2736816406
epoch time: 140.7133636475s, input_fired_time: 0.9295951128, output_fired_time: 0.0000000000, propagate_forward_time: 137.7754364014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0083312988
validation[          , genome  6297] predictions:    6544/  10000 (65.44%), best:    6585/10000 (65.85%), error:      9974.95410, best error:      9867.68164 on epoch:    49, epoch:   50/50, mu: 0.9465445876, learning_rate: 0.0000268719, weight_decay: 0.0000156733

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 717.9846801758s, input_fired_time: 4.5764484406, output_fired_time: 0.0000000000, propagate_forward_time: 702.9881591797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.4200439453
best training[          , genome  6297] predictions:   36661/  50000 (73.32%), best:    6585/10000 (65.85%), error:     40047.50391, best error:      9867.68164 on epoch:    49, epoch:   51/50, mu: 0.9486085773, learning_rate: 0.0000247696, weight_decay: 0.0000146106
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 148.0754394531s, input_fired_time: 0.9669027328, output_fired_time: 0.0000000000, propagate_forward_time: 144.9514617920, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1570739746
   testing[          , genome  6297] predictions:    6585/  10000 (65.85%), best:    6585/10000 (65.85%), error:      9867.68164, best error:      9867.68164 on epoch:    49, epoch:   51/50, mu: 0.9486085773, learning_rate: 0.0000247696, weight_decay: 0.0000146106
backpropagation finished successfully!
05:26:22 (4527): called boinc_finish(0)

</stderr_txt>
]]>