Nameexact_genome_1542069603_180_5779_4
Workunit4086117
Created1 Mar 2019, 17:55:35 UTC
Sent1 Mar 2019, 18:06:00 UTC
Report deadline24 Apr 2019, 0:05:26 UTC
Received2 Mar 2019, 15:11:09 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84860
Run time20 hours 54 min 24 sec
CPU time20 hours 5 min 10 sec
Validate stateValid
Credit13,312.40
Device peak FLOPS4.84 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size557.11 MB
Peak swap size559.85 MB
Peak disk usage7.37 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_180_5779.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_180_5779_4_r1741490751_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542069603_180_5779.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.492263
read mu: 0.492263
read mu_delta: 0.936286
read initial_learning_rate: 0.000757427
read learning_rate: 0.000757427
read learning_rate_delta: 0.915493
read initial_weight_decay: 0.000513472
read weight_decay: 0.000513472
read weight_decay_delta: 0.942871
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.107724
read input_dropout_probability: 0.00140216
read hidden_dropout_probability: 0.0901352
read velocity_reset: 1688
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5779
read normal distribution: '1 0.0636798143386841 1.14619112014771'
generator_str: '517901718'
read generator: 517901718
read generated_by_map:
1 change_size_x 1
reading 85 nodes.
reading 321 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 622953135
post shuffle 1: 1986159379
epoch time: 1218.73803710938s, input_fired_time: 18.9085102081299, output_fired_time: 6.39668989181519, propagate_forward_time: 426.649261474609, propagate_backward_time: 756.453247070312, weight_update_time: 0.266530424356461, other_time: 10.3302001953125
epoch time: 78.724723815918s, input_fired_time: 0.902069687843323, output_fired_time: 0, propagate_forward_time: 76.3263854980469, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.49626922607422
validation[          , genome  5779] predictions:    5457/  10000 (54.57%), best:    5457/10000 (54.57%), error:     13130.36621, best error:     13130.36621 on epoch:     0, epoch:    0/50, mu: 0.4922626913, learning_rate: 0.0007574271, weight_decay: 0.0005134715

epoch time: 1177.5003662109s, input_fired_time: 18.8357429504, output_fired_time: 6.1772336960, propagate_forward_time: 413.1323547363, propagate_backward_time: 729.4146728516, weight_update_time: 0.2602585554, other_time: 9.9404296875
epoch time: 100.3029098511s, input_fired_time: 0.9701554775, output_fired_time: 0.0000000000, propagate_forward_time: 97.3056488037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0271072388
validation[          , genome  5779] predictions:    5703/  10000 (57.03%), best:    5703/10000 (57.03%), error:     12365.08398, best error:     12365.08398 on epoch:     1, epoch:    1/50, mu: 0.5239753723, learning_rate: 0.0006934189, weight_decay: 0.0004841376

epoch time: 1352.2968750000s, input_fired_time: 19.3728656769, output_fired_time: 7.3490772247, propagate_forward_time: 477.5179138184, propagate_backward_time: 836.2670288086, weight_update_time: 0.2948985994, other_time: 11.7898559570
epoch time: 97.4184494019s, input_fired_time: 0.9537692070, output_fired_time: 0.0000000000, propagate_forward_time: 94.5325622559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9321212769
validation[          , genome  5779] predictions:    5748/  10000 (57.48%), best:    5703/10000 (57.03%), error:     12527.83301, best error:     12365.08398 on epoch:     1, epoch:    2/50, mu: 0.5536674857, learning_rate: 0.0006348199, weight_decay: 0.0004564795

epoch time: 1362.1557617188s, input_fired_time: 19.3817729950, output_fired_time: 7.3804769516, propagate_forward_time: 482.1786193848, propagate_backward_time: 841.3518676758, weight_update_time: 0.2979961634, other_time: 11.8630981445
epoch time: 100.7134246826s, input_fired_time: 0.9653996825, output_fired_time: 0.0000000000, propagate_forward_time: 97.7079849243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0400390625
validation[          , genome  5779] predictions:    5915/  10000 (59.15%), best:    5915/10000 (59.15%), error:     11978.61523, best error:     11978.61523 on epoch:     3, epoch:    3/50, mu: 0.5814678073, learning_rate: 0.0005811729, weight_decay: 0.0004304015

epoch time: 1325.5356445312s, input_fired_time: 19.2875766754, output_fired_time: 7.0810470581, propagate_forward_time: 468.8017272949, propagate_backward_time: 818.9634399414, weight_update_time: 0.2885267437, other_time: 11.4017944336
epoch time: 93.0037536621s, input_fired_time: 0.9378534555, output_fired_time: 0.0000000000, propagate_forward_time: 90.1604614258, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9054412842
validation[          , genome  5779] predictions:    5651/  10000 (56.51%), best:    5915/10000 (59.15%), error:     12475.91504, best error:     11978.61523 on epoch:     3, epoch:    4/50, mu: 0.6074969172, learning_rate: 0.0005320595, weight_decay: 0.0004058132

epoch time: 1189.2009277344s, input_fired_time: 18.8477096558, output_fired_time: 6.1714863777, propagate_forward_time: 416.0750732422, propagate_backward_time: 737.8552246094, weight_update_time: 0.2625399530, other_time: 10.2514648438
epoch time: 100.5255050659s, input_fired_time: 0.9613549113, output_fired_time: 0.0000000000, propagate_forward_time: 97.4992675781, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0648803711
validation[          , genome  5779] predictions:    5387/  10000 (53.87%), best:    5915/10000 (59.15%), error:     13238.96094, best error:     11978.61523 on epoch:     3, epoch:    5/50, mu: 0.6318675876, learning_rate: 0.0004870965, weight_decay: 0.0003826297

epoch time: 1221.8183593750s, input_fired_time: 18.9433593750, output_fired_time: 6.4074473381, propagate_forward_time: 428.6833496094, propagate_backward_time: 757.2695312500, weight_update_time: 0.2674885392, other_time: 10.5146484375
epoch time: 73.6989746094s, input_fired_time: 0.8734734654, output_fired_time: 0.0000000000, propagate_forward_time: 71.3952941895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4302062988
validation[          , genome  5779] predictions:    5868/  10000 (58.68%), best:    5915/10000 (59.15%), error:     12057.87891, best error:     11978.61523 on epoch:     3, epoch:    6/50, mu: 0.6546854973, learning_rate: 0.0004459332, weight_decay: 0.0003607706

epoch time: 1344.6741943359s, input_fired_time: 19.3960552216, output_fired_time: 7.2626943588, propagate_forward_time: 475.9101562500, propagate_backward_time: 830.3627319336, weight_update_time: 0.2942763567, other_time: 11.7426147461
epoch time: 94.3521804810s, input_fired_time: 0.9504963756, output_fired_time: 0.0000000000, propagate_forward_time: 91.4729156494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9287719727
validation[          , genome  5779] predictions:    5518/  10000 (55.18%), best:    5915/10000 (59.15%), error:     13109.56836, best error:     11978.61523 on epoch:     3, epoch:    7/50, mu: 0.6760495901, learning_rate: 0.0004082486, weight_decay: 0.0003401603

epoch time: 1321.7066650391s, input_fired_time: 19.4366588593, output_fired_time: 7.0471320152, propagate_forward_time: 468.0171203613, propagate_backward_time: 815.5911865234, weight_update_time: 0.3002431095, other_time: 11.6146240234
epoch time: 96.9186172485s, input_fired_time: 0.9690111279, output_fired_time: 0.0000000000, propagate_forward_time: 93.9214477539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0281600952
validation[          , genome  5779] predictions:    5783/  10000 (57.83%), best:    5915/10000 (59.15%), error:     12234.57422, best error:     11978.61523 on epoch:     3, epoch:    8/50, mu: 0.6960525513, learning_rate: 0.0003737485, weight_decay: 0.0003207274

epoch time: 1334.9578857422s, input_fired_time: 19.4052391052, output_fired_time: 7.1266989708, propagate_forward_time: 471.5144653320, propagate_backward_time: 825.1732177734, weight_update_time: 0.2923054099, other_time: 11.7382202148
epoch time: 92.9118804932s, input_fired_time: 0.9521067142, output_fired_time: 0.0000000000, propagate_forward_time: 90.0645599365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8952102661
validation[          , genome  5779] predictions:    5850/  10000 (58.50%), best:    5915/10000 (59.15%), error:     12073.48828, best error:     11978.61523 on epoch:     3, epoch:    9/50, mu: 0.7147810459, learning_rate: 0.0003421640, weight_decay: 0.0003024047

epoch time: 1302.5343017578s, input_fired_time: 19.3366146088, output_fired_time: 7.0122070312, propagate_forward_time: 460.0389099121, propagate_backward_time: 804.7464599609, weight_update_time: 0.2865470648, other_time: 11.4000244141
epoch time: 97.5877227783s, input_fired_time: 0.9522405267, output_fired_time: 0.0000000000, propagate_forward_time: 94.6776809692, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9578018188
validation[          , genome  5779] predictions:    5585/  10000 (55.85%), best:    5915/10000 (59.15%), error:     12494.65430, best error:     11978.61523 on epoch:     3, epoch:   10/50, mu: 0.7323162556, learning_rate: 0.0003132486, weight_decay: 0.0002851287

epoch time: 1331.6011962891s, input_fired_time: 19.4179210663, output_fired_time: 7.1217818260, propagate_forward_time: 471.6548767090, propagate_backward_time: 821.7842407227, weight_update_time: 0.2919822633, other_time: 11.6222534180
epoch time: 97.5554962158s, input_fired_time: 0.9891072512, output_fired_time: 0.0000000000, propagate_forward_time: 94.4985580444, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0678329468
validation[          , genome  5779] predictions:    5727/  10000 (57.27%), best:    5915/10000 (59.15%), error:     12236.91113, best error:     11978.61523 on epoch:     3, epoch:   11/50, mu: 0.7487342358, learning_rate: 0.0002867768, weight_decay: 0.0002688397

epoch time: 1575.8688964844s, input_fired_time: 23.9327564240, output_fired_time: 8.6065912247, propagate_forward_time: 569.7032470703, propagate_backward_time: 959.4262695312, weight_update_time: 0.3566919267, other_time: 14.2000732422
epoch time: 172.8897399902s, input_fired_time: 1.9075691700, output_fired_time: 0.0000000000, propagate_forward_time: 167.3477020264, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6344757080
validation[          , genome  5779] predictions:    5574/  10000 (55.74%), best:    5915/10000 (59.15%), error:     13010.09473, best error:     11978.61523 on epoch:     3, epoch:   12/50, mu: 0.7641061544, learning_rate: 0.0002625420, weight_decay: 0.0002534813

epoch time: 2045.0662841797s, input_fired_time: 32.3249893188, output_fired_time: 11.0929946899, propagate_forward_time: 743.4565429688, propagate_backward_time: 1239.1380615234, weight_update_time: 0.4717822671, other_time: 19.0537109375
epoch time: 100.6536483765s, input_fired_time: 0.9772797823, output_fired_time: 0.0000000000, propagate_forward_time: 97.6210403442, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0553283691
validation[          , genome  5779] predictions:    6020/  10000 (60.20%), best:    6020/10000 (60.20%), error:     11745.43750, best error:     11745.43750 on epoch:    13, epoch:   13/50, mu: 0.7784987092, learning_rate: 0.0002403553, weight_decay: 0.0002390003

epoch time: 1838.7695312500s, input_fired_time: 28.4317836761, output_fired_time: 9.7784948349, propagate_forward_time: 672.6658325195, propagate_backward_time: 1110.8957519531, weight_update_time: 0.4601509869, other_time: 16.9978027344
epoch time: 194.5716552734s, input_fired_time: 2.0495371819, output_fired_time: 0.0000000000, propagate_forward_time: 188.4605560303, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0615692139
validation[          , genome  5779] predictions:    5717/  10000 (57.17%), best:    6020/10000 (60.20%), error:     12303.04590, best error:     11745.43750 on epoch:    13, epoch:   14/50, mu: 0.7919742465, learning_rate: 0.0002200435, weight_decay: 0.0002253465

epoch time: 2210.4748535156s, input_fired_time: 33.7694664001, output_fired_time: 12.2170381546, propagate_forward_time: 820.6834716797, propagate_backward_time: 1323.4975585938, weight_update_time: 0.4991496503, other_time: 20.3072509766
epoch time: 92.5276565552s, input_fired_time: 0.9535056353, output_fired_time: 0.0000000000, propagate_forward_time: 89.6816864014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8924636841
validation[          , genome  5779] predictions:    5842/  10000 (58.42%), best:    6020/10000 (60.20%), error:     11966.01758, best error:     11745.43750 on epoch:    13, epoch:   15/50, mu: 0.8045911789, learning_rate: 0.0002014482, weight_decay: 0.0002124728

epoch time: 1355.4041748047s, input_fired_time: 19.6265888214, output_fired_time: 7.2049412727, propagate_forward_time: 487.4316406250, propagate_backward_time: 829.7398681641, weight_update_time: 0.2947782874, other_time: 11.4011230469
epoch time: 98.5437469482s, input_fired_time: 0.9751642346, output_fired_time: 0.0000000000, propagate_forward_time: 95.6941986084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8743820190
validation[          , genome  5779] predictions:    5728/  10000 (57.28%), best:    6020/10000 (60.20%), error:     12260.60352, best error:     11745.43750 on epoch:    13, epoch:   16/50, mu: 0.8164042830, learning_rate: 0.0001844243, weight_decay: 0.0002003345

epoch time: 1346.4421386719s, input_fired_time: 19.6980094910, output_fired_time: 7.2402353287, propagate_forward_time: 481.0125732422, propagate_backward_time: 826.8945312500, weight_update_time: 0.2944521606, other_time: 11.5968017578
epoch time: 98.5767517090s, input_fired_time: 0.9755600691, output_fired_time: 0.0000000000, propagate_forward_time: 95.5850677490, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0161209106
validation[          , genome  5779] predictions:    5996/  10000 (59.96%), best:    5996/10000 (59.96%), error:     11533.00098, best error:     11533.00098 on epoch:    17, epoch:   17/50, mu: 0.8274646997, learning_rate: 0.0001688391, weight_decay: 0.0001888897

epoch time: 1343.6395263672s, input_fired_time: 19.6824779510, output_fired_time: 7.1545338631, propagate_forward_time: 479.0741271973, propagate_backward_time: 826.1672973633, weight_update_time: 0.2933542728, other_time: 11.5610961914
epoch time: 100.2317352295s, input_fired_time: 1.0035047531, output_fired_time: 0.0000000000, propagate_forward_time: 97.2683792114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9598541260
validation[          , genome  5779] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11341.45117, best error:     11341.45117 on epoch:    18, epoch:   18/50, mu: 0.8378204107, learning_rate: 0.0001545710, weight_decay: 0.0001780987

epoch time: 1356.9124755859s, input_fired_time: 19.8164558411, output_fired_time: 7.2824378014, propagate_forward_time: 484.9207458496, propagate_backward_time: 833.1874389648, weight_update_time: 0.2960501909, other_time: 11.7053833008
epoch time: 99.1277770996s, input_fired_time: 0.9959275126, output_fired_time: 0.0000000000, propagate_forward_time: 96.1047668457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0270843506
validation[          , genome  5779] predictions:    6054/  10000 (60.54%), best:    6049/10000 (60.49%), error:     11441.40820, best error:     11341.45117 on epoch:    18, epoch:   19/50, mu: 0.8475163579, learning_rate: 0.0001415086, weight_decay: 0.0001679241

epoch time: 1357.1392822266s, input_fired_time: 19.9915714264, output_fired_time: 7.3361167908, propagate_forward_time: 484.3203430176, propagate_backward_time: 833.7058105469, weight_update_time: 0.2965753675, other_time: 11.7855224609
epoch time: 100.6414947510s, input_fired_time: 1.0104515553, output_fired_time: 0.0000000000, propagate_forward_time: 97.5434570312, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0875854492
validation[          , genome  5779] predictions:    6057/  10000 (60.57%), best:    6057/10000 (60.57%), error:     11324.32910, best error:     11324.32910 on epoch:    20, epoch:   20/50, mu: 0.8565945029, learning_rate: 0.0001295500, weight_decay: 0.0001583309

epoch time: 1353.7845458984s, input_fired_time: 19.5893459320, output_fired_time: 7.3339982033, propagate_forward_time: 480.9852600098, propagate_backward_time: 834.0097045898, weight_update_time: 0.2986596823, other_time: 11.8662719727
epoch time: 100.5280456543s, input_fired_time: 0.9892683625, output_fired_time: 0.0000000000, propagate_forward_time: 97.4727172852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0660629272
validation[          , genome  5779] predictions:    5966/  10000 (59.66%), best:    6057/10000 (60.57%), error:     11669.58203, best error:     11324.32910 on epoch:    20, epoch:   21/50, mu: 0.8650942445, learning_rate: 0.0001186021, weight_decay: 0.0001492856

epoch time: 1313.5628662109s, input_fired_time: 19.4599723816, output_fired_time: 7.0418338776, propagate_forward_time: 469.5436706543, propagate_backward_time: 806.3017578125, weight_update_time: 0.2867040634, other_time: 11.2155761719
epoch time: 101.7389678955s, input_fired_time: 0.9848452210, output_fired_time: 0.0000000000, propagate_forward_time: 98.7062225342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0478973389
validation[          , genome  5779] predictions:    5804/  10000 (58.04%), best:    6057/10000 (60.57%), error:     12075.68164, best error:     11324.32910 on epoch:    20, epoch:   22/50, mu: 0.8730524778, learning_rate: 0.0001085793, weight_decay: 0.0001407572

epoch time: 1273.9476318359s, input_fired_time: 19.3728923798, output_fired_time: 6.7841806412, propagate_forward_time: 457.8241882324, propagate_backward_time: 779.1618652344, weight_update_time: 0.2813983262, other_time: 10.8044433594
epoch time: 93.5383071899s, input_fired_time: 0.9764263630, output_fired_time: 0.0000000000, propagate_forward_time: 90.8145904541, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7472915649
validation[          , genome  5779] predictions:    6004/  10000 (60.04%), best:    6057/10000 (60.57%), error:     11417.58008, best error:     11324.32910 on epoch:    20, epoch:   23/50, mu: 0.8805036545, learning_rate: 0.0000994036, weight_decay: 0.0001327159

epoch time: 1315.8459472656s, input_fired_time: 19.4402217865, output_fired_time: 7.0356316566, propagate_forward_time: 467.4489440918, propagate_backward_time: 810.6473999023, weight_update_time: 0.2895157933, other_time: 11.2737426758
epoch time: 102.8166885376s, input_fired_time: 1.0041393042, output_fired_time: 0.0000000000, propagate_forward_time: 99.7626190186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0499267578
validation[          , genome  5779] predictions:    6141/  10000 (61.41%), best:    6141/10000 (61.41%), error:     11323.78418, best error:     11323.78418 on epoch:    24, epoch:   24/50, mu: 0.8874800801, learning_rate: 0.0000910032, weight_decay: 0.0001251340

epoch time: 1319.9766845703s, input_fired_time: 19.4554901123, output_fired_time: 7.0487670898, propagate_forward_time: 468.0897216797, propagate_backward_time: 813.9746704102, weight_update_time: 0.2957125008, other_time: 11.4080200195
epoch time: 93.9418640137s, input_fired_time: 0.9676756859, output_fired_time: 0.0000000000, propagate_forward_time: 91.0680694580, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9061203003
validation[          , genome  5779] predictions:    5994/  10000 (59.94%), best:    6141/10000 (61.41%), error:     11633.63965, best error:     11323.78418 on epoch:    24, epoch:   25/50, mu: 0.8940119743, learning_rate: 0.0000833128, weight_decay: 0.0001179853

epoch time: 1352.9016113281s, input_fired_time: 19.5418853760, output_fired_time: 7.2879753113, propagate_forward_time: 480.7978820801, propagate_backward_time: 833.5392456055, weight_update_time: 0.2917854786, other_time: 11.7346801758
epoch time: 100.4808349609s, input_fired_time: 1.0002818108, output_fired_time: 0.0000000000, propagate_forward_time: 97.4198303223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0607223511
validation[          , genome  5779] predictions:    6001/  10000 (60.01%), best:    6141/10000 (61.41%), error:     11543.31055, best error:     11323.78418 on epoch:    24, epoch:   26/50, mu: 0.9001277089, learning_rate: 0.0000762722, weight_decay: 0.0001112450

epoch time: 1298.4454345703s, input_fired_time: 19.4406394958, output_fired_time: 6.9509997368, propagate_forward_time: 462.8352050781, propagate_backward_time: 798.0350952148, weight_update_time: 0.2833736539, other_time: 11.1834106445
epoch time: 90.4350204468s, input_fired_time: 0.9691867828, output_fired_time: 0.0000000000, propagate_forward_time: 87.8838348389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5820007324
validation[          , genome  5779] predictions:    5969/  10000 (59.69%), best:    6141/10000 (61.41%), error:     11550.74121, best error:     11323.78418 on epoch:    24, epoch:   27/50, mu: 0.9058538079, learning_rate: 0.0000698267, weight_decay: 0.0001048897

epoch time: 1296.3950195312s, input_fired_time: 19.4356803894, output_fired_time: 6.9687166214, propagate_forward_time: 461.2061767578, propagate_backward_time: 797.6726684570, weight_update_time: 0.2855795026, other_time: 11.1117553711
epoch time: 91.7967300415s, input_fired_time: 0.9805681109, output_fired_time: 0.0000000000, propagate_forward_time: 89.0251846313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7909774780
validation[          , genome  5779] predictions:    6052/  10000 (60.52%), best:    6052/10000 (60.52%), error:     11296.54199, best error:     11296.54199 on epoch:    28, epoch:   28/50, mu: 0.9112150669, learning_rate: 0.0000639258, weight_decay: 0.0000988975

epoch time: 1332.8453369141s, input_fired_time: 19.5277633667, output_fired_time: 7.1772518158, propagate_forward_time: 472.7958068848, propagate_backward_time: 821.8388061523, weight_update_time: 0.2913899124, other_time: 11.5057983398
epoch time: 99.3156890869s, input_fired_time: 1.0041232109, output_fired_time: 0.0000000000, propagate_forward_time: 96.2256851196, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0858840942
validation[          , genome  5779] predictions:    6021/  10000 (60.21%), best:    6052/10000 (60.52%), error:     11508.42969, best error:     11296.54199 on epoch:    28, epoch:   29/50, mu: 0.9162347317, learning_rate: 0.0000585236, weight_decay: 0.0000932476

epoch time: 1361.2747802734s, input_fired_time: 19.5611667633, output_fired_time: 7.3321628571, propagate_forward_time: 482.3784484863, propagate_backward_time: 840.2130126953, weight_update_time: 0.2938317657, other_time: 11.7900390625
epoch time: 98.3660964966s, input_fired_time: 0.9981612563, output_fired_time: 0.0000000000, propagate_forward_time: 95.3593902588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0085449219
validation[          , genome  5779] predictions:    6171/  10000 (61.71%), best:    6171/10000 (61.71%), error:     10997.14160, best error:     10997.14160 on epoch:    30, epoch:   30/50, mu: 0.9209345579, learning_rate: 0.0000535779, weight_decay: 0.0000879205

epoch time: 1330.5297851562s, input_fired_time: 19.5165042877, output_fired_time: 7.1498875618, propagate_forward_time: 473.2875671387, propagate_backward_time: 819.2022705078, weight_update_time: 0.2903967500, other_time: 11.3735351562
epoch time: 96.8091888428s, input_fired_time: 0.9934977293, output_fired_time: 0.0000000000, propagate_forward_time: 93.8537521362, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9619369507
validation[          , genome  5779] predictions:    6163/  10000 (61.63%), best:    6171/10000 (61.71%), error:     11139.54199, best error:     10997.14160 on epoch:    30, epoch:   31/50, mu: 0.9253349900, learning_rate: 0.0000490502, weight_decay: 0.0000828977

epoch time: 1307.8392333984s, input_fired_time: 19.4985904694, output_fired_time: 6.9855017662, propagate_forward_time: 466.2345581055, propagate_backward_time: 803.9858398438, weight_update_time: 0.2847915590, other_time: 11.1348266602
epoch time: 101.3197326660s, input_fired_time: 1.0099409819, output_fired_time: 0.0000000000, propagate_forward_time: 98.1999816895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1098098755
validation[          , genome  5779] predictions:    6154/  10000 (61.54%), best:    6171/10000 (61.71%), error:     11052.16504, best error:     10997.14160 on epoch:    30, epoch:   32/50, mu: 0.9294550419, learning_rate: 0.0000449051, weight_decay: 0.0000781619

epoch time: 1306.9058837891s, input_fired_time: 19.5105724335, output_fired_time: 7.0052332878, propagate_forward_time: 465.8361206055, propagate_backward_time: 803.3131103516, weight_update_time: 0.2897900641, other_time: 11.2407836914
epoch time: 92.9194793701s, input_fired_time: 0.9970092773, output_fired_time: 0.0000000000, propagate_forward_time: 90.1156387329, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8068313599
validation[          , genome  5779] predictions:    6192/  10000 (61.92%), best:    6171/10000 (61.71%), error:     11005.19336, best error:     10997.14160 on epoch:    30, epoch:   33/50, mu: 0.9333125949, learning_rate: 0.0000411103, weight_decay: 0.0000736966

epoch time: 1331.4663085938s, input_fired_time: 19.5760650635, output_fired_time: 7.0958185196, propagate_forward_time: 475.8521118164, propagate_backward_time: 817.7383422852, weight_update_time: 0.2890353799, other_time: 11.2039794922
epoch time: 90.3363265991s, input_fired_time: 0.9929112792, output_fired_time: 0.0000000000, propagate_forward_time: 87.6787414551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6646728516
validation[          , genome  5779] predictions:    6158/  10000 (61.58%), best:    6171/10000 (61.71%), error:     11060.62598, best error:     10997.14160 on epoch:    30, epoch:   34/50, mu: 0.9369243383, learning_rate: 0.0000376361, weight_decay: 0.0000694864

epoch time: 1319.3565673828s, input_fired_time: 19.5416564941, output_fired_time: 7.0326189995, propagate_forward_time: 469.2377014160, propagate_backward_time: 812.2593383789, weight_update_time: 0.2890246511, other_time: 11.2853393555
epoch time: 93.3603591919s, input_fired_time: 0.9871079326, output_fired_time: 0.0000000000, propagate_forward_time: 90.5598144531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8134384155
validation[          , genome  5779] predictions:    6208/  10000 (62.08%), best:    6171/10000 (61.71%), error:     11020.49316, best error:     10997.14160 on epoch:    30, epoch:   35/50, mu: 0.9403060079, learning_rate: 0.0000344556, weight_decay: 0.0000655168

epoch time: 1305.6502685547s, input_fired_time: 19.4810485840, output_fired_time: 7.0031452179, propagate_forward_time: 465.6079101562, propagate_backward_time: 802.3684082031, weight_update_time: 0.2864424288, other_time: 11.1896972656
epoch time: 87.4876937866s, input_fired_time: 0.9786133170, output_fired_time: 0.0000000000, propagate_forward_time: 84.7956008911, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7134780884
validation[          , genome  5779] predictions:    6242/  10000 (62.42%), best:    6242/10000 (62.42%), error:     10888.44629, best error:     10888.44629 on epoch:    36, epoch:   36/50, mu: 0.9434722066, learning_rate: 0.0000315439, weight_decay: 0.0000617739

epoch time: 1277.2379150391s, input_fired_time: 19.4437217712, output_fired_time: 6.7631545067, propagate_forward_time: 458.7060852051, propagate_backward_time: 781.6209716797, weight_update_time: 0.2809853256, other_time: 10.7039794922
epoch time: 93.0554504395s, input_fired_time: 1.0028548241, output_fired_time: 0.0000000000, propagate_forward_time: 90.2732467651, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7793502808
validation[          , genome  5779] predictions:    6289/  10000 (62.89%), best:    6289/10000 (62.89%), error:     10841.73633, best error:     10841.73633 on epoch:    37, epoch:   37/50, mu: 0.9464366436, learning_rate: 0.0000288782, weight_decay: 0.0000582448

epoch time: 1323.3018798828s, input_fired_time: 19.5544338226, output_fired_time: 7.1062083244, propagate_forward_time: 471.6873474121, propagate_backward_time: 813.5598144531, weight_update_time: 0.2897042036, other_time: 11.3940429688
epoch time: 90.9734497070s, input_fired_time: 0.9831765294, output_fired_time: 0.0000000000, propagate_forward_time: 88.3626556396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6276168823
validation[          , genome  5779] predictions:    6124/  10000 (61.24%), best:    6289/10000 (62.89%), error:     11129.40234, best error:     10841.73633 on epoch:    37, epoch:   38/50, mu: 0.9492122531, learning_rate: 0.0000264378, weight_decay: 0.0000549174

epoch time: 1277.9218750000s, input_fired_time: 19.4436149597, output_fired_time: 6.7815709114, propagate_forward_time: 454.6153259277, propagate_backward_time: 786.1328735352, weight_update_time: 0.2831600606, other_time: 10.9484252930
epoch time: 100.7669601440s, input_fired_time: 1.0248298645, output_fired_time: 0.0000000000, propagate_forward_time: 97.6758422852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0662841797
validation[          , genome  5779] predictions:    6211/  10000 (62.11%), best:    6289/10000 (62.89%), error:     10859.15723, best error:     10841.73633 on epoch:    37, epoch:   39/50, mu: 0.9518110156, learning_rate: 0.0000242036, weight_decay: 0.0000517800

epoch time: 1283.9978027344s, input_fired_time: 19.5222129822, output_fired_time: 6.8703651428, propagate_forward_time: 462.2243347168, propagate_backward_time: 784.5290527344, weight_update_time: 0.2815206647, other_time: 10.8518066406
epoch time: 95.0236511230s, input_fired_time: 1.0056104660, output_fired_time: 0.0000000000, propagate_forward_time: 92.1338500977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8841934204
validation[          , genome  5779] predictions:    6295/  10000 (62.95%), best:    6289/10000 (62.89%), error:     10871.84668, best error:     10841.73633 on epoch:    37, epoch:   40/50, mu: 0.9542441964, learning_rate: 0.0000221582, weight_decay: 0.0000488219

epoch time: 1257.8197021484s, input_fired_time: 19.4419670105, output_fired_time: 6.6721520424, propagate_forward_time: 452.0169982910, propagate_backward_time: 769.1394653320, weight_update_time: 0.2803455591, other_time: 10.5491333008
epoch time: 94.6748199463s, input_fired_time: 0.9979115725, output_fired_time: 0.0000000000, propagate_forward_time: 91.8861770630, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7907333374
validation[          , genome  5779] predictions:    6266/  10000 (62.66%), best:    6266/10000 (62.66%), error:     10826.73242, best error:     10826.73242 on epoch:    41, epoch:   41/50, mu: 0.9565223455, learning_rate: 0.0000202857, weight_decay: 0.0000460328

epoch time: 1324.7750244141s, input_fired_time: 19.6111373901, output_fired_time: 7.1297655106, propagate_forward_time: 472.1473999023, propagate_backward_time: 814.4459838867, weight_update_time: 0.2897135317, other_time: 11.4407958984
epoch time: 96.8946685791s, input_fired_time: 1.0168038607, output_fired_time: 0.0000000000, propagate_forward_time: 93.8919601440, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9859008789
validation[          , genome  5779] predictions:    6256/  10000 (62.56%), best:    6256/10000 (62.56%), error:     10818.12207, best error:     10818.12207 on epoch:    42, epoch:   42/50, mu: 0.9586553574, learning_rate: 0.0000185714, weight_decay: 0.0000434030

epoch time: 1347.2255859375s, input_fired_time: 19.6250991821, output_fired_time: 7.2724871635, propagate_forward_time: 479.9743957520, propagate_backward_time: 828.8194580078, weight_update_time: 0.2922280729, other_time: 11.5341796875
epoch time: 100.6449661255s, input_fired_time: 1.0213220119, output_fired_time: 0.0000000000, propagate_forward_time: 97.5711364746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0525054932
validation[          , genome  5779] predictions:    6239/  10000 (62.39%), best:    6256/10000 (62.56%), error:     11013.75293, best error:     10818.12207 on epoch:    42, epoch:   43/50, mu: 0.9606524110, learning_rate: 0.0000170019, weight_decay: 0.0000409234

epoch time: 1352.4829101562s, input_fired_time: 19.6341400146, output_fired_time: 7.2845282555, propagate_forward_time: 479.3743591309, propagate_backward_time: 834.4205322266, weight_update_time: 0.2951309085, other_time: 11.7692871094
epoch time: 100.8461608887s, input_fired_time: 1.0314159393, output_fired_time: 0.0000000000, propagate_forward_time: 97.7700347900, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0447082520
validation[          , genome  5779] predictions:    6360/  10000 (63.60%), best:    6360/10000 (63.60%), error:     10620.72754, best error:     10620.72754 on epoch:    44, epoch:   44/50, mu: 0.9625222683, learning_rate: 0.0000155652, weight_decay: 0.0000385855

epoch time: 1314.0335693359s, input_fired_time: 19.5666332245, output_fired_time: 7.0136699677, propagate_forward_time: 468.3666076660, propagate_backward_time: 807.7933349609, weight_update_time: 0.2857133746, other_time: 11.2933349609
epoch time: 100.3324279785s, input_fired_time: 1.0211113691, output_fired_time: 0.0000000000, propagate_forward_time: 97.2418518066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0694656372
validation[          , genome  5779] predictions:    6338/  10000 (63.38%), best:    6360/10000 (63.60%), error:     10682.46973, best error:     10620.72754 on epoch:    44, epoch:   45/50, mu: 0.9642729759, learning_rate: 0.0000142498, weight_decay: 0.0000363812

epoch time: 1353.5822753906s, input_fired_time: 19.6716232300, output_fired_time: 7.3237137794, propagate_forward_time: 478.7046203613, propagate_backward_time: 836.0842895508, weight_update_time: 0.2927190363, other_time: 11.7980346680
epoch time: 99.2529754639s, input_fired_time: 1.0296378136, output_fired_time: 0.0000000000, propagate_forward_time: 96.1861267090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0372085571
validation[          , genome  5779] predictions:    6299/  10000 (62.99%), best:    6360/10000 (63.60%), error:     10762.20215, best error:     10620.72754 on epoch:    44, epoch:   46/50, mu: 0.9659121633, learning_rate: 0.0000130456, weight_decay: 0.0000343028

epoch time: 1338.2435302734s, input_fired_time: 19.6561813354, output_fired_time: 7.2630586624, propagate_forward_time: 474.1081542969, propagate_backward_time: 825.5656127930, weight_update_time: 0.2934125066, other_time: 11.6505737305
epoch time: 99.2717514038s, input_fired_time: 1.0629216433, output_fired_time: 0.0000000000, propagate_forward_time: 96.2627792358, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9460525513
validation[          , genome  5779] predictions:    6321/  10000 (63.21%), best:    6360/10000 (63.60%), error:     10679.65723, best error:     10620.72754 on epoch:    44, epoch:   47/50, mu: 0.9674468637, learning_rate: 0.0000119431, weight_decay: 0.0000323431

epoch time: 1358.6259765625s, input_fired_time: 20.2377681732, output_fired_time: 7.3009095192, propagate_forward_time: 485.8422241211, propagate_backward_time: 833.3453369141, weight_update_time: 0.3002591431, other_time: 11.8997192383
epoch time: 104.6383285522s, input_fired_time: 1.0985752344, output_fired_time: 0.0000000000, propagate_forward_time: 101.4389648438, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1007919312
validation[          , genome  5779] predictions:    6245/  10000 (62.45%), best:    6360/10000 (63.60%), error:     10967.83496, best error:     10620.72754 on epoch:    44, epoch:   48/50, mu: 0.9688838124, learning_rate: 0.0000109338, weight_decay: 0.0000304954

epoch time: 1368.9134521484s, input_fired_time: 20.1084136963, output_fired_time: 7.3302760124, propagate_forward_time: 486.7460937500, propagate_backward_time: 842.7939453125, weight_update_time: 0.2946924865, other_time: 11.9346923828
epoch time: 104.8354339600s, input_fired_time: 1.0699075460, output_fired_time: 0.0000000000, propagate_forward_time: 101.6690444946, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0964813232
validation[          , genome  5779] predictions:    6405/  10000 (64.05%), best:    6405/10000 (64.05%), error:     10615.85938, best error:     10615.85938 on epoch:    49, epoch:   49/50, mu: 0.9702292085, learning_rate: 0.0000100099, weight_decay: 0.0000287533

epoch time: 1351.3868408203s, input_fired_time: 20.0434474945, output_fired_time: 7.2081189156, propagate_forward_time: 482.9376220703, propagate_backward_time: 829.5573120117, weight_update_time: 0.3053462803, other_time: 11.6403198242
epoch time: 98.4469299316s, input_fired_time: 1.0951721668, output_fired_time: 0.0000000000, propagate_forward_time: 95.5173873901, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8343734741
validation[          , genome  5779] predictions:    6342/  10000 (63.42%), best:    6405/10000 (64.05%), error:     10648.13770, best error:     10615.85938 on epoch:    49, epoch:   50/50, mu: 0.9714888930, learning_rate: 0.0000091639, weight_decay: 0.0000271106

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 492.7590026855s, input_fired_time: 5.2907648087, output_fired_time: 0.0000000000, propagate_forward_time: 477.4937133789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.9745178223
best training[          , genome  5779] predictions:   33903/  50000 (67.81%), best:    6405/10000 (64.05%), error:     49024.71094, best error:     10615.85938 on epoch:    49, epoch:   51/50, mu: 0.9726682901, learning_rate: 0.0000083895, weight_decay: 0.0000255618
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 98.7176437378s, input_fired_time: 1.0382726192, output_fired_time: 0.0000000000, propagate_forward_time: 95.7040481567, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9753265381
   testing[          , genome  5779] predictions:    6405/  10000 (64.05%), best:    6405/10000 (64.05%), error:     10615.85938, best error:     10615.85938 on epoch:    49, epoch:   51/50, mu: 0.9726682901, learning_rate: 0.0000083895, weight_decay: 0.0000255618
backpropagation finished successfully!
10:10:56 (30287): called boinc_finish(0)

</stderr_txt>
]]>