Nameexact_genome_1542605141_181_6306_1
Workunit4098588
Created19 Nov 2018, 7:10:48 UTC
Sent19 Nov 2018, 7:36:01 UTC
Report deadline13 Jan 2019, 22:03:45 UTC
Received22 Nov 2018, 12:04:03 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID31505
Run time20 hours 30 min 3 sec
CPU time20 hours 20 min 28 sec
Validate stateValid
Credit13,900.65
Device peak FLOPS4.20 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size547.81 MB
Peak swap size545.62 MB
Peak disk usage11.59 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_181_6306.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542605141_181_6306_1_r1424005818_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542605141_181_6306.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.487345
read mu: 0.487345
read mu_delta: 0.93836
read initial_learning_rate: 0.00195435
read learning_rate: 0.00195435
read learning_rate_delta: 0.919037
read initial_weight_decay: 0.00040593
read weight_decay: 0.00040593
read weight_decay_delta: 0.930476
read batch_size: 107
read epsilon: 1e-07
read alpha: 0.0574939
read input_dropout_probability: 0.00131542
read hidden_dropout_probability: 0.109307
read velocity_reset: 1082
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6306
read normal distribution: '0 -0.244194388389587 -0.986739099025726'
generator_str: '1527695616'
read generator: 1527695616
read generated_by_map:
1 split_edge 1
reading 92 nodes.
reading 339 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 665734580
post shuffle 1: 1275299046
epoch time: 1323.94946289063s, input_fired_time: 14.1016340255737, output_fired_time: 6.86535263061523, propagate_forward_time: 510.357116699219, propagate_backward_time: 782.886596679688, weight_update_time: 0.331299304962158, other_time: 9.73870849609375
epoch time: 104.836898803711s, input_fired_time: 1.02283561229706, output_fired_time: 0, propagate_forward_time: 102.161842346191, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.6522216796875
validation[          , genome  6306] predictions:    5215/  10000 (52.15%), best:    5215/10000 (52.15%), error:     13937.74512, best error:     13937.74512 on epoch:     0, epoch:    0/50, mu: 0.4873450398, learning_rate: 0.0019543483, weight_decay: 0.0004059296

epoch time: 1329.8347167969s, input_fired_time: 14.2369184494, output_fired_time: 6.8619761467, propagate_forward_time: 512.9877319336, propagate_backward_time: 785.9648437500, weight_update_time: 0.3384382129, other_time: 9.7832641602
epoch time: 105.2580184937s, input_fired_time: 1.0267590284, output_fired_time: 0.0000000000, propagate_forward_time: 102.5684051514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6628570557
validation[          , genome  6306] predictions:    5455/  10000 (54.55%), best:    5455/10000 (54.55%), error:     13041.39063, best error:     13041.39063 on epoch:     1, epoch:    1/50, mu: 0.5183287859, learning_rate: 0.0017961186, weight_decay: 0.0003777080

epoch time: 1323.6790771484s, input_fired_time: 14.2215003967, output_fired_time: 6.8985176086, propagate_forward_time: 510.2681884766, propagate_backward_time: 782.5685424805, weight_update_time: 0.3362779319, other_time: 9.7222290039
epoch time: 106.3489837646s, input_fired_time: 1.0517426729, output_fired_time: 0.0000000000, propagate_forward_time: 103.6340179443, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6632232666
validation[          , genome  6306] predictions:    5582/  10000 (55.82%), best:    5582/10000 (55.82%), error:     12853.43164, best error:     12853.43164 on epoch:     2, epoch:    2/50, mu: 0.5474026799, learning_rate: 0.0016506996, weight_decay: 0.0003514484

epoch time: 1323.6917724609s, input_fired_time: 14.2248582840, output_fired_time: 6.8640480042, propagate_forward_time: 510.3050842285, propagate_backward_time: 782.5588378906, weight_update_time: 0.3356648982, other_time: 9.7390136719
epoch time: 104.9020385742s, input_fired_time: 1.0453121662, output_fired_time: 0.0000000000, propagate_forward_time: 102.2182998657, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6384277344
validation[          , genome  6306] predictions:    5608/  10000 (56.08%), best:    5608/10000 (56.08%), error:     12847.39258, best error:     12847.39258 on epoch:     3, epoch:    3/50, mu: 0.5746844411, learning_rate: 0.0015170543, weight_decay: 0.0003270144

epoch time: 1324.7899169922s, input_fired_time: 14.2511653900, output_fired_time: 6.8934011459, propagate_forward_time: 510.6358032227, propagate_backward_time: 783.2570800781, weight_update_time: 0.3360863924, other_time: 9.7523803711
epoch time: 104.8638305664s, input_fired_time: 1.0415816307, output_fired_time: 0.0000000000, propagate_forward_time: 102.1585083008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6637420654
validation[          , genome  6306] predictions:    5676/  10000 (56.76%), best:    5676/10000 (56.76%), error:     12419.74023, best error:     12419.74023 on epoch:     4, epoch:    4/50, mu: 0.6002845764, learning_rate: 0.0013942291, weight_decay: 0.0003042792

epoch time: 1324.5371093750s, input_fired_time: 14.2576007843, output_fired_time: 6.8398509026, propagate_forward_time: 510.9464111328, propagate_backward_time: 782.7230224609, weight_update_time: 0.3337257504, other_time: 9.7702636719
epoch time: 105.2685165405s, input_fired_time: 1.0453529358, output_fired_time: 0.0000000000, propagate_forward_time: 102.5561676025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6669921875
validation[          , genome  6306] predictions:    5718/  10000 (57.18%), best:    5676/10000 (56.76%), error:     12426.98535, best error:     12419.74023 on epoch:     4, epoch:    5/50, mu: 0.6243067384, learning_rate: 0.0012813483, weight_decay: 0.0002831246

epoch time: 1323.7545166016s, input_fired_time: 14.2628965378, output_fired_time: 6.8617553711, propagate_forward_time: 510.1649780273, propagate_backward_time: 782.6894531250, weight_update_time: 0.3358912766, other_time: 9.7754516602
epoch time: 106.9737167358s, input_fired_time: 1.0584136248, output_fired_time: 0.0000000000, propagate_forward_time: 104.2392883301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6760177612
validation[          , genome  6306] predictions:    5695/  10000 (56.95%), best:    5676/10000 (56.76%), error:     12513.33984, best error:     12419.74023 on epoch:     4, epoch:    6/50, mu: 0.6468481421, learning_rate: 0.0011776066, weight_decay: 0.0002634408

epoch time: 1323.4058837891s, input_fired_time: 14.2594757080, output_fired_time: 6.8789834976, propagate_forward_time: 510.3361816406, propagate_backward_time: 782.1697998047, weight_update_time: 0.3343790770, other_time: 9.7613525391
epoch time: 105.1515960693s, input_fired_time: 1.0503332615, output_fired_time: 0.0000000000, propagate_forward_time: 102.4558486938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6454162598
validation[          , genome  6306] predictions:    5835/  10000 (58.35%), best:    5835/10000 (58.35%), error:     12229.33887, best error:     12229.33887 on epoch:     7, epoch:    7/50, mu: 0.6680001020, learning_rate: 0.0010822642, weight_decay: 0.0002451255

epoch time: 1328.8077392578s, input_fired_time: 14.3397645950, output_fired_time: 6.8961110115, propagate_forward_time: 512.0848388672, propagate_backward_time: 785.7404785156, weight_update_time: 0.3389943540, other_time: 9.7465820313
epoch time: 105.2056121826s, input_fired_time: 1.0455546379, output_fired_time: 0.0000000000, propagate_forward_time: 102.5190505981, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6410064697
validation[          , genome  6306] predictions:    5672/  10000 (56.72%), best:    5835/10000 (58.35%), error:     12447.98828, best error:     12229.33887 on epoch:     7, epoch:    8/50, mu: 0.6878482699, learning_rate: 0.0009946410, weight_decay: 0.0002280835

epoch time: 1324.0979003906s, input_fired_time: 14.2885503769, output_fired_time: 6.8504552841, propagate_forward_time: 510.5128173828, propagate_backward_time: 782.6792602539, weight_update_time: 0.3399316967, other_time: 9.7667846680
epoch time: 105.1481323242s, input_fired_time: 1.0475822687, output_fired_time: 0.0000000000, propagate_forward_time: 102.4484252930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6521224976
validation[          , genome  6306] predictions:    5729/  10000 (57.29%), best:    5729/10000 (57.29%), error:     12220.35938, best error:     12220.35938 on epoch:     9, epoch:    9/50, mu: 0.7064729929, learning_rate: 0.0009141120, weight_decay: 0.0002122263

epoch time: 1325.7315673828s, input_fired_time: 14.3020553589, output_fired_time: 6.8872537613, propagate_forward_time: 510.8965148926, propagate_backward_time: 783.8925170898, weight_update_time: 0.3374766707, other_time: 9.7533569336
epoch time: 105.1759338379s, input_fired_time: 1.0484529734, output_fired_time: 0.0000000000, propagate_forward_time: 102.4806518555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6468276978
validation[          , genome  6306] predictions:    5874/  10000 (58.74%), best:    5874/10000 (58.74%), error:     11897.14746, best error:     11897.14746 on epoch:    10, epoch:   10/50, mu: 0.7239496708, learning_rate: 0.0008401028, weight_decay: 0.0001974716

epoch time: 1324.7761230469s, input_fired_time: 14.2845897675, output_fired_time: 6.8606181145, propagate_forward_time: 511.1198730469, propagate_backward_time: 782.7753906250, weight_update_time: 0.3369356096, other_time: 9.7357177734
epoch time: 105.2074737549s, input_fired_time: 1.0590636730, output_fired_time: 0.0000000000, propagate_forward_time: 102.5005493164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6478576660
validation[          , genome  6306] predictions:    5620/  10000 (56.20%), best:    5874/10000 (58.74%), error:     12870.14551, best error:     11897.14746 on epoch:    10, epoch:   11/50, mu: 0.7403490543, learning_rate: 0.0007720856, weight_decay: 0.0001837426

epoch time: 1324.5944824219s, input_fired_time: 14.3757190704, output_fired_time: 6.8669772148, propagate_forward_time: 510.8509216309, propagate_backward_time: 782.7340087891, weight_update_time: 0.3383334279, other_time: 9.7668457031
epoch time: 105.1129989624s, input_fired_time: 1.0539675951, output_fired_time: 0.0000000000, propagate_forward_time: 102.4166793823, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6423492432
validation[          , genome  6306] predictions:    5643/  10000 (56.43%), best:    5874/10000 (58.74%), error:     12429.12109, best error:     11897.14746 on epoch:    10, epoch:   12/50, mu: 0.7557376027, learning_rate: 0.0007095753, weight_decay: 0.0001709682

epoch time: 1326.0374755859s, input_fired_time: 14.3286132813, output_fired_time: 6.8697476387, propagate_forward_time: 511.5243835449, propagate_backward_time: 783.5648803711, weight_update_time: 0.3413383961, other_time: 9.7498168945
epoch time: 105.2318649292s, input_fired_time: 1.0596008301, output_fired_time: 0.0000000000, propagate_forward_time: 102.5344009399, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6378631592
validation[          , genome  6306] predictions:    5753/  10000 (57.53%), best:    5874/10000 (58.74%), error:     12368.81152, best error:     11897.14746 on epoch:    10, epoch:   13/50, mu: 0.7701776028, learning_rate: 0.0006521260, weight_decay: 0.0001590819

epoch time: 1325.5582275391s, input_fired_time: 14.3489389420, output_fired_time: 6.8923735619, propagate_forward_time: 511.0752258301, propagate_backward_time: 783.5238647461, weight_update_time: 0.3361269236, other_time: 9.7178344727
epoch time: 105.3551177979s, input_fired_time: 1.0608673096, output_fired_time: 0.0000000000, propagate_forward_time: 102.6480484009, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6462020874
validation[          , genome  6306] predictions:    5859/  10000 (58.59%), best:    5859/10000 (58.59%), error:     11888.16406, best error:     11888.16406 on epoch:    14, epoch:   14/50, mu: 0.7837275267, learning_rate: 0.0005993280, weight_decay: 0.0001480219

epoch time: 1340.0261230469s, input_fired_time: 14.4441709518, output_fired_time: 6.9155273438, propagate_forward_time: 517.2235107422, propagate_backward_time: 791.6316528320, weight_update_time: 0.3420077264, other_time: 9.8112182617
epoch time: 107.3246917725s, input_fired_time: 1.0646365881, output_fired_time: 0.0000000000, propagate_forward_time: 104.5881576538, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6718978882
validation[          , genome  6306] predictions:    5935/  10000 (59.35%), best:    5935/10000 (59.35%), error:     11739.18359, best error:     11739.18359 on epoch:    15, epoch:   15/50, mu: 0.7964422107, learning_rate: 0.0005508047, weight_decay: 0.0001377309

epoch time: 1328.9266357422s, input_fired_time: 14.3776273727, output_fired_time: 6.8931441307, propagate_forward_time: 513.5974731445, propagate_backward_time: 784.2867431641, weight_update_time: 0.3340973556, other_time: 9.7715454102
epoch time: 105.0561904907s, input_fired_time: 1.0644477606, output_fired_time: 0.0000000000, propagate_forward_time: 102.3439102173, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6478347778
validation[          , genome  6306] predictions:    5971/  10000 (59.71%), best:    5971/10000 (59.71%), error:     11571.79199, best error:     11571.79199 on epoch:    16, epoch:   16/50, mu: 0.8083731532, learning_rate: 0.0005062099, weight_decay: 0.0001281554

epoch time: 1325.5880126953s, input_fired_time: 14.3763170242, output_fired_time: 6.8591341972, propagate_forward_time: 511.2254638672, propagate_backward_time: 783.3502197266, weight_update_time: 0.3380260766, other_time: 9.7768554688
epoch time: 105.3075408936s, input_fired_time: 1.0693666935, output_fired_time: 0.0000000000, propagate_forward_time: 102.5871887207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6509857178
validation[          , genome  6306] predictions:    6015/  10000 (60.15%), best:    6015/10000 (60.15%), error:     11543.86523, best error:     11543.86523 on epoch:    17, epoch:   17/50, mu: 0.8195686936, learning_rate: 0.0004652257, weight_decay: 0.0001192456

epoch time: 1324.9067382813s, input_fired_time: 14.3748130798, output_fired_time: 6.8617548943, propagate_forward_time: 511.1856384277, propagate_backward_time: 782.7129516602, weight_update_time: 0.3362136483, other_time: 9.7716674805
epoch time: 105.0685119629s, input_fired_time: 1.0512733459, output_fired_time: 0.0000000000, propagate_forward_time: 102.3495941162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6676483154
validation[          , genome  6306] predictions:    5984/  10000 (59.84%), best:    6015/10000 (60.15%), error:     11669.08301, best error:     11543.86523 on epoch:    17, epoch:   18/50, mu: 0.8300741315, learning_rate: 0.0004275597, weight_decay: 0.0001109552

epoch time: 1325.9060058594s, input_fired_time: 14.3876934052, output_fired_time: 6.8370203972, propagate_forward_time: 511.4700927734, propagate_backward_time: 783.4700927734, weight_update_time: 0.3348115385, other_time: 9.7410888672
epoch time: 105.1172180176s, input_fired_time: 1.0822827816, output_fired_time: 0.0000000000, propagate_forward_time: 102.4040603638, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6308746338
validation[          , genome  6306] predictions:    6005/  10000 (60.05%), best:    6015/10000 (60.15%), error:     11546.42188, best error:     11543.86523 on epoch:    17, epoch:   19/50, mu: 0.8399319649, learning_rate: 0.0003929432, weight_decay: 0.0001032412

epoch time: 1325.2611083984s, input_fired_time: 14.4111156464, output_fired_time: 6.8605313301, propagate_forward_time: 510.9635925293, propagate_backward_time: 783.3161010742, weight_update_time: 0.3338047862, other_time: 9.7097778320
epoch time: 105.1386032104s, input_fired_time: 1.0660650730, output_fired_time: 0.0000000000, propagate_forward_time: 102.4319992065, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6405410767
validation[          , genome  6306] predictions:    6053/  10000 (60.53%), best:    6053/10000 (60.53%), error:     11389.07520, best error:     11389.07520 on epoch:    20, epoch:   20/50, mu: 0.8491821885, learning_rate: 0.0003611294, weight_decay: 0.0000960635

epoch time: 1325.2218017578s, input_fired_time: 14.3973445892, output_fired_time: 6.8742828369, propagate_forward_time: 511.1258850098, propagate_backward_time: 783.0902099609, weight_update_time: 0.3355945051, other_time: 9.7341308594
epoch time: 105.2402114868s, input_fired_time: 1.0683807135, output_fired_time: 0.0000000000, propagate_forward_time: 102.5375823975, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6342468262
validation[          , genome  6306] predictions:    5942/  10000 (59.42%), best:    6053/10000 (60.53%), error:     11743.07715, best error:     11389.07520 on epoch:    20, epoch:   21/50, mu: 0.8578622341, learning_rate: 0.0003318913, weight_decay: 0.0000893848

epoch time: 1324.8645019531s, input_fired_time: 14.4006872177, output_fired_time: 6.8598675728, propagate_forward_time: 510.8346862793, propagate_backward_time: 783.0515136719, weight_update_time: 0.3315904140, other_time: 9.7177734375
epoch time: 105.2462387085s, input_fired_time: 1.0678243637, output_fired_time: 0.0000000000, propagate_forward_time: 102.5420913696, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6363220215
validation[          , genome  6306] predictions:    6008/  10000 (60.08%), best:    6053/10000 (60.53%), error:     11466.62598, best error:     11389.07520 on epoch:    20, epoch:   22/50, mu: 0.8660072088, learning_rate: 0.0003050204, weight_decay: 0.0000831705

epoch time: 1327.3891601563s, input_fired_time: 14.4103250504, output_fired_time: 6.8702878952, propagate_forward_time: 512.2496948242, propagate_backward_time: 783.9722900391, weight_update_time: 0.3421104848, other_time: 9.8866577148
epoch time: 105.3138122559s, input_fired_time: 1.0659499168, output_fired_time: 0.0000000000, propagate_forward_time: 102.6054687500, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6423950195
validation[          , genome  6306] predictions:    6011/  10000 (60.11%), best:    6053/10000 (60.53%), error:     11399.65039, best error:     11389.07520 on epoch:    20, epoch:   23/50, mu: 0.8736501336, learning_rate: 0.0002803251, weight_decay: 0.0000773881

epoch time: 1324.9113769531s, input_fired_time: 14.3919897079, output_fired_time: 6.8774018288, propagate_forward_time: 510.9477539063, propagate_backward_time: 782.9786376953, weight_update_time: 0.3328959048, other_time: 9.7155761719
epoch time: 105.2975997925s, input_fired_time: 1.0711498260, output_fired_time: 0.0000000000, propagate_forward_time: 102.5622863770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6641616821
validation[          , genome  6306] predictions:    6029/  10000 (60.29%), best:    6053/10000 (60.53%), error:     11594.16016, best error:     11389.07520 on epoch:    20, epoch:   24/50, mu: 0.8808219433, learning_rate: 0.0002576292, weight_decay: 0.0000720078

epoch time: 1325.6827392578s, input_fired_time: 14.4270639420, output_fired_time: 6.8443284035, propagate_forward_time: 511.0956420898, propagate_backward_time: 783.5554809570, weight_update_time: 0.3368864954, other_time: 9.7601318359
epoch time: 105.0856704712s, input_fired_time: 1.0748279095, output_fired_time: 0.0000000000, propagate_forward_time: 102.3838424683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6269989014
validation[          , genome  6306] predictions:    5999/  10000 (59.99%), best:    6053/10000 (60.53%), error:     11479.13086, best error:     11389.07520 on epoch:    20, epoch:   25/50, mu: 0.8875517249, learning_rate: 0.0002367708, weight_decay: 0.0000670016

epoch time: 1325.1303710938s, input_fired_time: 14.4436407089, output_fired_time: 6.8720722198, propagate_forward_time: 511.0902404785, propagate_backward_time: 783.0274047852, weight_update_time: 0.3326732218, other_time: 9.6970825195
epoch time: 105.1256256104s, input_fired_time: 1.0735639334, output_fired_time: 0.0000000000, propagate_forward_time: 102.4152908325, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6367721558
validation[          , genome  6306] predictions:    6090/  10000 (60.90%), best:    6090/10000 (60.90%), error:     11276.77051, best error:     11276.77051 on epoch:    26, epoch:   26/50, mu: 0.8938666582, learning_rate: 0.0002176011, weight_decay: 0.0000623434

epoch time: 1325.2587890625s, input_fired_time: 14.4896783829, output_fired_time: 6.8606300354, propagate_forward_time: 511.2420959473, propagate_backward_time: 782.9481201172, weight_update_time: 0.3353067636, other_time: 9.7183837891
epoch time: 105.1996459961s, input_fired_time: 1.0809900761, output_fired_time: 0.0000000000, propagate_forward_time: 102.4499511719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6687011719
validation[          , genome  6306] predictions:    6134/  10000 (61.34%), best:    6134/10000 (61.34%), error:     11233.33789, best error:     11233.33789 on epoch:    27, epoch:   27/50, mu: 0.8997923136, learning_rate: 0.0001999835, weight_decay: 0.0000580091

epoch time: 1327.3983154297s, input_fired_time: 14.4742927551, output_fired_time: 6.8770337105, propagate_forward_time: 512.4366455078, propagate_backward_time: 783.9019165039, weight_update_time: 0.3319121599, other_time: 9.7084350586
epoch time: 105.5509185791s, input_fired_time: 1.0892771482, output_fired_time: 0.0000000000, propagate_forward_time: 102.8138046265, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6478347778
validation[          , genome  6306] predictions:    6116/  10000 (61.16%), best:    6116/10000 (61.16%), error:     11185.35156, best error:     11185.35156 on epoch:    28, epoch:   28/50, mu: 0.9053527117, learning_rate: 0.0001837923, weight_decay: 0.0000539761

epoch time: 1325.1888427734s, input_fired_time: 14.4345273972, output_fired_time: 6.8669638634, propagate_forward_time: 511.2225036621, propagate_backward_time: 782.9333496094, weight_update_time: 0.3322813213, other_time: 9.7314453125
epoch time: 105.1882781982s, input_fired_time: 1.0824823380, output_fired_time: 0.0000000000, propagate_forward_time: 102.4638519287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6419448853
validation[          , genome  6306] predictions:    6224/  10000 (62.24%), best:    6224/10000 (62.24%), error:     10969.82617, best error:     10969.82617 on epoch:    29, epoch:   29/50, mu: 0.9105703831, learning_rate: 0.0001689119, weight_decay: 0.0000502235

epoch time: 1324.8856201172s, input_fired_time: 14.4565391541, output_fired_time: 6.8934073448, propagate_forward_time: 510.6817016602, propagate_backward_time: 783.1050415039, weight_update_time: 0.3397538960, other_time: 9.7489013672
epoch time: 108.5392074585s, input_fired_time: 1.1098728180, output_fired_time: 0.0000000000, propagate_forward_time: 105.7764358521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6529006958
validation[          , genome  6306] predictions:    6182/  10000 (61.82%), best:    6224/10000 (62.24%), error:     11073.28809, best error:     10969.82617 on epoch:    29, epoch:   30/50, mu: 0.9154664278, learning_rate: 0.0001552363, weight_decay: 0.0000467318

epoch time: 1328.8505859375s, input_fired_time: 14.4970273972, output_fired_time: 6.9195394516, propagate_forward_time: 512.5971069336, propagate_backward_time: 785.0755004883, weight_update_time: 0.3351721466, other_time: 9.7613525391
epoch time: 105.0616607666s, input_fired_time: 1.0883982182, output_fired_time: 0.0000000000, propagate_forward_time: 102.3493804932, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6238784790
validation[          , genome  6306] predictions:    6189/  10000 (61.89%), best:    6224/10000 (62.24%), error:     11020.89063, best error:     10969.82617 on epoch:    29, epoch:   31/50, mu: 0.9200606942, learning_rate: 0.0001426679, weight_decay: 0.0000434828

epoch time: 1325.5487060547s, input_fired_time: 14.4815025330, output_fired_time: 6.8604664803, propagate_forward_time: 511.1871337891, propagate_backward_time: 783.2992553711, weight_update_time: 0.3351908028, other_time: 9.7203979492
epoch time: 105.5094375610s, input_fired_time: 1.0809856653, output_fired_time: 0.0000000000, propagate_forward_time: 102.7739868164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6544647217
validation[          , genome  6306] predictions:    6111/  10000 (61.11%), best:    6224/10000 (62.24%), error:     11192.31543, best error:     10969.82617 on epoch:    29, epoch:   32/50, mu: 0.9243717790, learning_rate: 0.0001311171, weight_decay: 0.0000404597

epoch time: 1326.2829589844s, input_fired_time: 14.4860000610, output_fired_time: 6.8736834526, propagate_forward_time: 511.1618957520, propagate_backward_time: 784.0791625977, weight_update_time: 0.3331182301, other_time: 9.6823120117
epoch time: 105.6223602295s, input_fired_time: 1.0915188789, output_fired_time: 0.0000000000, propagate_forward_time: 102.8935852051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6372528076
validation[          , genome  6306] predictions:    6302/  10000 (63.02%), best:    6302/10000 (63.02%), error:     10907.55078, best error:     10907.55078 on epoch:    33, epoch:   33/50, mu: 0.9284170866, learning_rate: 0.0001205015, weight_decay: 0.0000376468

epoch time: 1324.5118408203s, input_fired_time: 14.5434160233, output_fired_time: 6.8562588692, propagate_forward_time: 510.9084472656, propagate_backward_time: 782.4763793945, weight_update_time: 0.3326494992, other_time: 9.7273559570
epoch time: 105.2402420044s, input_fired_time: 1.0887728930, output_fired_time: 0.0000000000, propagate_forward_time: 102.5162429810, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6352233887
validation[          , genome  6306] predictions:    6178/  10000 (61.78%), best:    6302/10000 (63.02%), error:     10972.90723, best error:     10907.55078 on epoch:    33, epoch:   34/50, mu: 0.9322130680, learning_rate: 0.0001107453, weight_decay: 0.0000350295

epoch time: 1326.3858642578s, input_fired_time: 14.4973382950, output_fired_time: 6.8720774651, propagate_forward_time: 511.4638671875, propagate_backward_time: 783.8215332031, weight_update_time: 0.3328051269, other_time: 9.7310791016
epoch time: 106.9215545654s, input_fired_time: 1.1072782278, output_fired_time: 0.0000000000, propagate_forward_time: 104.1551437378, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6591339111
validation[          , genome  6306] predictions:    6241/  10000 (62.41%), best:    6241/10000 (62.41%), error:     10860.59082, best error:     10860.59082 on epoch:    35, epoch:   35/50, mu: 0.9357750416, learning_rate: 0.0001017791, weight_decay: 0.0000325941

epoch time: 1422.6307373047s, input_fired_time: 15.2024345398, output_fired_time: 7.4092841148, propagate_forward_time: 546.8027954102, propagate_backward_time: 843.0883178711, weight_update_time: 0.3546343446, other_time: 10.1279296875
epoch time: 119.1602859497s, input_fired_time: 1.2220895290, output_fired_time: 0.0000000000, propagate_forward_time: 116.0706634521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8675308228
validation[          , genome  6306] predictions:    6240/  10000 (62.40%), best:    6241/10000 (62.41%), error:     10929.93750, best error:     10860.59082 on epoch:    35, epoch:   36/50, mu: 0.9391174912, learning_rate: 0.0000935388, weight_decay: 0.0000303280

epoch time: 1325.7502441406s, input_fired_time: 14.5167303085, output_fired_time: 6.8857917786, propagate_forward_time: 511.5902099609, propagate_backward_time: 783.0223388672, weight_update_time: 0.3346972466, other_time: 9.7352294922
epoch time: 105.2066879272s, input_fired_time: 1.0908496380, output_fired_time: 0.0000000000, propagate_forward_time: 102.4634780884, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6523590088
validation[          , genome  6306] predictions:    6211/  10000 (62.11%), best:    6241/10000 (62.41%), error:     11056.08496, best error:     10860.59082 on epoch:    35, epoch:   37/50, mu: 0.9422538877, learning_rate: 0.0000859656, weight_decay: 0.0000282195

epoch time: 1333.9461669922s, input_fired_time: 14.5594625473, output_fired_time: 6.8993186951, propagate_forward_time: 514.6052856445, propagate_backward_time: 788.1311035156, weight_update_time: 0.3383838236, other_time: 9.7510375977
epoch time: 105.6948928833s, input_fired_time: 1.0956848860, output_fired_time: 0.0000000000, propagate_forward_time: 102.9439544678, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6552505493
validation[          , genome  6306] predictions:    6271/  10000 (62.71%), best:    6271/10000 (62.71%), error:     10807.20605, best error:     10807.20605 on epoch:    38, epoch:   38/50, mu: 0.9451969862, learning_rate: 0.0000790056, weight_decay: 0.0000262576

epoch time: 1324.4805908203s, input_fired_time: 14.5252647400, output_fired_time: 6.8697247505, propagate_forward_time: 510.7169799805, propagate_backward_time: 782.6784057617, weight_update_time: 0.3329765201, other_time: 9.6901855469
epoch time: 105.2605667114s, input_fired_time: 1.0877624750, output_fired_time: 0.0000000000, propagate_forward_time: 102.4900741577, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6827316284
validation[          , genome  6306] predictions:    6211/  10000 (62.11%), best:    6271/10000 (62.71%), error:     10995.79004, best error:     10807.20605 on epoch:    38, epoch:   39/50, mu: 0.9479586482, learning_rate: 0.0000726091, weight_decay: 0.0000244321

epoch time: 1325.8741455078s, input_fired_time: 14.5275993347, output_fired_time: 6.8799386024, propagate_forward_time: 511.5207214355, propagate_backward_time: 783.2787475586, weight_update_time: 0.3352762759, other_time: 9.6671752930
epoch time: 105.0587921143s, input_fired_time: 1.0937806368, output_fired_time: 0.0000000000, propagate_forward_time: 102.3289260864, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6360855103
validation[          , genome  6306] predictions:    6223/  10000 (62.23%), best:    6271/10000 (62.71%), error:     10963.38965, best error:     10807.20605 on epoch:    38, epoch:   40/50, mu: 0.9505500793, learning_rate: 0.0000667304, weight_decay: 0.0000227335

epoch time: 1326.7327880859s, input_fired_time: 14.5868711472, output_fired_time: 6.8764495850, propagate_forward_time: 511.7224731445, propagate_backward_time: 783.8273925781, weight_update_time: 0.3371206224, other_time: 9.7195434570
epoch time: 105.3758468628s, input_fired_time: 1.0982687473, output_fired_time: 0.0000000000, propagate_forward_time: 102.6227493286, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6548309326
validation[          , genome  6306] predictions:    6263/  10000 (62.63%), best:    6271/10000 (62.71%), error:     10839.15820, best error:     10807.20605 on epoch:    38, epoch:   41/50, mu: 0.9529817700, learning_rate: 0.0000613277, weight_decay: 0.0000211530

epoch time: 1325.2398681641s, input_fired_time: 14.5323925018, output_fired_time: 6.8524818420, propagate_forward_time: 511.1489257813, propagate_backward_time: 783.0181884766, weight_update_time: 0.3336827159, other_time: 9.6878662109
epoch time: 105.2273025513s, input_fired_time: 1.1009712219, output_fired_time: 0.0000000000, propagate_forward_time: 102.4949111938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6314163208
validation[          , genome  6306] predictions:    6241/  10000 (62.41%), best:    6271/10000 (62.71%), error:     10831.76465, best error:     10807.20605 on epoch:    38, epoch:   42/50, mu: 0.9552635550, learning_rate: 0.0000563625, weight_decay: 0.0000196823

epoch time: 1326.2792968750s, input_fired_time: 14.5507411957, output_fired_time: 6.8626799583, propagate_forward_time: 511.8296508789, propagate_backward_time: 783.2965087891, weight_update_time: 0.3364916742, other_time: 9.7396850586
epoch time: 105.5034408569s, input_fired_time: 1.0983430147, output_fired_time: 0.0000000000, propagate_forward_time: 102.7596969604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6454010010
validation[          , genome  6306] predictions:    6262/  10000 (62.62%), best:    6271/10000 (62.71%), error:     10839.63281, best error:     10807.20605 on epoch:    38, epoch:   43/50, mu: 0.9574047327, learning_rate: 0.0000517992, weight_decay: 0.0000183139

epoch time: 1325.4733886719s, input_fired_time: 14.5360126495, output_fired_time: 6.8589711189, propagate_forward_time: 511.0550231934, propagate_backward_time: 783.2853393555, weight_update_time: 0.3340241015, other_time: 9.7379760742
epoch time: 105.3003997803s, input_fired_time: 1.1093267202, output_fired_time: 0.0000000000, propagate_forward_time: 102.5459671021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6451034546
validation[          , genome  6306] predictions:    6305/  10000 (63.05%), best:    6305/10000 (63.05%), error:     10649.50879, best error:     10649.50879 on epoch:    44, epoch:   44/50, mu: 0.9594138861, learning_rate: 0.0000476054, weight_decay: 0.0000170407

epoch time: 1327.3168945313s, input_fired_time: 14.5680150986, output_fired_time: 6.8593244553, propagate_forward_time: 511.9000854492, propagate_backward_time: 784.2739257813, weight_update_time: 0.3349134624, other_time: 9.7155151367
epoch time: 105.0709838867s, input_fired_time: 1.1011632681, output_fired_time: 0.0000000000, propagate_forward_time: 102.3350067139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6348114014
validation[          , genome  6306] predictions:    6293/  10000 (62.93%), best:    6305/10000 (63.05%), error:     10745.87500, best error:     10649.50879 on epoch:    44, epoch:   45/50, mu: 0.9612992406, learning_rate: 0.0000437511, weight_decay: 0.0000158560

epoch time: 1346.4670410156s, input_fired_time: 14.6901521683, output_fired_time: 6.9613609314, propagate_forward_time: 520.1487426758, propagate_backward_time: 794.8758544922, weight_update_time: 0.3377680779, other_time: 9.7909545898
epoch time: 105.3608016968s, input_fired_time: 1.0984145403, output_fired_time: 0.0000000000, propagate_forward_time: 102.6085586548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6538314819
validation[          , genome  6306] predictions:    6309/  10000 (63.09%), best:    6305/10000 (63.05%), error:     10684.66992, best error:     10649.50879 on epoch:    44, epoch:   46/50, mu: 0.9630683661, learning_rate: 0.0000402089, weight_decay: 0.0000147536

epoch time: 1324.8131103516s, input_fired_time: 14.5618886948, output_fired_time: 6.8253836632, propagate_forward_time: 511.0401306152, propagate_backward_time: 782.6038818359, weight_update_time: 0.3376618624, other_time: 9.7817382813
epoch time: 105.4358139038s, input_fired_time: 1.1096909046, output_fired_time: 0.0000000000, propagate_forward_time: 102.6613235474, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6648025513
validation[          , genome  6306] predictions:    6327/  10000 (63.27%), best:    6305/10000 (63.05%), error:     10652.95215, best error:     10649.50879 on epoch:    44, epoch:   47/50, mu: 0.9647284150, learning_rate: 0.0000369535, weight_decay: 0.0000137279

epoch time: 1325.1854248047s, input_fired_time: 14.5683784485, output_fired_time: 6.8393149376, propagate_forward_time: 511.0492248535, propagate_backward_time: 782.9854736328, weight_update_time: 0.3363968432, other_time: 9.7430419922
epoch time: 105.1929550171s, input_fired_time: 1.1011371613, output_fired_time: 0.0000000000, propagate_forward_time: 102.4516296387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6401901245
validation[          , genome  6306] predictions:    6305/  10000 (63.05%), best:    6305/10000 (63.05%), error:     10705.94629, best error:     10649.50879 on epoch:    44, epoch:   48/50, mu: 0.9662861824, learning_rate: 0.0000339616, weight_decay: 0.0000127735

epoch time: 1325.2901611328s, input_fired_time: 14.6062889099, output_fired_time: 6.8772382736, propagate_forward_time: 511.4429321289, propagate_backward_time: 782.6548461914, weight_update_time: 0.3353332877, other_time: 9.7088623047
epoch time: 104.9917297363s, input_fired_time: 1.0987707376, output_fired_time: 0.0000000000, propagate_forward_time: 102.2690658569, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6238937378
validation[          , genome  6306] predictions:    6258/  10000 (62.58%), best:    6305/10000 (63.05%), error:     10898.99023, best error:     10649.50879 on epoch:    44, epoch:   49/50, mu: 0.9677479267, learning_rate: 0.0000312120, weight_decay: 0.0000118854

epoch time: 1326.9216308594s, input_fired_time: 14.5904388428, output_fired_time: 6.8714137077, propagate_forward_time: 511.9377746582, propagate_backward_time: 783.7893676758, weight_update_time: 0.3318216503, other_time: 9.7326049805
epoch time: 105.3718261719s, input_fired_time: 1.1048656702, output_fired_time: 0.0000000000, propagate_forward_time: 102.6209106445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6460494995
validation[          , genome  6306] predictions:    6294/  10000 (62.94%), best:    6305/10000 (63.05%), error:     10683.43652, best error:     10649.50879 on epoch:    44, epoch:   50/50, mu: 0.9691195488, learning_rate: 0.0000286850, weight_decay: 0.0000110591

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 524.5498046875s, input_fired_time: 5.5163230896, output_fired_time: 0.0000000000, propagate_forward_time: 510.7446289063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2888793945
best training[          , genome  6306] predictions:   34878/  50000 (69.76%), best:    6305/10000 (63.05%), error:     45854.98438, best error:     10649.50879 on epoch:    44, epoch:   51/50, mu: 0.9704066515, learning_rate: 0.0000263625, weight_decay: 0.0000102902
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 105.1671066284s, input_fired_time: 1.1032162905, output_fired_time: 0.0000000000, propagate_forward_time: 102.4231338501, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6407546997
   testing[          , genome  6306] predictions:    6305/  10000 (63.05%), best:    6305/10000 (63.05%), error:     10649.50879, best error:     10649.50879 on epoch:    44, epoch:   51/50, mu: 0.9704066515, learning_rate: 0.0000263625, weight_decay: 0.0000102902
backpropagation finished successfully!
17:52:37 (5744): called boinc_finish(0)

</stderr_txt>
]]>