Nameexact_genome_1542189904_181_5896_4
Workunit4088610
Created26 Dec 2018, 3:29:02 UTC
Sent26 Dec 2018, 3:29:20 UTC
Report deadline5 Feb 2019, 18:04:34 UTC
Received26 Dec 2018, 22:11:05 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID23692
Run time17 hours 35 min 29 sec
CPU time17 hours 17 min 27 sec
Validate stateValid
Credit10,401.95
Device peak FLOPS4.14 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size543.95 MB
Peak swap size563.68 MB
Peak disk usage9.94 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542189904_181_5896.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542189904_181_5896_4_r1604686625_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542189904_181_5896.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.489359
read mu: 0.489359
read mu_delta: 0.957829
read initial_learning_rate: 0.00317199
read learning_rate: 0.00317199
read learning_rate_delta: 0.921602
read initial_weight_decay: 0.000624759
read weight_decay: 0.000624759
read weight_decay_delta: 0.935568
read batch_size: 108
read epsilon: 1e-07
read alpha: 0.0860703
read input_dropout_probability: 0.00129223
read hidden_dropout_probability: 0.120087
read velocity_reset: 1568
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5896
read normal distribution: '1 -0.712941706180573 0.417061507701874'
generator_str: '892833809'
read generator: 892833809
read generated_by_map:
1 add_edge 1
reading 76 nodes.
reading 275 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1389586274
post shuffle 1: 2130794575
epoch time: 1149.265625s, input_fired_time: 12.2151031494141, output_fired_time: 6.24030923843384, propagate_forward_time: 453.181549072266, propagate_backward_time: 666.981201171875, weight_update_time: 0.317552179098129, other_time: 10.6474609375
epoch time: 92.7657470703125s, input_fired_time: 0.902770757675171, output_fired_time: 0, propagate_forward_time: 90.0519638061523, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.81101226806641
validation[          , genome  5896] predictions:    5453/  10000 (54.53%), best:    5453/10000 (54.53%), error:     13114.53027, best error:     13114.53027 on epoch:     0, epoch:    0/50, mu: 0.4893587828, learning_rate: 0.0031719869, weight_decay: 0.0006247585

epoch time: 1180.6105957031s, input_fired_time: 12.2472095490, output_fired_time: 6.3970618248, propagate_forward_time: 474.9868469238, propagate_backward_time: 676.7158203125, weight_update_time: 0.3311379850, other_time: 10.2636718750
epoch time: 100.1889801025s, input_fired_time: 0.9218680263, output_fired_time: 0.0000000000, propagate_forward_time: 97.5784988403, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6886138916
validation[          , genome  5896] predictions:    5411/  10000 (54.11%), best:    5411/10000 (54.11%), error:     13062.33105, best error:     13062.33105 on epoch:     1, epoch:    1/50, mu: 0.5104711652, learning_rate: 0.0029233098, weight_decay: 0.0005845043

epoch time: 1165.4313964844s, input_fired_time: 12.0284690857, output_fired_time: 6.2878055573, propagate_forward_time: 469.7246398926, propagate_backward_time: 667.4546508789, weight_update_time: 0.3241665363, other_time: 9.9358520508
epoch time: 95.1190261841s, input_fired_time: 0.8943986893, output_fired_time: 0.0000000000, propagate_forward_time: 92.5519180298, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6727066040
validation[          , genome  5896] predictions:    5448/  10000 (54.48%), best:    5448/10000 (54.48%), error:     12974.21973, best error:     12974.21973 on epoch:     2, epoch:    2/50, mu: 0.5306932330, learning_rate: 0.0026941283, weight_decay: 0.0005468436

epoch time: 1505.3404541016s, input_fired_time: 11.8953256607, output_fired_time: 6.2476515770, propagate_forward_time: 814.8274536133, propagate_backward_time: 662.4845581055, weight_update_time: 0.3236443400, other_time: 9.8853759766
epoch time: 98.1742401123s, input_fired_time: 0.9042617083, output_fired_time: 0.0000000000, propagate_forward_time: 95.5767822266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6931991577
validation[          , genome  5896] predictions:    5365/  10000 (53.65%), best:    5448/10000 (54.48%), error:     13368.62500, best error:     12974.21973 on epoch:     2, epoch:    3/50, mu: 0.5500624776, learning_rate: 0.0024829141, weight_decay: 0.0005116095

epoch time: 1210.5040283203s, input_fired_time: 12.6034145355, output_fired_time: 6.6439166069, propagate_forward_time: 489.1500854492, propagate_backward_time: 691.4934082031, weight_update_time: 0.3371577859, other_time: 10.6132202148
epoch time: 102.1220245361s, input_fired_time: 0.9753088951, output_fired_time: 0.0000000000, propagate_forward_time: 99.3132400513, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8334732056
validation[          , genome  5896] predictions:    5492/  10000 (54.92%), best:    5492/10000 (54.92%), error:     12697.61719, best error:     12697.61719 on epoch:     4, epoch:    4/50, mu: 0.5686149001, learning_rate: 0.0022882589, weight_decay: 0.0004786456

epoch time: 1246.9942626953s, input_fired_time: 12.9783163071, output_fired_time: 6.8076734543, propagate_forward_time: 503.7829284668, propagate_backward_time: 712.4382324219, weight_update_time: 0.3613919914, other_time: 10.9871826172
epoch time: 101.3094406128s, input_fired_time: 0.9787062407, output_fired_time: 0.0000000000, propagate_forward_time: 98.4284667969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9022674561
validation[          , genome  5896] predictions:    5517/  10000 (55.17%), best:    5492/10000 (54.92%), error:     12740.84473, best error:     12697.61719 on epoch:     4, epoch:    5/50, mu: 0.5863849521, learning_rate: 0.0021088640, weight_decay: 0.0004478057

epoch time: 1219.4758300781s, input_fired_time: 12.7865447998, output_fired_time: 6.6270022392, propagate_forward_time: 484.5489807129, propagate_backward_time: 704.3850708008, weight_update_time: 0.3299969733, other_time: 11.1283569336
epoch time: 101.3005218506s, input_fired_time: 0.9652152658, output_fired_time: 0.0000000000, propagate_forward_time: 98.5029067993, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8323974609
validation[          , genome  5896] predictions:    5616/  10000 (56.16%), best:    5616/10000 (56.16%), error:     12570.37207, best error:     12570.37207 on epoch:     6, epoch:    6/50, mu: 0.6034056544, learning_rate: 0.0019435334, weight_decay: 0.0004189528

epoch time: 1268.9139404297s, input_fired_time: 13.2469816208, output_fired_time: 6.8218374252, propagate_forward_time: 507.4459533691, propagate_backward_time: 730.0078125000, weight_update_time: 0.3548979759, other_time: 11.3914794922
epoch time: 105.4261398315s, input_fired_time: 1.0195448399, output_fired_time: 0.0000000000, propagate_forward_time: 102.4546813965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9519119263
validation[          , genome  5896] predictions:    5639/  10000 (56.39%), best:    5639/10000 (56.39%), error:     12413.52344, best error:     12413.52344 on epoch:     7, epoch:    7/50, mu: 0.6197085977, learning_rate: 0.0017911644, weight_decay: 0.0003919589

epoch time: 1237.2374267578s, input_fired_time: 12.9492664337, output_fired_time: 6.6081638336, propagate_forward_time: 493.5516967773, propagate_backward_time: 713.0123291016, weight_update_time: 0.3384103477, other_time: 11.1160278320
epoch time: 92.6078948975s, input_fired_time: 0.9012812376, output_fired_time: 0.0000000000, propagate_forward_time: 89.9465560913, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7600555420
validation[          , genome  5896] predictions:    5604/  10000 (56.04%), best:    5639/10000 (56.39%), error:     12536.91895, best error:     12413.52344 on epoch:     7, epoch:    8/50, mu: 0.6353240013, learning_rate: 0.0016507409, weight_decay: 0.0003667043

epoch time: 1137.9998779297s, input_fired_time: 11.8859653473, output_fired_time: 6.0776882172, propagate_forward_time: 445.3820800781, propagate_backward_time: 664.3831787109, weight_update_time: 0.3065589368, other_time: 10.2709960938
epoch time: 90.5967483521s, input_fired_time: 0.8899365664, output_fired_time: 0.0000000000, propagate_forward_time: 87.9790649414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7277450562
validation[          , genome  5896] predictions:    5628/  10000 (56.28%), best:    5639/10000 (56.39%), error:     12505.88770, best error:     12413.52344 on epoch:     7, epoch:    9/50, mu: 0.6502808928, learning_rate: 0.0015213261, weight_decay: 0.0003430769

epoch time: 1125.0118408203s, input_fired_time: 11.7523431778, output_fired_time: 5.9817190170, propagate_forward_time: 438.6124267578, propagate_backward_time: 658.4954833984, weight_update_time: 0.3094851375, other_time: 10.1699218750
epoch time: 90.9538497925s, input_fired_time: 0.8811458945, output_fired_time: 0.0000000000, propagate_forward_time: 88.3569946289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7157058716
validation[          , genome  5896] predictions:    5743/  10000 (57.43%), best:    5743/10000 (57.43%), error:     12236.40918, best error:     12236.40918 on epoch:    10, epoch:   10/50, mu: 0.6646070480, learning_rate: 0.0014020574, weight_decay: 0.0003209719

epoch time: 1125.2153320313s, input_fired_time: 11.8033008575, output_fired_time: 5.9593725204, propagate_forward_time: 439.8557128906, propagate_backward_time: 657.3695678711, weight_update_time: 0.3016619384, other_time: 10.2273559570
epoch time: 90.9756088257s, input_fired_time: 0.8879718781, output_fired_time: 0.0000000000, propagate_forward_time: 88.3320617676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7555770874
validation[          , genome  5896] predictions:    5625/  10000 (56.25%), best:    5743/10000 (57.43%), error:     12372.45313, best error:     12236.40918 on epoch:    10, epoch:   11/50, mu: 0.6783290505, learning_rate: 0.0012921389, weight_decay: 0.0003002911

epoch time: 1123.4071044922s, input_fired_time: 11.7678470612, output_fired_time: 5.9915184975, propagate_forward_time: 438.8429870605, propagate_backward_time: 656.6400146484, weight_update_time: 0.2998396754, other_time: 10.1646728516
epoch time: 90.6248321533s, input_fired_time: 0.8807200193, output_fired_time: 0.0000000000, propagate_forward_time: 87.9659118652, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7781982422
validation[          , genome  5896] predictions:    5765/  10000 (57.65%), best:    5765/10000 (57.65%), error:     12120.58887, best error:     12120.58887 on epoch:    12, epoch:   12/50, mu: 0.6914724112, learning_rate: 0.0011908379, weight_decay: 0.0002809429

epoch time: 1129.9080810547s, input_fired_time: 11.8029031754, output_fired_time: 5.9568266869, propagate_forward_time: 440.0481262207, propagate_backward_time: 661.7457275391, weight_update_time: 0.3048318028, other_time: 10.3546142578
epoch time: 90.8345794678s, input_fired_time: 0.8873060346, output_fired_time: 0.0000000000, propagate_forward_time: 88.2158279419, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7314453125
validation[          , genome  5896] predictions:    5894/  10000 (58.94%), best:    5894/10000 (58.94%), error:     11860.95703, best error:     11860.95703 on epoch:    13, epoch:   13/50, mu: 0.7040615082, learning_rate: 0.0010974787, weight_decay: 0.0002628412

epoch time: 1119.6967773438s, input_fired_time: 11.7048530579, output_fired_time: 5.8771181107, propagate_forward_time: 436.4328002930, propagate_backward_time: 655.5474853516, weight_update_time: 0.2994815409, other_time: 10.1345825195
epoch time: 90.5810546875s, input_fired_time: 0.9109516144, output_fired_time: 0.0000000000, propagate_forward_time: 87.9224395752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7476654053
validation[          , genome  5896] predictions:    5769/  10000 (57.69%), best:    5894/10000 (58.94%), error:     12117.16895, best error:     11860.95703 on epoch:    13, epoch:   14/50, mu: 0.7161197066, learning_rate: 0.0010114387, weight_decay: 0.0002459059

epoch time: 1118.2158203125s, input_fired_time: 11.7279806137, output_fired_time: 5.8966331482, propagate_forward_time: 435.7519836426, propagate_backward_time: 654.7149658203, weight_update_time: 0.3004659116, other_time: 10.1242675781
epoch time: 89.7845077515s, input_fired_time: 0.8807762861, output_fired_time: 0.0000000000, propagate_forward_time: 87.1373977661, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7663345337
validation[          , genome  5896] predictions:    5771/  10000 (57.71%), best:    5894/10000 (58.94%), error:     12037.41797, best error:     11860.95703 on epoch:    13, epoch:   15/50, mu: 0.7276694179, learning_rate: 0.0009321439, weight_decay: 0.0002300618

epoch time: 1121.6848144531s, input_fired_time: 11.7784299850, output_fired_time: 5.9517693520, propagate_forward_time: 436.9840393066, propagate_backward_time: 656.7788085938, weight_update_time: 0.3003980219, other_time: 10.1917724609
epoch time: 90.5315856934s, input_fired_time: 0.8939550519, output_fired_time: 0.0000000000, propagate_forward_time: 87.8855438232, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7520904541
validation[          , genome  5896] predictions:    5638/  10000 (56.38%), best:    5894/10000 (58.94%), error:     12427.44727, best error:     11860.95703 on epoch:    13, epoch:   16/50, mu: 0.7387320399, learning_rate: 0.0008590658, weight_decay: 0.0002152385

epoch time: 1131.3492431641s, input_fired_time: 11.8556070328, output_fired_time: 5.9729623795, propagate_forward_time: 440.9870300293, propagate_backward_time: 662.2388305664, weight_update_time: 0.3055666983, other_time: 10.2947387695
epoch time: 90.8190231323s, input_fired_time: 0.9058013558, output_fired_time: 0.0000000000, propagate_forward_time: 88.1676101685, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7456130981
validation[          , genome  5896] predictions:    5830/  10000 (58.30%), best:    5894/10000 (58.94%), error:     11981.73535, best error:     11860.95703 on epoch:    13, epoch:   17/50, mu: 0.7493281960, learning_rate: 0.0007917168, weight_decay: 0.0002013703

epoch time: 1126.8049316406s, input_fired_time: 11.8515625000, output_fired_time: 5.9577755928, propagate_forward_time: 439.1044616699, propagate_backward_time: 659.6936035156, weight_update_time: 0.3052315414, other_time: 10.1975097656
epoch time: 90.3850479126s, input_fired_time: 0.8881230950, output_fired_time: 0.0000000000, propagate_forward_time: 87.7613067627, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7356185913
validation[          , genome  5896] predictions:    5786/  10000 (57.86%), best:    5894/10000 (58.94%), error:     12134.78418, best error:     11860.95703 on epoch:    13, epoch:   18/50, mu: 0.7594774961, learning_rate: 0.0007296478, weight_decay: 0.0001883957

epoch time: 1123.4597167969s, input_fired_time: 11.7849674225, output_fired_time: 5.9399290085, propagate_forward_time: 437.9826965332, propagate_backward_time: 657.5922241211, weight_update_time: 0.2992189527, other_time: 10.1599731445
epoch time: 91.2114410400s, input_fired_time: 0.8858181834, output_fired_time: 0.0000000000, propagate_forward_time: 88.5920486450, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7335739136
validation[          , genome  5896] predictions:    5930/  10000 (59.30%), best:    5930/10000 (59.30%), error:     11693.42773, best error:     11693.42773 on epoch:    19, epoch:   19/50, mu: 0.7691987753, learning_rate: 0.0006724449, weight_decay: 0.0001762570

epoch time: 1135.8043212891s, input_fired_time: 11.8527688980, output_fired_time: 6.0187311172, propagate_forward_time: 446.7275085449, propagate_backward_time: 661.0423583984, weight_update_time: 0.3047287762, other_time: 10.1629638672
epoch time: 93.9373626709s, input_fired_time: 0.9021767974, output_fired_time: 0.0000000000, propagate_forward_time: 91.2367324829, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7984542847
validation[          , genome  5896] predictions:    5885/  10000 (58.85%), best:    5930/10000 (59.30%), error:     11852.35449, best error:     11693.42773 on epoch:    19, epoch:   20/50, mu: 0.7785100937, learning_rate: 0.0006197267, weight_decay: 0.0001649005

epoch time: 1151.7484130859s, input_fired_time: 11.9604692459, output_fired_time: 6.2320680618, propagate_forward_time: 459.3049011230, propagate_backward_time: 663.9468383789, weight_update_time: 0.3227947056, other_time: 10.3041381836
epoch time: 93.9215393066s, input_fired_time: 0.9040149450, output_fired_time: 0.0000000000, propagate_forward_time: 91.2530136108, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7645111084
validation[          , genome  5896] predictions:    5817/  10000 (58.17%), best:    5930/10000 (59.30%), error:     11883.08887, best error:     11693.42773 on epoch:    19, epoch:   21/50, mu: 0.7874287367, learning_rate: 0.0005711414, weight_decay: 0.0001542757

epoch time: 1147.8032226563s, input_fired_time: 11.9371595383, output_fired_time: 6.1745815277, propagate_forward_time: 456.3278503418, propagate_backward_time: 663.2162475586, weight_update_time: 0.3148559332, other_time: 10.1473999023
epoch time: 93.9067687988s, input_fired_time: 0.8942474723, output_fired_time: 0.0000000000, propagate_forward_time: 91.3416671753, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6708526611
validation[          , genome  5896] predictions:    5895/  10000 (58.95%), best:    5930/10000 (59.30%), error:     11853.66211, best error:     11693.42773 on epoch:    19, epoch:   22/50, mu: 0.7959712744, learning_rate: 0.0005263651, weight_decay: 0.0001443354

epoch time: 1135.3604736328s, input_fired_time: 11.8841695786, output_fired_time: 6.0710759163, propagate_forward_time: 448.4554138184, propagate_backward_time: 659.0436401367, weight_update_time: 0.3142232001, other_time: 9.9061889648
epoch time: 92.0522918701s, input_fired_time: 0.9021344185, output_fired_time: 0.0000000000, propagate_forward_time: 89.4485397339, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7016143799
validation[          , genome  5896] predictions:    5827/  10000 (58.27%), best:    5930/10000 (59.30%), error:     11935.66895, best error:     11693.42773 on epoch:    19, epoch:   23/50, mu: 0.8041535616, learning_rate: 0.0004850991, weight_decay: 0.0001350356

epoch time: 1134.1236572266s, input_fired_time: 11.8402824402, output_fired_time: 6.0408797264, propagate_forward_time: 446.6329040527, propagate_backward_time: 659.6134643555, weight_update_time: 0.3062601686, other_time: 9.9960327148
epoch time: 92.2471160889s, input_fired_time: 0.9034475088, output_fired_time: 0.0000000000, propagate_forward_time: 89.6430206299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7006454468
validation[          , genome  5896] predictions:    5855/  10000 (58.55%), best:    5855/10000 (58.55%), error:     11676.49512, best error:     11676.49512 on epoch:    24, epoch:   24/50, mu: 0.8119907975, learning_rate: 0.0004470684, weight_decay: 0.0001263350

epoch time: 1133.3861083984s, input_fired_time: 11.8644084930, output_fired_time: 6.0263385773, propagate_forward_time: 446.7205810547, propagate_backward_time: 658.8095703125, weight_update_time: 0.3071838021, other_time: 9.9652099609
epoch time: 89.7514038086s, input_fired_time: 0.9043108821, output_fired_time: 0.0000000000, propagate_forward_time: 87.1783447266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6687469482
validation[          , genome  5896] predictions:    5978/  10000 (59.78%), best:    5978/10000 (59.78%), error:     11535.01660, best error:     11535.01660 on epoch:    25, epoch:   25/50, mu: 0.8194975853, learning_rate: 0.0004120191, weight_decay: 0.0001181951

epoch time: 1121.7226562500s, input_fired_time: 11.8447446823, output_fired_time: 5.9300837517, propagate_forward_time: 436.4268798828, propagate_backward_time: 657.5156860352, weight_update_time: 0.2970615923, other_time: 10.0053100586
epoch time: 89.8839645386s, input_fired_time: 0.8982676268, output_fired_time: 0.0000000000, propagate_forward_time: 87.2727890015, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7129058838
validation[          , genome  5896] predictions:    5998/  10000 (59.98%), best:    5978/10000 (59.78%), error:     11545.15625, best error:     11535.01660 on epoch:    25, epoch:   26/50, mu: 0.8266877532, learning_rate: 0.0003797177, weight_decay: 0.0001105795

epoch time: 1119.7152099609s, input_fired_time: 11.7850151062, output_fired_time: 5.9293990135, propagate_forward_time: 435.5743713379, propagate_backward_time: 656.3654785156, weight_update_time: 0.2981838882, other_time: 10.0609130859
epoch time: 90.3276748657s, input_fired_time: 0.9056101441, output_fired_time: 0.0000000000, propagate_forward_time: 87.7033615112, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7187042236
validation[          , genome  5896] predictions:    5867/  10000 (58.67%), best:    5978/10000 (59.78%), error:     11703.46094, best error:     11535.01660 on epoch:    25, epoch:   27/50, mu: 0.8335747123, learning_rate: 0.0003499486, weight_decay: 0.0001034547

epoch time: 1123.5102539063s, input_fired_time: 11.8672876358, output_fired_time: 5.9417572021, propagate_forward_time: 436.8574523926, propagate_backward_time: 658.7676391602, weight_update_time: 0.2978205979, other_time: 10.0761108398
epoch time: 89.5239944458s, input_fired_time: 0.8925471902, output_fired_time: 0.0000000000, propagate_forward_time: 86.9707717896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6606750488
validation[          , genome  5896] predictions:    5895/  10000 (58.95%), best:    5978/10000 (59.78%), error:     11595.35254, best error:     11535.01660 on epoch:    25, epoch:   28/50, mu: 0.8401712775, learning_rate: 0.0003225133, weight_decay: 0.0000967890

epoch time: 1120.7296142578s, input_fired_time: 11.8362932205, output_fired_time: 5.9500794411, propagate_forward_time: 436.3572387695, propagate_backward_time: 656.5506591797, weight_update_time: 0.2977800667, other_time: 10.0353393555
epoch time: 89.7381744385s, input_fired_time: 0.9054256082, output_fired_time: 0.0000000000, propagate_forward_time: 87.1274032593, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7053451538
validation[          , genome  5896] predictions:    5976/  10000 (59.76%), best:    5978/10000 (59.78%), error:     11602.99023, best error:     11535.01660 on epoch:    25, epoch:   29/50, mu: 0.8464896679, learning_rate: 0.0002972290, weight_decay: 0.0000905527

epoch time: 1116.3057861328s, input_fired_time: 11.8562393188, output_fired_time: 5.9352469444, propagate_forward_time: 435.0750427246, propagate_backward_time: 653.5087890625, weight_update_time: 0.2996684015, other_time: 9.9304199219
epoch time: 90.2815475464s, input_fired_time: 0.9048744440, output_fired_time: 0.0000000000, propagate_forward_time: 87.6626205444, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7140502930
validation[          , genome  5896] predictions:    6008/  10000 (60.08%), best:    6008/10000 (60.08%), error:     11427.93359, best error:     11427.93359 on epoch:    30, epoch:   30/50, mu: 0.8525415659, learning_rate: 0.0002739268, weight_decay: 0.0000847182

epoch time: 1117.3618164063s, input_fired_time: 11.8270616531, output_fired_time: 5.8974370956, propagate_forward_time: 435.4508056641, propagate_backward_time: 654.2691040039, weight_update_time: 0.3007885516, other_time: 9.9174194336
epoch time: 90.4401245117s, input_fired_time: 0.9044926167, output_fired_time: 0.0000000000, propagate_forward_time: 87.8631134033, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6725158691
validation[          , genome  5896] predictions:    6019/  10000 (60.19%), best:    6019/10000 (60.19%), error:     11416.50391, best error:     11416.50391 on epoch:    31, epoch:   31/50, mu: 0.8583382964, learning_rate: 0.0002524515, weight_decay: 0.0000792597

epoch time: 1118.7777099609s, input_fired_time: 11.8553562164, output_fired_time: 5.8811817169, propagate_forward_time: 434.9347229004, propagate_backward_time: 656.1557006836, weight_update_time: 0.3005252182, other_time: 9.9507446289
epoch time: 89.9154205322s, input_fired_time: 0.9071254730, output_fired_time: 0.0000000000, propagate_forward_time: 87.3351821899, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6731109619
validation[          , genome  5896] predictions:    6027/  10000 (60.27%), best:    6019/10000 (60.19%), error:     11432.42188, best error:     11416.50391 on epoch:    31, epoch:   32/50, mu: 0.8638905287, learning_rate: 0.0002326598, weight_decay: 0.0000741528

epoch time: 1119.5865478516s, input_fired_time: 11.8434009552, output_fired_time: 5.8651781082, propagate_forward_time: 435.2922363281, propagate_backward_time: 656.6541748047, weight_update_time: 0.2941443920, other_time: 9.9315185547
epoch time: 89.6191940308s, input_fired_time: 0.9041628242, output_fired_time: 0.0000000000, propagate_forward_time: 86.9917984009, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7232360840
validation[          , genome  5896] predictions:    5973/  10000 (59.73%), best:    6019/10000 (60.19%), error:     11480.46875, best error:     11416.50391 on epoch:    31, epoch:   33/50, mu: 0.8692086339, learning_rate: 0.0002144198, weight_decay: 0.0000693750

epoch time: 1118.7438964844s, input_fired_time: 11.8175506592, output_fired_time: 5.8449459076, propagate_forward_time: 434.1453552246, propagate_backward_time: 657.1152954102, weight_update_time: 0.2946498990, other_time: 9.8207397461
epoch time: 90.1145172119s, input_fired_time: 0.9067286849, output_fired_time: 0.0000000000, propagate_forward_time: 87.5186767578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6891098022
validation[          , genome  5896] predictions:    6084/  10000 (60.84%), best:    6084/10000 (60.84%), error:     11281.43750, best error:     11281.43750 on epoch:    34, epoch:   34/50, mu: 0.8743025064, learning_rate: 0.0001976097, weight_decay: 0.0000649051

epoch time: 1116.8060302734s, input_fired_time: 11.8259849548, output_fired_time: 5.8374214172, propagate_forward_time: 433.0710449219, propagate_backward_time: 656.2621459961, weight_update_time: 0.2909348011, other_time: 9.8095092773
epoch time: 89.4384765625s, input_fired_time: 0.9050851464, output_fired_time: 0.0000000000, propagate_forward_time: 86.8868026733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6465911865
validation[          , genome  5896] predictions:    5996/  10000 (59.96%), best:    6084/10000 (60.84%), error:     11487.29590, best error:     11281.43750 on epoch:    34, epoch:   35/50, mu: 0.8791815639, learning_rate: 0.0001821175, weight_decay: 0.0000607231

epoch time: 1123.1434326172s, input_fired_time: 11.9145488739, output_fired_time: 5.9073295593, propagate_forward_time: 435.9721069336, propagate_backward_time: 659.5309448242, weight_update_time: 0.2942963839, other_time: 9.8184814453
epoch time: 89.3288192749s, input_fired_time: 0.9018908739, output_fired_time: 0.0000000000, propagate_forward_time: 86.7361602783, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6907653809
validation[          , genome  5896] predictions:    6077/  10000 (60.77%), best:    6077/10000 (60.77%), error:     11274.34277, best error:     11274.34277 on epoch:    36, epoch:   36/50, mu: 0.8838548660, learning_rate: 0.0001678399, weight_decay: 0.0000568106

epoch time: 1117.6962890625s, input_fired_time: 11.8205213547, output_fired_time: 5.8732652664, propagate_forward_time: 434.0197143555, propagate_backward_time: 656.2486572266, weight_update_time: 0.2952996492, other_time: 9.7340698242
epoch time: 89.4628372192s, input_fired_time: 0.8932825923, output_fired_time: 0.0000000000, propagate_forward_time: 86.9291915894, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6403656006
validation[          , genome  5896] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11261.61426, best error:     11261.61426 on epoch:    37, epoch:   37/50, mu: 0.8883310556, learning_rate: 0.0001546816, weight_decay: 0.0000531502

epoch time: 1118.4653320313s, input_fired_time: 11.7939367294, output_fired_time: 5.8505353928, propagate_forward_time: 433.1724548340, propagate_backward_time: 657.8540649414, weight_update_time: 0.2909427583, other_time: 9.7942504883
epoch time: 96.5951156616s, input_fired_time: 0.9370131493, output_fired_time: 0.0000000000, propagate_forward_time: 93.9170227051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7410812378
validation[          , genome  5896] predictions:    6015/  10000 (60.15%), best:    6049/10000 (60.49%), error:     11409.08496, best error:     11261.61426 on epoch:    37, epoch:   38/50, mu: 0.8926184773, learning_rate: 0.0001425549, weight_decay: 0.0000497257

epoch time: 1113.4437255859s, input_fired_time: 11.8308887482, output_fired_time: 5.8456287384, propagate_forward_time: 431.3529357910, propagate_backward_time: 654.6227416992, weight_update_time: 0.2949101329, other_time: 9.7915649414
epoch time: 89.1633834839s, input_fired_time: 0.9000708461, output_fired_time: 0.0000000000, propagate_forward_time: 86.6320877075, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6312255859
validation[          , genome  5896] predictions:    6140/  10000 (61.40%), best:    6140/10000 (61.40%), error:     11131.50000, best error:     11131.50000 on epoch:    39, epoch:   39/50, mu: 0.8967251182, learning_rate: 0.0001313789, weight_decay: 0.0000465218

epoch time: 1114.9929199219s, input_fired_time: 11.8083143234, output_fired_time: 5.8492932320, propagate_forward_time: 432.4083251953, propagate_backward_time: 655.1708374023, weight_update_time: 0.2908023298, other_time: 9.7561645508
epoch time: 89.4654693604s, input_fired_time: 0.9115005732, output_fired_time: 0.0000000000, propagate_forward_time: 86.9115753174, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6423950195
validation[          , genome  5896] predictions:    6041/  10000 (60.41%), best:    6140/10000 (61.40%), error:     11349.00684, best error:     11131.50000 on epoch:    39, epoch:   40/50, mu: 0.9006586075, learning_rate: 0.0001210790, weight_decay: 0.0000435243

epoch time: 1115.7358398438s, input_fired_time: 11.8168725967, output_fired_time: 5.8209686279, propagate_forward_time: 431.7704162598, propagate_backward_time: 656.6712646484, weight_update_time: 0.2870974541, other_time: 9.6563720703
epoch time: 89.7133178711s, input_fired_time: 0.9213100672, output_fired_time: 0.0000000000, propagate_forward_time: 87.1786422729, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6133651733
validation[          , genome  5896] predictions:    6095/  10000 (60.95%), best:    6140/10000 (61.40%), error:     11220.84570, best error:     11131.50000 on epoch:    39, epoch:   41/50, mu: 0.9044262171, learning_rate: 0.0001115867, weight_decay: 0.0000407199

epoch time: 1109.6595458984s, input_fired_time: 11.6785964966, output_fired_time: 5.7521452904, propagate_forward_time: 428.7677917480, propagate_backward_time: 654.0182495117, weight_update_time: 0.2836411893, other_time: 9.4426879883
epoch time: 89.7972412109s, input_fired_time: 0.9198709726, output_fired_time: 0.0000000000, propagate_forward_time: 87.2260513306, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6513214111
validation[          , genome  5896] predictions:    6083/  10000 (60.83%), best:    6140/10000 (61.40%), error:     11184.48926, best error:     11131.50000 on epoch:    39, epoch:   42/50, mu: 0.9080349207, learning_rate: 0.0001028385, weight_decay: 0.0000380963

epoch time: 1111.9105224609s, input_fired_time: 11.7365512848, output_fired_time: 5.7583293915, propagate_forward_time: 430.1296691895, propagate_backward_time: 654.8619384766, weight_update_time: 0.2837363482, other_time: 9.4240722656
epoch time: 88.3804244995s, input_fired_time: 0.8909135461, output_fired_time: 0.0000000000, propagate_forward_time: 85.8955612183, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5939483643
validation[          , genome  5896] predictions:    6121/  10000 (61.21%), best:    6140/10000 (61.40%), error:     11159.85547, best error:     11131.50000 on epoch:    39, epoch:   43/50, mu: 0.9114914536, learning_rate: 0.0000947762, weight_decay: 0.0000356417

epoch time: 1110.8206787109s, input_fired_time: 11.7016000748, output_fired_time: 5.7550101280, propagate_forward_time: 429.1517639160, propagate_backward_time: 654.7518310547, weight_update_time: 0.2830794454, other_time: 9.4604492188
epoch time: 88.5015945435s, input_fired_time: 0.9233261943, output_fired_time: 0.0000000000, propagate_forward_time: 85.9769897461, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6012802124
validation[          , genome  5896] predictions:    6132/  10000 (61.32%), best:    6140/10000 (61.40%), error:     11187.29102, best error:     11131.50000 on epoch:    39, epoch:   44/50, mu: 0.9148021936, learning_rate: 0.0000873459, weight_decay: 0.0000333452

epoch time: 1107.7650146484s, input_fired_time: 11.6751003265, output_fired_time: 5.7388501167, propagate_forward_time: 428.2723999023, propagate_backward_time: 652.6846923828, weight_update_time: 0.2840091586, other_time: 9.3939819336
epoch time: 88.4365844727s, input_fired_time: 0.8951128721, output_fired_time: 0.0000000000, propagate_forward_time: 85.9446945190, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5967788696
validation[          , genome  5896] predictions:    6129/  10000 (61.29%), best:    6129/10000 (61.29%), error:     11093.11914, best error:     11093.11914 on epoch:    45, epoch:   45/50, mu: 0.9179733396, learning_rate: 0.0000804982, weight_decay: 0.0000311967

epoch time: 1110.7792968750s, input_fired_time: 11.7053804398, output_fired_time: 5.7529926300, propagate_forward_time: 429.2951660156, propagate_backward_time: 654.6113281250, weight_update_time: 0.2884685993, other_time: 9.4144287109
epoch time: 88.6837615967s, input_fired_time: 0.8962439895, output_fired_time: 0.0000000000, propagate_forward_time: 86.2121887207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5753326416
validation[          , genome  5896] predictions:    6096/  10000 (60.96%), best:    6129/10000 (61.29%), error:     11168.42188, best error:     11093.11914 on epoch:    45, epoch:   46/50, mu: 0.9210107327, learning_rate: 0.0000741873, weight_decay: 0.0000291867

epoch time: 1109.4417724609s, input_fired_time: 11.7357330322, output_fired_time: 5.7516088486, propagate_forward_time: 428.8591308594, propagate_backward_time: 653.6663818359, weight_update_time: 0.2830553651, other_time: 9.4289550781
epoch time: 88.7798614502s, input_fired_time: 0.9037851691, output_fired_time: 0.0000000000, propagate_forward_time: 86.2287521362, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6473236084
validation[          , genome  5896] predictions:    6089/  10000 (60.89%), best:    6129/10000 (61.29%), error:     11191.85645, best error:     11093.11914 on epoch:    45, epoch:   47/50, mu: 0.9239200354, learning_rate: 0.0000683712, weight_decay: 0.0000273061

epoch time: 1109.3594970703s, input_fired_time: 11.7740306854, output_fired_time: 5.8063254356, propagate_forward_time: 428.3610839844, propagate_backward_time: 653.8747558594, weight_update_time: 0.2866479754, other_time: 9.5433349609
epoch time: 88.4051513672s, input_fired_time: 0.9030858278, output_fired_time: 0.0000000000, propagate_forward_time: 85.8831481934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6189193726
validation[          , genome  5896] predictions:    6156/  10000 (61.56%), best:    6129/10000 (61.29%), error:     11093.63770, best error:     11093.11914 on epoch:    45, epoch:   48/50, mu: 0.9267066717, learning_rate: 0.0000630110, weight_decay: 0.0000255467

epoch time: 1113.4338378906s, input_fired_time: 11.7947549820, output_fired_time: 5.8320698738, propagate_forward_time: 430.0291442871, propagate_backward_time: 656.2395629883, weight_update_time: 0.2862895131, other_time: 9.5382690430
epoch time: 88.5050811768s, input_fired_time: 0.9043323398, output_fired_time: 0.0000000000, propagate_forward_time: 85.9815444946, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6192016602
validation[          , genome  5896] predictions:    6147/  10000 (61.47%), best:    6129/10000 (61.29%), error:     11220.22949, best error:     11093.11914 on epoch:    45, epoch:   49/50, mu: 0.9293757677, learning_rate: 0.0000580711, weight_decay: 0.0000239007

epoch time: 1123.0903320313s, input_fired_time: 11.8613796234, output_fired_time: 5.8502745628, propagate_forward_time: 434.3436889648, propagate_backward_time: 661.3796386719, weight_update_time: 0.2861563563, other_time: 9.6554565430
epoch time: 89.4136352539s, input_fired_time: 0.9090079069, output_fired_time: 0.0000000000, propagate_forward_time: 86.8734893799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6311416626
validation[          , genome  5896] predictions:    6108/  10000 (61.08%), best:    6129/10000 (61.29%), error:     11132.91113, best error:     11093.11914 on epoch:    45, epoch:   50/50, mu: 0.9319323301, learning_rate: 0.0000535184, weight_decay: 0.0000223608

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 447.8065185547s, input_fired_time: 4.5759406090, output_fired_time: 0.0000000000, propagate_forward_time: 434.9212646484, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3093261719
best training[          , genome  5896] predictions:   33604/  50000 (67.21%), best:    6129/10000 (61.29%), error:     48675.60547, best error:     11093.11914 on epoch:    45, epoch:   51/50, mu: 0.9343810678, learning_rate: 0.0000493227, weight_decay: 0.0000209200
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 93.4933471680s, input_fired_time: 0.9755468369, output_fired_time: 0.0000000000, propagate_forward_time: 90.8281860352, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6896133423
   testing[          , genome  5896] predictions:    6129/  10000 (61.29%), best:    6129/10000 (61.29%), error:     11093.11914, best error:     11093.11914 on epoch:    45, epoch:   51/50, mu: 0.9343810678, learning_rate: 0.0000493227, weight_decay: 0.0000209200
backpropagation finished successfully!
17:10:39 (5784): called boinc_finish(0)

</stderr_txt>
]]>