Nameexact_genome_1542531604_178_6271_3
Workunit4097836
Created18 Nov 2018, 18:15:06 UTC
Sent18 Nov 2018, 18:56:24 UTC
Report deadline25 Jan 2019, 3:51:21 UTC
Received21 Nov 2018, 0:22:18 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID80138
Run time17 hours 8 min 29 sec
CPU time16 hours 58 min 4 sec
Validate stateValid
Credit16,842.88
Device peak FLOPS4.49 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size562.90 MB
Peak swap size582.93 MB
Peak disk usage14.79 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_178_6271.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_178_6271_3_r960532313_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542531604_178_6271.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.48838
read mu: 0.48838
read mu_delta: 0.952441
read initial_learning_rate: 0.000439411
read learning_rate: 0.000439411
read learning_rate_delta: 0.922662
read initial_weight_decay: 0.000549118
read weight_decay: 0.000549118
read weight_decay_delta: 0.924184
read batch_size: 106
read epsilon: 1e-07
read alpha: 0.076575
read input_dropout_probability: 0.00118668
read hidden_dropout_probability: 0.110619
read velocity_reset: 2318
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6271
read normal distribution: '1 1.28742301464081 -1.10027384757996'
generator_str: '1905782696'
read generator: 1905782696
read generated_by_map:
1 crossover 1
reading 107 nodes.
reading 556 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 771176667
post shuffle 1: 728404878
epoch time: 1121.21850585938s, input_fired_time: 14.48619556427, output_fired_time: 4.79065752029419, propagate_forward_time: 330.9921875, propagate_backward_time: 762.878662109375, weight_update_time: 0.247519135475159, other_time: 8.07080078125
epoch time: 68.5316619873047s, input_fired_time: 0.850381672382355, output_fired_time: 0, propagate_forward_time: 66.4756927490234, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.20558929443359
validation[          , genome  6271] predictions:    6039/  10000 (60.39%), best:    6039/10000 (60.39%), error:     11417.39063, best error:     11417.39063 on epoch:     0, epoch:    0/50, mu: 0.4883801341, learning_rate: 0.0004394107, weight_decay: 0.0005491177

epoch time: 1120.8721923828s, input_fired_time: 14.4433813095, output_fired_time: 4.7817125320, propagate_forward_time: 330.9432678223, propagate_backward_time: 762.4634399414, weight_update_time: 0.2482492924, other_time: 8.2404174805
epoch time: 68.4940109253s, input_fired_time: 0.8492853045, output_fired_time: 0.0000000000, propagate_forward_time: 66.4378509521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2068710327
validation[          , genome  6271] predictions:    6040/  10000 (60.40%), best:    6039/10000 (60.39%), error:     11453.63477, best error:     11417.39063 on epoch:     0, epoch:    1/50, mu: 0.5122368932, learning_rate: 0.0004054275, weight_decay: 0.0005074858

epoch time: 1120.5028076172s, input_fired_time: 14.4180030823, output_fired_time: 4.7801990509, propagate_forward_time: 330.8973693848, propagate_backward_time: 762.1416015625, weight_update_time: 0.2476511300, other_time: 8.2657470703
epoch time: 68.7670059204s, input_fired_time: 0.8510386348, output_fired_time: 0.0000000000, propagate_forward_time: 66.7036285400, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2123413086
validation[          , genome  6271] predictions:    5965/  10000 (59.65%), best:    6039/10000 (60.39%), error:     11686.21387, best error:     11417.39063 on epoch:     0, epoch:    2/50, mu: 0.5349590182, learning_rate: 0.0003740726, weight_decay: 0.0004690102

epoch time: 1121.0010986328s, input_fired_time: 14.4150466919, output_fired_time: 4.7895388603, propagate_forward_time: 331.1000061035, propagate_backward_time: 762.3781127930, weight_update_time: 0.2480534613, other_time: 8.3184204102
epoch time: 68.4952087402s, input_fired_time: 0.8421236873, output_fired_time: 0.0000000000, propagate_forward_time: 66.4470672607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2060165405
validation[          , genome  6271] predictions:    5957/  10000 (59.57%), best:    6039/10000 (60.39%), error:     11752.76855, best error:     11417.39063 on epoch:     0, epoch:    3/50, mu: 0.5566005111, learning_rate: 0.0003451425, weight_decay: 0.0004334517

epoch time: 1120.7420654297s, input_fired_time: 14.3681554794, output_fired_time: 4.7870712280, propagate_forward_time: 330.7534484863, propagate_backward_time: 762.5180053711, weight_update_time: 0.2485468686, other_time: 8.3153686523
epoch time: 68.6112060547s, input_fired_time: 0.8371113539, output_fired_time: 0.0000000000, propagate_forward_time: 66.5759658813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1981277466
validation[          , genome  6271] predictions:    6011/  10000 (60.11%), best:    6039/10000 (60.39%), error:     11797.77637, best error:     11417.39063 on epoch:     0, epoch:    4/50, mu: 0.5772127509, learning_rate: 0.0003184499, weight_decay: 0.0004005891

epoch time: 1122.3823242188s, input_fired_time: 14.3525257111, output_fired_time: 4.7865495682, propagate_forward_time: 331.4236145020, propagate_backward_time: 763.5388183594, weight_update_time: 0.2493022978, other_time: 8.2808837891
epoch time: 69.4755096436s, input_fired_time: 0.8380094767, output_fired_time: 0.0000000000, propagate_forward_time: 66.6470718384, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9904251099
validation[          , genome  6271] predictions:    5916/  10000 (59.16%), best:    6039/10000 (60.39%), error:     11941.42480, best error:     11417.39063 on epoch:     0, epoch:    5/50, mu: 0.5968446732, learning_rate: 0.0002938216, weight_decay: 0.0003702180

epoch time: 1120.9487304688s, input_fired_time: 14.3358049393, output_fired_time: 4.7881999016, propagate_forward_time: 330.9258728027, propagate_backward_time: 762.3462524414, weight_update_time: 0.2494869232, other_time: 8.5525512695
epoch time: 69.3709640503s, input_fired_time: 0.8308233619, output_fired_time: 0.0000000000, propagate_forward_time: 66.5364913940, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0036468506
validation[          , genome  6271] predictions:    5996/  10000 (59.96%), best:    6039/10000 (60.39%), error:     11766.80957, best error:     11417.39063 on epoch:     0, epoch:    6/50, mu: 0.6155429482, learning_rate: 0.0002710980, weight_decay: 0.0003421495

epoch time: 1120.8277587891s, input_fired_time: 14.3242406845, output_fired_time: 4.7820572853, propagate_forward_time: 330.9372863770, propagate_backward_time: 762.5056152344, weight_update_time: 0.2470325381, other_time: 8.2785644531
epoch time: 68.5330657959s, input_fired_time: 0.8231218457, output_fired_time: 0.0000000000, propagate_forward_time: 66.4897308350, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2202148438
validation[          , genome  6271] predictions:    5509/  10000 (55.09%), best:    6039/10000 (60.39%), error:     12972.98047, best error:     11417.39063 on epoch:     0, epoch:    7/50, mu: 0.6333519220, learning_rate: 0.0002501318, weight_decay: 0.0003162090

epoch time: 1119.9337158203s, input_fired_time: 14.3086662292, output_fired_time: 4.7866139412, propagate_forward_time: 330.7559509277, propagate_backward_time: 761.9914550781, weight_update_time: 0.2479882836, other_time: 8.0909423828
epoch time: 68.9250411987s, input_fired_time: 0.8265656829, output_fired_time: 0.0000000000, propagate_forward_time: 66.9037475586, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1947250366
validation[          , genome  6271] predictions:    5742/  10000 (57.42%), best:    6039/10000 (60.39%), error:     12180.33105, best error:     11417.39063 on epoch:     0, epoch:    8/50, mu: 0.6503139138, learning_rate: 0.0002307871, weight_decay: 0.0002922353

epoch time: 1120.6801757813s, input_fired_time: 14.3338499069, output_fired_time: 4.7811665535, propagate_forward_time: 331.0071716309, propagate_backward_time: 762.5345458984, weight_update_time: 0.2495477647, other_time: 8.0234375000
epoch time: 68.5256118774s, input_fired_time: 0.8295731544, output_fired_time: 0.0000000000, propagate_forward_time: 66.4879226685, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2081146240
validation[          , genome  6271] predictions:    5776/  10000 (57.76%), best:    6039/10000 (60.39%), error:     12231.65039, best error:     11417.39063 on epoch:     0, epoch:    9/50, mu: 0.6664692163, learning_rate: 0.0002129385, weight_decay: 0.0002700792

epoch time: 1120.4809570313s, input_fired_time: 14.3309679031, output_fired_time: 4.7842063904, propagate_forward_time: 330.9723510742, propagate_backward_time: 762.4070434570, weight_update_time: 0.2499211580, other_time: 7.9864501953
epoch time: 68.7410812378s, input_fired_time: 0.8296806216, output_fired_time: 0.0000000000, propagate_forward_time: 66.6885833740, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2228164673
validation[          , genome  6271] predictions:    6008/  10000 (60.08%), best:    6039/10000 (60.39%), error:     11766.82031, best error:     11417.39063 on epoch:     0, epoch:   10/50, mu: 0.6818561554, learning_rate: 0.0001964702, weight_decay: 0.0002496028

epoch time: 1121.0347900391s, input_fired_time: 14.3390789032, output_fired_time: 4.7900710106, propagate_forward_time: 331.1392822266, propagate_backward_time: 762.7257080078, weight_update_time: 0.2491964996, other_time: 8.0406494141
epoch time: 68.5811462402s, input_fired_time: 0.8312866688, output_fired_time: 0.0000000000, propagate_forward_time: 66.5531158447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1967468262
validation[          , genome  6271] predictions:    5835/  10000 (58.35%), best:    6039/10000 (60.39%), error:     12050.79785, best error:     11417.39063 on epoch:     0, epoch:   11/50, mu: 0.6965113282, learning_rate: 0.0001812756, weight_decay: 0.0002306789

epoch time: 1120.8082275391s, input_fired_time: 14.3486013412, output_fired_time: 4.7977061272, propagate_forward_time: 331.1689453125, propagate_backward_time: 762.3726806641, weight_update_time: 0.2484437823, other_time: 8.1202392578
epoch time: 68.6797714233s, input_fired_time: 0.8319745064, output_fired_time: 0.0000000000, propagate_forward_time: 66.6473083496, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2004852295
validation[          , genome  6271] predictions:    5992/  10000 (59.92%), best:    6039/10000 (60.39%), error:     11633.66113, best error:     11417.39063 on epoch:     0, epoch:   12/50, mu: 0.7104694843, learning_rate: 0.0001672561, weight_decay: 0.0002131897

epoch time: 1120.7196044922s, input_fired_time: 14.3271160126, output_fired_time: 4.7809953690, propagate_forward_time: 331.0903320313, propagate_backward_time: 762.3986816406, weight_update_time: 0.2467523068, other_time: 8.1224365234
epoch time: 68.6782302856s, input_fired_time: 0.8310689926, output_fired_time: 0.0000000000, propagate_forward_time: 66.6469726563, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2001876831
validation[          , genome  6271] predictions:    6039/  10000 (60.39%), best:    6039/10000 (60.39%), error:     11646.20020, best error:     11417.39063 on epoch:     0, epoch:   13/50, mu: 0.7237638235, learning_rate: 0.0001543208, weight_decay: 0.0001970265

epoch time: 1120.9041748047s, input_fired_time: 14.3354167938, output_fired_time: 4.7920126915, propagate_forward_time: 331.0992126465, propagate_backward_time: 762.5121459961, weight_update_time: 0.2473063171, other_time: 8.1653442383
epoch time: 68.7153244019s, input_fired_time: 0.8290410042, output_fired_time: 0.0000000000, propagate_forward_time: 66.6883773804, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1979064941
validation[          , genome  6271] predictions:    5973/  10000 (59.73%), best:    6039/10000 (60.39%), error:     11739.33887, best error:     11417.39063 on epoch:     0, epoch:   14/50, mu: 0.7364258766, learning_rate: 0.0001423860, weight_decay: 0.0001820887

epoch time: 1127.7164306641s, input_fired_time: 14.4850854874, output_fired_time: 4.8221206665, propagate_forward_time: 333.3223266602, propagate_backward_time: 766.5313720703, weight_update_time: 0.2501514852, other_time: 8.5554809570
epoch time: 70.5291976929s, input_fired_time: 0.8508079648, output_fired_time: 0.0000000000, propagate_forward_time: 67.7002029419, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9781875610
validation[          , genome  6271] predictions:    5967/  10000 (59.67%), best:    6039/10000 (60.39%), error:     11640.93555, best error:     11417.39063 on epoch:     0, epoch:   15/50, mu: 0.7484857440, learning_rate: 0.0001313741, weight_decay: 0.0001682835

epoch time: 1133.8956298828s, input_fired_time: 14.4864444733, output_fired_time: 4.8789267540, propagate_forward_time: 334.5140075684, propagate_backward_time: 771.9843139648, weight_update_time: 0.2531121075, other_time: 8.0319213867
epoch time: 69.7514801025s, input_fired_time: 0.8506133556, output_fired_time: 0.0000000000, propagate_forward_time: 67.6680526733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2328109741
validation[          , genome  6271] predictions:    6130/  10000 (61.30%), best:    6130/10000 (61.30%), error:     11307.62988, best error:     11307.62988 on epoch:    16, epoch:   16/50, mu: 0.7599720359, learning_rate: 0.0001212139, weight_decay: 0.0001555249

epoch time: 1139.9810791016s, input_fired_time: 14.5869178772, output_fired_time: 4.9078917503, propagate_forward_time: 337.0403137207, propagate_backward_time: 775.8184814453, weight_update_time: 0.2529295981, other_time: 7.6275634766
epoch time: 74.1982116699s, input_fired_time: 0.9550299644, output_fired_time: 0.0000000000, propagate_forward_time: 71.8853759766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3578033447
validation[          , genome  6271] predictions:    6134/  10000 (61.34%), best:    6134/10000 (61.34%), error:     11239.82715, best error:     11239.82715 on epoch:    17, epoch:   17/50, mu: 0.7709120512, learning_rate: 0.0001118395, weight_decay: 0.0001437336

epoch time: 1132.7607421875s, input_fired_time: 14.5004272461, output_fired_time: 4.8624815941, propagate_forward_time: 334.2492675781, propagate_backward_time: 771.3465576172, weight_update_time: 0.2840625048, other_time: 7.8020019531
epoch time: 69.6350250244s, input_fired_time: 0.8517140150, output_fired_time: 0.0000000000, propagate_forward_time: 67.5523605347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2309494019
validation[          , genome  6271] predictions:    6171/  10000 (61.71%), best:    6134/10000 (61.34%), error:     11289.36230, best error:     11239.82715 on epoch:    17, epoch:   18/50, mu: 0.7813317776, learning_rate: 0.0001031900, weight_decay: 0.0001328363

epoch time: 1133.0086669922s, input_fired_time: 14.5248661041, output_fired_time: 4.8669424057, propagate_forward_time: 334.7270202637, propagate_backward_time: 771.3709716797, weight_update_time: 0.2518503368, other_time: 7.5187988281
epoch time: 69.1031341553s, input_fired_time: 0.8451578021, output_fired_time: 0.0000000000, propagate_forward_time: 67.0275268555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2304458618
validation[          , genome  6271] predictions:    6168/  10000 (61.68%), best:    6168/10000 (61.68%), error:     11181.92871, best error:     11181.92871 on epoch:    19, epoch:   19/50, mu: 0.7912559509, learning_rate: 0.0000952095, weight_decay: 0.0001227652

epoch time: 1163.2025146484s, input_fired_time: 15.0142774582, output_fired_time: 5.0213809013, propagate_forward_time: 343.8307495117, propagate_backward_time: 791.4443359375, weight_update_time: 0.2617121041, other_time: 7.8917846680
epoch time: 69.1902084351s, input_fired_time: 0.8462120891, output_fired_time: 0.0000000000, propagate_forward_time: 67.1145324707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2294616699
validation[          , genome  6271] predictions:    6047/  10000 (60.47%), best:    6168/10000 (61.68%), error:     11411.80273, best error:     11181.92871 on epoch:    19, epoch:   20/50, mu: 0.8007081151, learning_rate: 0.0000878462, weight_decay: 0.0001134576

epoch time: 1136.4831542969s, input_fired_time: 14.5578403473, output_fired_time: 4.8850388527, propagate_forward_time: 335.2333068848, propagate_backward_time: 774.1299438477, weight_update_time: 0.2535633743, other_time: 7.6770629883
epoch time: 69.0911407471s, input_fired_time: 0.8488713503, output_fired_time: 0.0000000000, propagate_forward_time: 66.9642562866, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2780151367
validation[          , genome  6271] predictions:    6219/  10000 (62.19%), best:    6219/10000 (62.19%), error:     11003.77930, best error:     11003.77930 on epoch:    21, epoch:   21/50, mu: 0.8097107410, learning_rate: 0.0000810523, weight_decay: 0.0001048557

epoch time: 1132.0283203125s, input_fired_time: 14.6704797745, output_fired_time: 4.8626632690, propagate_forward_time: 335.2512512207, propagate_backward_time: 769.6837158203, weight_update_time: 0.2520980537, other_time: 7.5601806641
epoch time: 69.3120498657s, input_fired_time: 0.8440371752, output_fired_time: 0.0000000000, propagate_forward_time: 67.2436370850, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2243728638
validation[          , genome  6271] predictions:    6079/  10000 (60.79%), best:    6219/10000 (62.19%), error:     11342.11914, best error:     11003.77930 on epoch:    21, epoch:   22/50, mu: 0.8182852268, learning_rate: 0.0000747839, weight_decay: 0.0000969059

epoch time: 1130.9963378906s, input_fired_time: 14.5056982040, output_fired_time: 4.8661012650, propagate_forward_time: 334.0876159668, propagate_backward_time: 769.7971191406, weight_update_time: 0.2550663352, other_time: 7.7397460938
epoch time: 69.4729766846s, input_fired_time: 0.8520897627, output_fired_time: 0.0000000000, propagate_forward_time: 67.3853378296, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2355499268
validation[          , genome  6271] predictions:    6273/  10000 (62.73%), best:    6273/10000 (62.73%), error:     10945.55176, best error:     10945.55176 on epoch:    23, epoch:   23/50, mu: 0.8264518976, learning_rate: 0.0000690003, weight_decay: 0.0000895589

epoch time: 1133.3068847656s, input_fired_time: 14.5197191238, output_fired_time: 4.8718066216, propagate_forward_time: 334.4314270020, propagate_backward_time: 771.7551269531, weight_update_time: 0.2534364164, other_time: 7.7287597656
epoch time: 69.2873229980s, input_fired_time: 0.8523158431, output_fired_time: 0.0000000000, propagate_forward_time: 67.1914901733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2435150146
validation[          , genome  6271] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10835.88184, best error:     10835.88184 on epoch:    24, epoch:   24/50, mu: 0.8342301250, learning_rate: 0.0000636639, weight_decay: 0.0000827689

epoch time: 1131.4810791016s, input_fired_time: 14.5185728073, output_fired_time: 4.8846817017, propagate_forward_time: 333.9938964844, propagate_backward_time: 770.0565795898, weight_update_time: 0.2523910105, other_time: 8.0274047852
epoch time: 69.4250946045s, input_fired_time: 0.8560269475, output_fired_time: 0.0000000000, propagate_forward_time: 67.3293380737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2397308350
validation[          , genome  6271] predictions:    6174/  10000 (61.74%), best:    6317/10000 (63.17%), error:     11111.82324, best error:     10835.88184 on epoch:    24, epoch:   25/50, mu: 0.8416384459, learning_rate: 0.0000587403, weight_decay: 0.0000764937

epoch time: 1133.6799316406s, input_fired_time: 14.5887947083, output_fired_time: 4.8805284500, propagate_forward_time: 334.9328918457, propagate_backward_time: 771.6835327148, weight_update_time: 0.2554725111, other_time: 7.5942993164
epoch time: 70.3332672119s, input_fired_time: 0.8519198298, output_fired_time: 0.0000000000, propagate_forward_time: 68.2136688232, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2676773071
validation[          , genome  6271] predictions:    6315/  10000 (63.15%), best:    6315/10000 (63.15%), error:     10817.09766, best error:     10817.09766 on epoch:    26, epoch:   26/50, mu: 0.8486944437, learning_rate: 0.0000541974, weight_decay: 0.0000706942

epoch time: 1133.9476318359s, input_fired_time: 14.5398569107, output_fired_time: 4.8609380722, propagate_forward_time: 335.2952270508, propagate_backward_time: 771.5816040039, weight_update_time: 0.2523430586, other_time: 7.6699218750
epoch time: 68.9411926270s, input_fired_time: 0.8470886946, output_fired_time: 0.0000000000, propagate_forward_time: 66.8646697998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2294311523
validation[          , genome  6271] predictions:    6368/  10000 (63.68%), best:    6368/10000 (63.68%), error:     10760.41504, best error:     10760.41504 on epoch:    27, epoch:   27/50, mu: 0.8554148674, learning_rate: 0.0000500059, weight_decay: 0.0000653345

epoch time: 1134.8487548828s, input_fired_time: 14.6016044617, output_fired_time: 4.8834595680, propagate_forward_time: 335.8747863770, propagate_backward_time: 771.5394287109, weight_update_time: 0.2519821227, other_time: 7.9495849609
epoch time: 69.0505142212s, input_fired_time: 0.8516122103, output_fired_time: 0.0000000000, propagate_forward_time: 66.9723510742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2265472412
validation[          , genome  6271] predictions:    6336/  10000 (63.36%), best:    6368/10000 (63.68%), error:     10782.47363, best error:     10760.41504 on epoch:    27, epoch:   28/50, mu: 0.8618156314, learning_rate: 0.0000461385, weight_decay: 0.0000603811

epoch time: 1135.0418701172s, input_fired_time: 14.5439414978, output_fired_time: 4.8770146370, propagate_forward_time: 334.4889221191, propagate_backward_time: 773.6344604492, weight_update_time: 0.2533168197, other_time: 7.4974975586
epoch time: 69.0022583008s, input_fired_time: 0.8504480124, output_fired_time: 0.0000000000, propagate_forward_time: 66.9273834229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2244262695
validation[          , genome  6271] predictions:    6354/  10000 (63.54%), best:    6354/10000 (63.54%), error:     10676.21680, best error:     10676.21680 on epoch:    29, epoch:   29/50, mu: 0.8679119945, learning_rate: 0.0000425703, weight_decay: 0.0000558032

epoch time: 1143.5349121094s, input_fired_time: 14.7193994522, output_fired_time: 4.9281387329, propagate_forward_time: 337.8894348145, propagate_backward_time: 778.3782348633, weight_update_time: 0.2590156794, other_time: 7.6198120117
epoch time: 69.3733444214s, input_fired_time: 0.8778269291, output_fired_time: 0.0000000000, propagate_forward_time: 67.2585983276, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2369155884
validation[          , genome  6271] predictions:    6348/  10000 (63.48%), best:    6348/10000 (63.48%), error:     10650.43164, best error:     10650.43164 on epoch:    30, epoch:   30/50, mu: 0.8737184405, learning_rate: 0.0000392780, weight_decay: 0.0000515724

epoch time: 1135.1154785156s, input_fired_time: 14.6497249603, output_fired_time: 4.9053401947, propagate_forward_time: 334.6536254883, propagate_backward_time: 773.3096313477, weight_update_time: 0.2573686838, other_time: 7.5970458984
epoch time: 69.3791961670s, input_fired_time: 0.8575589657, output_fired_time: 0.0000000000, propagate_forward_time: 67.2880325317, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2336044312
validation[          , genome  6271] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10552.73535, best error:     10552.73535 on epoch:    31, epoch:   31/50, mu: 0.8792487383, learning_rate: 0.0000362403, weight_decay: 0.0000476624

epoch time: 1138.3997802734s, input_fired_time: 14.6319932938, output_fired_time: 4.9148116112, propagate_forward_time: 335.9566345215, propagate_backward_time: 774.8519287109, weight_update_time: 0.2634149790, other_time: 8.0444335938
epoch time: 70.9920959473s, input_fired_time: 0.8807975054, output_fired_time: 0.0000000000, propagate_forward_time: 68.7062454224, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4050521851
validation[          , genome  6271] predictions:    6278/  10000 (62.78%), best:    6388/10000 (63.88%), error:     10724.73047, best error:     10552.73535 on epoch:    31, epoch:   32/50, mu: 0.8845160007, learning_rate: 0.0000334375, weight_decay: 0.0000440488

epoch time: 1135.9799804688s, input_fired_time: 14.6595563889, output_fired_time: 4.9143257141, propagate_forward_time: 336.1012573242, propagate_backward_time: 772.6856079102, weight_update_time: 0.2540805638, other_time: 7.6192626953
epoch time: 69.4794082642s, input_fired_time: 0.8609016538, output_fired_time: 0.0000000000, propagate_forward_time: 67.3921356201, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2263717651
validation[          , genome  6271] predictions:    6379/  10000 (63.79%), best:    6388/10000 (63.88%), error:     10662.76953, best error:     10552.73535 on epoch:    31, epoch:   33/50, mu: 0.8895327449, learning_rate: 0.0000308515, weight_decay: 0.0000407092

epoch time: 1131.2049560547s, input_fired_time: 14.5225086212, output_fired_time: 4.8512482643, propagate_forward_time: 333.8235778809, propagate_backward_time: 770.5517578125, weight_update_time: 0.2530213594, other_time: 7.4559326172
epoch time: 69.3258438110s, input_fired_time: 0.8589267135, output_fired_time: 0.0000000000, propagate_forward_time: 67.2357177734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2312011719
validation[          , genome  6271] predictions:    6395/  10000 (63.95%), best:    6388/10000 (63.88%), error:     10605.15625, best error:     10552.73535 on epoch:    31, epoch:   34/50, mu: 0.8943108916, learning_rate: 0.0000284655, weight_decay: 0.0000376228

epoch time: 1132.1140136719s, input_fired_time: 14.5396690369, output_fired_time: 4.8602905273, propagate_forward_time: 334.0550231934, propagate_backward_time: 771.1650390625, weight_update_time: 0.2570291460, other_time: 7.4940185547
epoch time: 69.2599411011s, input_fired_time: 0.8617133498, output_fired_time: 0.0000000000, propagate_forward_time: 67.1689300537, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2293014526
validation[          , genome  6271] predictions:    6391/  10000 (63.91%), best:    6388/10000 (63.88%), error:     10605.82324, best error:     10552.73535 on epoch:    31, epoch:   35/50, mu: 0.8988618255, learning_rate: 0.0000262641, weight_decay: 0.0000347704

epoch time: 1135.2957763672s, input_fired_time: 14.7309970856, output_fired_time: 4.8963761330, propagate_forward_time: 336.0173034668, propagate_backward_time: 771.6181640625, weight_update_time: 0.2520302236, other_time: 8.0329589844
epoch time: 69.5976333618s, input_fired_time: 0.8819971681, output_fired_time: 0.0000000000, propagate_forward_time: 67.4753341675, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2403030396
validation[          , genome  6271] predictions:    6351/  10000 (63.51%), best:    6388/10000 (63.88%), error:     10574.71094, best error:     10552.73535 on epoch:    31, epoch:   36/50, mu: 0.9031963348, learning_rate: 0.0000242329, weight_decay: 0.0000321342

epoch time: 1131.5430908203s, input_fired_time: 14.6379604340, output_fired_time: 4.8727974892, propagate_forward_time: 334.6123962402, propagate_backward_time: 769.7874755859, weight_update_time: 0.2583009005, other_time: 7.6324462891
epoch time: 72.6968536377s, input_fired_time: 0.9305959344, output_fired_time: 0.0000000000, propagate_forward_time: 70.4879989624, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2782592773
validation[          , genome  6271] predictions:    6340/  10000 (63.40%), best:    6388/10000 (63.88%), error:     10567.32520, best error:     10552.73535 on epoch:    31, epoch:   37/50, mu: 0.9073246717, learning_rate: 0.0000223587, weight_decay: 0.0000296979

epoch time: 1140.1204833984s, input_fired_time: 14.7147283554, output_fired_time: 4.9013237953, propagate_forward_time: 337.4645996094, propagate_backward_time: 775.4344482422, weight_update_time: 0.2550171316, other_time: 7.6053466797
epoch time: 69.4642181396s, input_fired_time: 0.8564613461, output_fired_time: 0.0000000000, propagate_forward_time: 67.3785858154, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2291717529
validation[          , genome  6271] predictions:    6386/  10000 (63.86%), best:    6388/10000 (63.88%), error:     10608.04492, best error:     10552.73535 on epoch:    31, epoch:   38/50, mu: 0.9112566710, learning_rate: 0.0000206296, weight_decay: 0.0000274464

epoch time: 1136.2266845703s, input_fired_time: 14.6506910324, output_fired_time: 4.8843297958, propagate_forward_time: 335.7433776855, propagate_backward_time: 773.2478637695, weight_update_time: 0.2564928830, other_time: 7.7005004883
epoch time: 69.0351486206s, input_fired_time: 0.8597472310, output_fired_time: 0.0000000000, propagate_forward_time: 66.9521255493, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2232742310
validation[          , genome  6271] predictions:    6473/  10000 (64.73%), best:    6473/10000 (64.73%), error:     10299.73828, best error:     10299.73828 on epoch:    39, epoch:   39/50, mu: 0.9150016308, learning_rate: 0.0000190341, weight_decay: 0.0000253655

epoch time: 1132.2789306641s, input_fired_time: 14.6062507629, output_fired_time: 4.8619132042, propagate_forward_time: 334.5932006836, propagate_backward_time: 770.5953369141, weight_update_time: 0.2528315485, other_time: 7.6222534180
epoch time: 69.2722702026s, input_fired_time: 0.8626487255, output_fired_time: 0.0000000000, propagate_forward_time: 67.1820373535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2275848389
validation[          , genome  6271] predictions:    6452/  10000 (64.52%), best:    6473/10000 (64.73%), error:     10423.28613, best error:     10299.73828 on epoch:    39, epoch:   40/50, mu: 0.9185684919, learning_rate: 0.0000175620, weight_decay: 0.0000234424

epoch time: 1133.0389404297s, input_fired_time: 14.6599426270, output_fired_time: 4.8581171036, propagate_forward_time: 334.6781005859, propagate_backward_time: 771.2590942383, weight_update_time: 0.2508561909, other_time: 7.5836791992
epoch time: 70.7732849121s, input_fired_time: 0.8766445518, output_fired_time: 0.0000000000, propagate_forward_time: 68.6288986206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2677383423
validation[          , genome  6271] predictions:    6498/  10000 (64.98%), best:    6498/10000 (64.98%), error:     10212.02832, best error:     10212.02832 on epoch:    41, epoch:   41/50, mu: 0.9219657183, learning_rate: 0.0000162038, weight_decay: 0.0000216651

epoch time: 1135.1125488281s, input_fired_time: 14.6624059677, output_fired_time: 4.8920903206, propagate_forward_time: 335.6116943359, propagate_backward_time: 772.4097290039, weight_update_time: 0.2654428184, other_time: 7.5366821289
epoch time: 68.7025222778s, input_fired_time: 0.8593457937, output_fired_time: 0.0000000000, propagate_forward_time: 66.6446075439, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1985702515
validation[          , genome  6271] predictions:    6548/  10000 (65.48%), best:    6548/10000 (65.48%), error:     10146.51367, best error:     10146.51367 on epoch:    42, epoch:   42/50, mu: 0.9252014160, learning_rate: 0.0000149507, weight_decay: 0.0000200225

epoch time: 1142.2761230469s, input_fired_time: 14.5949201584, output_fired_time: 4.8561458588, propagate_forward_time: 335.8085021973, propagate_backward_time: 772.2842407227, weight_update_time: 0.2489738017, other_time: 14.7322387695
epoch time: 93.9129104614s, input_fired_time: 1.0535370111, output_fired_time: 0.0000000000, propagate_forward_time: 85.3345336914, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5248413086
validation[          , genome  6271] predictions:    6529/  10000 (65.29%), best:    6548/10000 (65.48%), error:     10235.00488, best error:     10146.51367 on epoch:    42, epoch:   43/50, mu: 0.9282832146, learning_rate: 0.0000137944, weight_decay: 0.0000185045

epoch time: 1139.0032958984s, input_fired_time: 14.7385063171, output_fired_time: 4.8527207375, propagate_forward_time: 335.3526000977, propagate_backward_time: 772.3953857422, weight_update_time: 0.2526824772, other_time: 11.6641235352
epoch time: 68.9042358398s, input_fired_time: 0.8556277156, output_fired_time: 0.0000000000, propagate_forward_time: 66.8190078735, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2295989990
validation[          , genome  6271] predictions:    6523/  10000 (65.23%), best:    6548/10000 (65.48%), error:     10256.59668, best error:     10146.51367 on epoch:    42, epoch:   44/50, mu: 0.9312184453, learning_rate: 0.0000127276, weight_decay: 0.0000171015

epoch time: 1120.0111083984s, input_fired_time: 14.4434833527, output_fired_time: 4.7876939774, propagate_forward_time: 330.7677612305, propagate_backward_time: 762.1326293945, weight_update_time: 0.2487605214, other_time: 7.8795166016
epoch time: 68.5272827148s, input_fired_time: 0.8533355594, output_fired_time: 0.0000000000, propagate_forward_time: 66.4804840088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1934661865
validation[          , genome  6271] predictions:    6508/  10000 (65.08%), best:    6548/10000 (65.48%), error:     10221.76855, best error:     10146.51367 on epoch:    42, epoch:   45/50, mu: 0.9340140820, learning_rate: 0.0000117432, weight_decay: 0.0000158050

epoch time: 1119.5930175781s, input_fired_time: 14.4540710449, output_fired_time: 4.8057632446, propagate_forward_time: 330.5883483887, propagate_backward_time: 761.8486938477, weight_update_time: 0.2491331697, other_time: 7.8960571289
epoch time: 68.5397644043s, input_fired_time: 0.8560224771, output_fired_time: 0.0000000000, propagate_forward_time: 66.4776687622, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2060699463
validation[          , genome  6271] predictions:    6567/  10000 (65.67%), best:    6567/10000 (65.67%), error:     10143.68164, best error:     10143.68164 on epoch:    46, epoch:   46/50, mu: 0.9366767406, learning_rate: 0.0000108350, weight_decay: 0.0000146067

epoch time: 1118.6367187500s, input_fired_time: 14.4359683990, output_fired_time: 4.7930536270, propagate_forward_time: 330.3942260742, propagate_backward_time: 761.1353149414, weight_update_time: 0.2476218045, other_time: 7.8781738281
epoch time: 68.5382537842s, input_fired_time: 0.8549295664, output_fired_time: 0.0000000000, propagate_forward_time: 66.4771575928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2061691284
validation[          , genome  6271] predictions:    6511/  10000 (65.11%), best:    6511/10000 (65.11%), error:     10133.00684, best error:     10133.00684 on epoch:    47, epoch:   47/50, mu: 0.9392127395, learning_rate: 0.0000099971, weight_decay: 0.0000134993

epoch time: 1120.0061035156s, input_fired_time: 14.4604005814, output_fired_time: 4.7797265053, propagate_forward_time: 330.9567871094, propagate_backward_time: 761.9313964844, weight_update_time: 0.2472663820, other_time: 7.8776855469
epoch time: 68.6309890747s, input_fired_time: 0.8590382338, output_fired_time: 0.0000000000, propagate_forward_time: 66.5632247925, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2087249756
validation[          , genome  6271] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:     10036.22754, best error:     10036.22754 on epoch:    48, epoch:   48/50, mu: 0.9416281581, learning_rate: 0.0000092239, weight_decay: 0.0000124758

epoch time: 1119.5845947266s, input_fired_time: 14.4432611465, output_fired_time: 4.7841773033, propagate_forward_time: 330.6858520508, propagate_backward_time: 761.8142700195, weight_update_time: 0.2476998717, other_time: 7.8570556641
epoch time: 68.4933700562s, input_fired_time: 0.8571339250, output_fired_time: 0.0000000000, propagate_forward_time: 66.4225311279, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2137069702
validation[          , genome  6271] predictions:    6550/  10000 (65.50%), best:    6571/10000 (65.71%), error:     10050.71094, best error:     10036.22754 on epoch:    48, epoch:   49/50, mu: 0.9439287186, learning_rate: 0.0000085106, weight_decay: 0.0000115299

epoch time: 1119.3752441406s, input_fired_time: 14.4444990158, output_fired_time: 4.7815809250, propagate_forward_time: 330.5709838867, propagate_backward_time: 761.7026367188, weight_update_time: 0.2478913963, other_time: 7.8755493164
epoch time: 68.4346237183s, input_fired_time: 0.8573033214, output_fired_time: 0.0000000000, propagate_forward_time: 66.3757934570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2015304565
validation[          , genome  6271] predictions:    6591/  10000 (65.91%), best:    6571/10000 (65.71%), error:     10102.82813, best error:     10036.22754 on epoch:    48, epoch:   50/50, mu: 0.9461198449, learning_rate: 0.0000078524, weight_decay: 0.0000106558

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 340.7391357422s, input_fired_time: 4.2875313759, output_fired_time: 0.0000000000, propagate_forward_time: 330.4223327637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0292663574
best training[          , genome  6271] predictions:   35047/  50000 (70.09%), best:    6571/10000 (65.71%), error:     45589.64063, best error:     10036.22754 on epoch:    48, epoch:   51/50, mu: 0.9482067823, learning_rate: 0.0000072451, weight_decay: 0.0000098479
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 69.6979064941s, input_fired_time: 0.8642895818, output_fired_time: 0.0000000000, propagate_forward_time: 66.7791061401, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0545120239
   testing[          , genome  6271] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:     10036.22754, best error:     10036.22754 on epoch:    48, epoch:   51/50, mu: 0.9482067823, learning_rate: 0.0000072451, weight_decay: 0.0000098479
backpropagation finished successfully!
19:21:42 (6648): called boinc_finish(0)

</stderr_txt>
]]>