Nameexact_genome_1542031203_189_5601_0
Workunit4081788
Created12 Nov 2018, 19:31:25 UTC
Sent12 Nov 2018, 19:33:57 UTC
Report deadline16 Jan 2019, 10:38:23 UTC
Received14 Nov 2018, 2:49:13 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID23692
Run time1 days 4 hours 44 min 7 sec
CPU time1 days 2 hours 21 min 48 sec
Validate stateValid
Credit16,157.02
Device peak FLOPS4.15 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size526.45 MB
Peak swap size546.16 MB
Peak disk usage8.53 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
00 (58.98%), error:     12049.37500, best error:     11781.57031 on epoch:    20, epoch:   21/100, mu: 0.8357880116, learning_rate: 0.0003260902, weight_decay: 0.0001713041

epoch time: 863.1954345703s, input_fired_time: 9.1683149338, output_fired_time: 4.6779651642, propagate_forward_time: 332.9939270020, propagate_backward_time: 507.2033691406, weight_update_time: 0.3087858558, other_time: 9.1518554688
epoch time: 69.0515747070s, input_fired_time: 0.7055202723, output_fired_time: 0.0000000000, propagate_forward_time: 66.7924346924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5536193848
validation[          , genome  5601] predictions:    5766/  10000 (57.66%), best:    5898/10000 (58.98%), error:     12107.99609, best error:     11781.57031 on epoch:    20, epoch:   22/100, mu: 0.8439762592, learning_rate: 0.0003048496, weight_decay: 0.0001623588

epoch time: 865.0592651367s, input_fired_time: 9.1803808212, output_fired_time: 4.6745619774, propagate_forward_time: 333.6442871094, propagate_backward_time: 508.3716430664, weight_update_time: 0.3147767484, other_time: 9.1884155273
epoch time: 69.3900146484s, input_fired_time: 0.7110326290, output_fired_time: 0.0000000000, propagate_forward_time: 67.1240310669, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5549545288
validation[          , genome  5601] predictions:    5812/  10000 (58.12%), best:    5898/10000 (58.98%), error:     11866.32031, best error:     11781.57031 on epoch:    20, epoch:   23/100, mu: 0.8517297506, learning_rate: 0.0002849926, weight_decay: 0.0001538805

epoch time: 903.7216796875s, input_fired_time: 9.6844739914, output_fired_time: 4.8839364052, propagate_forward_time: 348.3819274902, propagate_backward_time: 531.1125488281, weight_update_time: 0.3265291154, other_time: 9.6588134766
epoch time: 71.9349212646s, input_fired_time: 0.7476623654, output_fired_time: 0.0000000000, propagate_forward_time: 69.5563201904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6309356689
validation[          , genome  5601] predictions:    5836/  10000 (58.36%), best:    5898/10000 (58.98%), error:     11949.24609, best error:     11781.57031 on epoch:    20, epoch:   24/100, mu: 0.8590715528, learning_rate: 0.0002664290, weight_decay: 0.0001458450

epoch time: 887.4361572266s, input_fired_time: 9.5008716583, output_fired_time: 4.7988457680, propagate_forward_time: 342.2048339844, propagate_backward_time: 521.4605102539, weight_update_time: 0.3195595741, other_time: 9.4711303711
epoch time: 70.4237442017s, input_fired_time: 0.7231764197, output_fired_time: 0.0000000000, propagate_forward_time: 68.1231460571, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5774230957
validation[          , genome  5601] predictions:    5786/  10000 (57.86%), best:    5898/10000 (58.98%), error:     12154.40625, best error:     11781.57031 on epoch:    20, epoch:   25/100, mu: 0.8660234809, learning_rate: 0.0002490746, weight_decay: 0.0001382291

epoch time: 895.6687011719s, input_fired_time: 9.5648412704, output_fired_time: 4.8416175842, propagate_forward_time: 345.2556762695, propagate_backward_time: 526.4411621094, weight_update_time: 0.3235728145, other_time: 9.5654296875
epoch time: 71.8503417969s, input_fired_time: 0.7603828311, output_fired_time: 0.0000000000, propagate_forward_time: 69.4518356323, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6381225586
validation[          , genome  5601] predictions:    5772/  10000 (57.72%), best:    5898/10000 (58.98%), error:     12358.57617, best error:     11781.57031 on epoch:    20, epoch:   26/100, mu: 0.8726063371, learning_rate: 0.0002328506, weight_decay: 0.0001310108

epoch time: 906.3237915039s, input_fired_time: 9.6768302917, output_fired_time: 4.9214911461, propagate_forward_time: 349.1250610352, propagate_backward_time: 532.8703613281, weight_update_time: 0.3242584765, other_time: 9.7300415039
epoch time: 72.6971817017s, input_fired_time: 0.7526065111, output_fired_time: 0.0000000000, propagate_forward_time: 70.2774200439, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6671524048
validation[          , genome  5601] predictions:    5727/  10000 (57.27%), best:    5898/10000 (58.98%), error:     12230.21191, best error:     11781.57031 on epoch:    20, epoch:   27/100, mu: 0.8788396120, learning_rate: 0.0002176834, weight_decay: 0.0001241696

epoch time: 885.1263427734s, input_fired_time: 9.4131336212, output_fired_time: 4.8083109856, propagate_forward_time: 341.1598510742, propagate_backward_time: 520.2957153320, weight_update_time: 0.3181022406, other_time: 9.4493408203
epoch time: 70.1425094604s, input_fired_time: 0.7262265086, output_fired_time: 0.0000000000, propagate_forward_time: 67.8520812988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5642013550
validation[          , genome  5601] predictions:    5929/  10000 (59.29%), best:    5929/10000 (59.29%), error:     11628.15918, best error:     11628.15918 on epoch:    28, epoch:   28/100, mu: 0.8847419620, learning_rate: 0.0002035041, weight_decay: 0.0001176855

epoch time: 892.0758666992s, input_fired_time: 9.4993982315, output_fired_time: 4.8327136040, propagate_forward_time: 343.7026672363, propagate_backward_time: 524.3580932617, weight_update_time: 0.3205660880, other_time: 9.6830444336
epoch time: 70.8973770142s, input_fired_time: 0.7332108021, output_fired_time: 0.0000000000, propagate_forward_time: 68.5681381226, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5960311890
validation[          , genome  5601] predictions:    5701/  10000 (57.01%), best:    5929/10000 (59.29%), error:     12251.09180, best error:     11628.15918 on epoch:    28, epoch:   29/100, mu: 0.8903309107, learning_rate: 0.0001902485, weight_decay: 0.0001115401

epoch time: 912.2127075195s, input_fired_time: 9.7737779617, output_fired_time: 4.9579725266, propagate_forward_time: 351.8064270020, propagate_backward_time: 535.7813720703, weight_update_time: 0.3298325539, other_time: 9.8931884766
epoch time: 75.6501846313s, input_fired_time: 0.7850259542, output_fired_time: 0.0000000000, propagate_forward_time: 73.1532058716, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7119522095
validation[          , genome  5601] predictions:    5996/  10000 (59.96%), best:    5996/10000 (59.96%), error:     11450.72656, best error:     11450.72656 on epoch:    30, epoch:   30/100, mu: 0.8956230879, learning_rate: 0.0001778562, weight_decay: 0.0001057155

epoch time: 942.3091430664s, input_fired_time: 10.1577920914, output_fired_time: 5.1429681778, propagate_forward_time: 363.2973632813, propagate_backward_time: 553.4504394531, weight_update_time: 0.3456642032, other_time: 10.2606201172
epoch time: 73.6347885132s, input_fired_time: 0.7698138356, output_fired_time: 0.0000000000, propagate_forward_time: 71.1839141846, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6810607910
validation[          , genome  5601] predictions:    5811/  10000 (58.11%), best:    5996/10000 (59.96%), error:     11975.58105, best error:     11450.72656 on epoch:    30, epoch:   31/100, mu: 0.9006342292, learning_rate: 0.0001662712, weight_decay: 0.0001001951

epoch time: 918.2974243164s, input_fired_time: 9.8533935547, output_fired_time: 4.9886002541, propagate_forward_time: 353.9478759766, propagate_backward_time: 539.5351562500, weight_update_time: 0.3348064125, other_time: 9.9724121094
epoch time: 72.8734664917s, input_fired_time: 0.7596489787, output_fired_time: 0.0000000000, propagate_forward_time: 70.4535217285, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6602935791
validation[          , genome  5601] predictions:    5796/  10000 (57.96%), best:    5996/10000 (59.96%), error:     11963.93457, best error:     11450.72656 on epoch:    30, epoch:   32/100, mu: 0.9053792953, learning_rate: 0.0001554408, weight_decay: 0.0000949630

epoch time: 931.9617309570s, input_fired_time: 10.0239067078, output_fired_time: 5.0953407288, propagate_forward_time: 358.5497436523, propagate_backward_time: 548.2145996094, weight_update_time: 0.3408079147, other_time: 10.0781250000
epoch time: 73.8179626465s, input_fired_time: 0.7873297930, output_fired_time: 0.0000000000, propagate_forward_time: 71.3540649414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6765670776
validation[          , genome  5601] predictions:    5964/  10000 (59.64%), best:    5996/10000 (59.96%), error:     11663.78418, best error:     11450.72656 on epoch:    30, epoch:   33/100, mu: 0.9098724127, learning_rate: 0.0001453158, weight_decay: 0.0000900041

epoch time: 926.8467407227s, input_fired_time: 9.9623394012, output_fired_time: 5.0515499115, propagate_forward_time: 356.9675903320, propagate_backward_time: 544.7599487305, weight_update_time: 0.3351895511, other_time: 10.1052856445
epoch time: 70.8182754517s, input_fired_time: 0.7611376643, output_fired_time: 0.0000000000, propagate_forward_time: 68.4635391235, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5935974121
validation[          , genome  5601] predictions:    5947/  10000 (59.47%), best:    5996/10000 (59.96%), error:     11806.85742, best error:     11450.72656 on epoch:    30, epoch:   34/100, mu: 0.9141269922, learning_rate: 0.0001358504, weight_decay: 0.0000853042

epoch time: 883.4920654297s, input_fired_time: 9.4675149918, output_fired_time: 4.7883877754, propagate_forward_time: 340.6367797852, propagate_backward_time: 519.1784057617, weight_update_time: 0.3172078133, other_time: 9.4209594727
epoch time: 70.2780914307s, input_fired_time: 0.7365739346, output_fired_time: 0.0000000000, propagate_forward_time: 67.9648971558, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5766220093
validation[          , genome  5601] predictions:    5857/  10000 (58.57%), best:    5996/10000 (59.96%), error:     11867.17285, best error:     11450.72656 on epoch:    30, epoch:   35/100, mu: 0.9181556702, learning_rate: 0.0001270014, weight_decay: 0.0000808496

epoch time: 879.3444824219s, input_fired_time: 9.4409656525, output_fired_time: 4.7962813377, propagate_forward_time: 339.2609252930, propagate_backward_time: 516.4827270508, weight_update_time: 0.3133000433, other_time: 9.3635864258
epoch time: 70.0693588257s, input_fired_time: 0.7272943854, output_fired_time: 0.0000000000, propagate_forward_time: 67.7432785034, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5987854004
validation[          , genome  5601] predictions:    5858/  10000 (58.58%), best:    5996/10000 (59.96%), error:     11780.74121, best error:     11450.72656 on epoch:    30, epoch:   36/100, mu: 0.9219704270, learning_rate: 0.0001187289, weight_decay: 0.0000766277

epoch time: 873.2406616211s, input_fired_time: 9.4186296463, output_fired_time: 4.7119660378, propagate_forward_time: 336.8166809082, propagate_backward_time: 512.9943847656, weight_update_time: 0.3127478063, other_time: 9.2989501953
epoch time: 72.3112411499s, input_fired_time: 0.7506759167, output_fired_time: 0.0000000000, propagate_forward_time: 69.9831542969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5774078369
validation[          , genome  5601] predictions:    5997/  10000 (59.97%), best:    5996/10000 (59.96%), error:     11536.33496, best error:     11450.72656 on epoch:    30, epoch:   37/100, mu: 0.9255826473, learning_rate: 0.0001109953, weight_decay: 0.0000726263

epoch time: 894.7276611328s, input_fired_time: 9.6362724304, output_fired_time: 4.8492302895, propagate_forward_time: 345.0515747070, propagate_backward_time: 525.6543579102, weight_update_time: 0.3193131089, other_time: 9.5361938477
epoch time: 71.3572006226s, input_fired_time: 0.7528355122, output_fired_time: 0.0000000000, propagate_forward_time: 69.0046386719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5997238159
validation[          , genome  5601] predictions:    5971/  10000 (59.71%), best:    5996/10000 (59.96%), error:     11627.31348, best error:     11450.72656 on epoch:    30, epoch:   38/100, mu: 0.9290030599, learning_rate: 0.0001037654, weight_decay: 0.0000688338

epoch time: 880.6276855469s, input_fired_time: 9.4646606445, output_fired_time: 4.7434959412, propagate_forward_time: 339.6402282715, propagate_backward_time: 517.4195556641, weight_update_time: 0.3135495484, other_time: 9.3597412109
epoch time: 69.8462371826s, input_fired_time: 0.7298871875, output_fired_time: 0.0000000000, propagate_forward_time: 67.5897750854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5265731812
validation[          , genome  5601] predictions:    5845/  10000 (58.45%), best:    5996/10000 (59.96%), error:     12008.07715, best error:     11450.72656 on epoch:    30, epoch:   39/100, mu: 0.9322418571, learning_rate: 0.0000970064, weight_decay: 0.0000652394

epoch time: 894.2259521484s, input_fired_time: 9.6361398697, output_fired_time: 4.8158574104, propagate_forward_time: 344.6650085449, propagate_backward_time: 525.6318969727, weight_update_time: 0.3225355148, other_time: 9.4769897461
epoch time: 72.1534194946s, input_fired_time: 0.7612853646, output_fired_time: 0.0000000000, propagate_forward_time: 69.7863769531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6057586670
validation[          , genome  5601] predictions:    6029/  10000 (60.29%), best:    5996/10000 (59.96%), error:     11479.64746, best error:     11450.72656 on epoch:    30, epoch:   40/100, mu: 0.9353086352, learning_rate: 0.0000906877, weight_decay: 0.0000618326

epoch time: 888.2924194336s, input_fired_time: 9.5705604553, output_fired_time: 4.7782196999, propagate_forward_time: 342.5853271484, propagate_backward_time: 521.9815063477, weight_update_time: 0.3161571622, other_time: 9.3768310547
epoch time: 72.8850250244s, input_fired_time: 0.7769517303, output_fired_time: 0.0000000000, propagate_forward_time: 70.4763946533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6316757202
validation[          , genome  5601] predictions:    5983/  10000 (59.83%), best:    5983/10000 (59.83%), error:     11374.06152, best error:     11374.06152 on epoch:    41, epoch:   41/100, mu: 0.9382125735, learning_rate: 0.0000847806, weight_decay: 0.0000586037

epoch time: 886.3745117188s, input_fired_time: 9.5647153854, output_fired_time: 4.7589774132, propagate_forward_time: 342.5476379395, propagate_backward_time: 520.1629028320, weight_update_time: 0.3138800263, other_time: 9.3402709961
epoch time: 69.0265579224s, input_fired_time: 0.7180222273, output_fired_time: 0.0000000000, propagate_forward_time: 66.8045043945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5040283203
validation[          , genome  5601] predictions:    5623/  10000 (56.23%), best:    5983/10000 (59.83%), error:     12806.98047, best error:     11374.06152 on epoch:    41, epoch:   42/100, mu: 0.9409623742, learning_rate: 0.0000792582, weight_decay: 0.0000555435

epoch time: 866.0404052734s, input_fired_time: 9.2857236862, output_fired_time: 4.6917476654, propagate_forward_time: 334.3710632324, propagate_backward_time: 508.6093444824, weight_update_time: 0.3084111810, other_time: 9.0825500488
epoch time: 70.4702529907s, input_fired_time: 0.7413467169, output_fired_time: 0.0000000000, propagate_forward_time: 68.1792907715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5496139526
validation[          , genome  5601] predictions:    6068/  10000 (60.68%), best:    6068/10000 (60.68%), error:     11322.42578, best error:     11322.42578 on epoch:    43, epoch:   43/100, mu: 0.9435661435, learning_rate: 0.0000740956, weight_decay: 0.0000526431

epoch time: 865.3466796875s, input_fired_time: 9.3061408997, output_fired_time: 4.6624259949, propagate_forward_time: 334.1893920898, propagate_backward_time: 508.1278076172, weight_update_time: 0.3069904149, other_time: 9.0609130859
epoch time: 68.4950256348s, input_fired_time: 0.7219476700, output_fired_time: 0.0000000000, propagate_forward_time: 66.2792816162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4937973022
validation[          , genome  5601] predictions:    6112/  10000 (61.12%), best:    6112/10000 (61.12%), error:     11111.00391, best error:     11111.00391 on epoch:    44, epoch:   44/100, mu: 0.9460316300, learning_rate: 0.0000692692, weight_decay: 0.0000498941

epoch time: 862.0740966797s, input_fired_time: 9.2904357910, output_fired_time: 4.6530566216, propagate_forward_time: 332.6632385254, propagate_backward_time: 506.4063415527, weight_update_time: 0.3090526760, other_time: 9.0610656738
epoch time: 68.9632339478s, input_fired_time: 0.7263762355, output_fired_time: 0.0000000000, propagate_forward_time: 66.7171249390, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5197296143
validation[          , genome  5601] predictions:    6075/  10000 (60.75%), best:    6112/10000 (61.12%), error:     11476.75195, best error:     11111.00391 on epoch:    44, epoch:   45/100, mu: 0.9483662248, learning_rate: 0.0000647572, weight_decay: 0.0000472886

epoch time: 864.8875122070s, input_fired_time: 9.3331785202, output_fired_time: 4.7000827789, propagate_forward_time: 333.5263061523, propagate_backward_time: 508.1789550781, weight_update_time: 0.3087671697, other_time: 9.1489868164
epoch time: 70.4552230835s, input_fired_time: 0.7481768131, output_fired_time: 0.0000000000, propagate_forward_time: 68.1352005005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5718460083
validation[          , genome  5601] predictions:    6097/  10000 (60.97%), best:    6112/10000 (61.12%), error:     11227.86035, best error:     11111.00391 on epoch:    44, epoch:   46/100, mu: 0.9505768418, learning_rate: 0.0000605391, weight_decay: 0.0000448193

epoch time: 881.0427856445s, input_fired_time: 9.5065612793, output_fired_time: 4.7866168022, propagate_forward_time: 340.3309020996, propagate_backward_time: 517.1152954102, weight_update_time: 0.3141345382, other_time: 9.3034057617
epoch time: 70.1991958618s, input_fired_time: 0.7448549271, output_fired_time: 0.0000000000, propagate_forward_time: 67.8834915161, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5708465576
validation[          , genome  5601] predictions:    6001/  10000 (60.01%), best:    6112/10000 (61.12%), error:     11452.84570, best error:     11111.00391 on epoch:    44, epoch:   47/100, mu: 0.9526700974, learning_rate: 0.0000565958, weight_decay: 0.0000424788

epoch time: 878.2196655273s, input_fired_time: 9.4710664749, output_fired_time: 4.7604618073, propagate_forward_time: 339.1666259766, propagate_backward_time: 515.5458374023, weight_update_time: 0.3129079640, other_time: 9.2756958008
epoch time: 69.2437744141s, input_fired_time: 0.7379574180, output_fired_time: 0.0000000000, propagate_forward_time: 66.9776077271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5282058716
validation[          , genome  5601] predictions:    6090/  10000 (60.90%), best:    6112/10000 (61.12%), error:     11193.85352, best error:     11111.00391 on epoch:    44, epoch:   48/100, mu: 0.9546521902, learning_rate: 0.0000529093, weight_decay: 0.0000402606

epoch time: 870.0082397461s, input_fired_time: 9.4138832092, output_fired_time: 4.7112922668, propagate_forward_time: 335.8531494141, propagate_backward_time: 510.6940917969, weight_update_time: 0.3141438961, other_time: 9.3358154297
epoch time: 69.3764495850s, input_fired_time: 0.7434207797, output_fired_time: 0.0000000000, propagate_forward_time: 67.0421524048, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5908737183
validation[          , genome  5601] predictions:    6146/  10000 (61.46%), best:    6146/10000 (61.46%), error:     11089.25391, best error:     11089.25391 on epoch:    49, epoch:   49/100, mu: 0.9565290809, learning_rate: 0.0000494629, weight_decay: 0.0000381582

epoch time: 872.7910766602s, input_fired_time: 9.4212799072, output_fired_time: 4.7520399094, propagate_forward_time: 336.9011230469, propagate_backward_time: 512.3418579102, weight_update_time: 0.3155641556, other_time: 9.3748168945
epoch time: 70.3513641357s, input_fired_time: 0.7531723976, output_fired_time: 0.0000000000, propagate_forward_time: 67.9745407104, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6236495972
validation[          , genome  5601] predictions:    6132/  10000 (61.32%), best:    6132/10000 (61.32%), error:     10971.47559, best error:     10971.47559 on epoch:    50, epoch:   50/100, mu: 0.9583063126, learning_rate: 0.0000462410, weight_decay: 0.0000361656

epoch time: 874.7558593750s, input_fired_time: 9.4724140167, output_fired_time: 4.7885313034, propagate_forward_time: 337.8949279785, propagate_backward_time: 513.0026855469, weight_update_time: 0.3187246323, other_time: 9.5972900391
epoch time: 69.5880661011s, input_fired_time: 0.7417471409, output_fired_time: 0.0000000000, propagate_forward_time: 67.2812423706, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5650787354
validation[          , genome  5601] predictions:    6134/  10000 (61.34%), best:    6132/10000 (61.32%), error:     11091.19238, best error:     10971.47559 on epoch:    50, epoch:   51/100, mu: 0.9599891901, learning_rate: 0.0000432290, weight_decay: 0.0000342771

epoch time: 871.1502075195s, input_fired_time: 9.4106559753, output_fired_time: 4.7428088188, propagate_forward_time: 336.6275024414, propagate_backward_time: 510.9701843262, weight_update_time: 0.3177908957, other_time: 9.3990783691
epoch time: 70.7782287598s, input_fired_time: 0.7488547564, output_fired_time: 0.0000000000, propagate_forward_time: 68.4595260620, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5698471069
validation[          , genome  5601] predictions:    6049/  10000 (60.49%), best:    6132/10000 (61.32%), error:     11273.43848, best error:     10971.47559 on epoch:    50, epoch:   52/100, mu: 0.9615826607, learning_rate: 0.0000404132, weight_decay: 0.0000324872

epoch time: 902.9519042969s, input_fired_time: 9.8266363144, output_fired_time: 4.9582419395, propagate_forward_time: 349.3161315918, propagate_backward_time: 529.0751342773, weight_update_time: 0.3329094350, other_time: 9.7756958008
epoch time: 2787.7807617188s, input_fired_time: 0.7874268889, output_fired_time: 0.0000000000, propagate_forward_time: 2785.4384765625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5549316406
validation[          , genome  5601] predictions:    6177/  10000 (61.77%), best:    6177/10000 (61.77%), error:     10914.87305, best error:     10914.87305 on epoch:    53, epoch:   53/100, mu: 0.9630915523, learning_rate: 0.0000377808, weight_decay: 0.0000307907

epoch time: 904.3390502930s, input_fired_time: 9.6108036041, output_fired_time: 4.9339823723, propagate_forward_time: 353.7148437500, propagate_backward_time: 527.1750488281, weight_update_time: 0.3242383897, other_time: 8.9044189453
epoch time: 70.9438552856s, input_fired_time: 0.7399618030, output_fired_time: 0.0000000000, propagate_forward_time: 68.7129287720, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4909667969
validation[          , genome  5601] predictions:    6192/  10000 (61.92%), best:    6177/10000 (61.77%), error:     10932.33887, best error:     10914.87305 on epoch:    53, epoch:   54/100, mu: 0.9645203352, learning_rate: 0.0000353199, weight_decay: 0.0000291829

epoch time: 899.3182983398s, input_fired_time: 9.5599937439, output_fired_time: 4.8977503777, propagate_forward_time: 351.5300598145, propagate_backward_time: 524.4066162109, weight_update_time: 0.3208552897, other_time: 8.9238281250
epoch time: 72.0979309082s, input_fired_time: 0.7742196321, output_fired_time: 0.0000000000, propagate_forward_time: 69.8245697021, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4991378784
validation[          , genome  5601] predictions:    5967/  10000 (59.67%), best:    6177/10000 (61.77%), error:     11525.87988, best error:     10914.87305 on epoch:    53, epoch:   55/100, mu: 0.9658732414, learning_rate: 0.0000330193, weight_decay: 0.0000276589

epoch time: 881.1783447266s, input_fired_time: 9.5190362930, output_fired_time: 4.8068618774, propagate_forward_time: 340.3068847656, propagate_backward_time: 517.1259155273, weight_update_time: 0.3200402558, other_time: 9.4196166992
epoch time: 72.2374420166s, input_fired_time: 0.7820636034, output_fired_time: 0.0000000000, propagate_forward_time: 69.8164749146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6389007568
validation[          , genome  5601] predictions:    6127/  10000 (61.27%), best:    6177/10000 (61.77%), error:     11010.10352, best error:     10914.87305 on epoch:    53, epoch:   56/100, mu: 0.9671543241, learning_rate: 0.0000308685, weight_decay: 0.0000262146

epoch time: 903.0442504883s, input_fired_time: 9.8110399246, output_fired_time: 4.9387149811, propagate_forward_time: 348.2499084473, propagate_backward_time: 530.3078002930, weight_update_time: 0.3275001645, other_time: 9.7367553711
epoch time: 70.4435653687s, input_fired_time: 0.7590439916, output_fired_time: 0.0000000000, propagate_forward_time: 68.1029281616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5815963745
validation[          , genome  5601] predictions:    6235/  10000 (62.35%), best:    6235/10000 (62.35%), error:     10890.28320, best error:     10890.28320 on epoch:    57, epoch:   57/100, mu: 0.9683673978, learning_rate: 0.0000288578, weight_decay: 0.0000248457

epoch time: 885.9608154297s, input_fired_time: 9.6000947952, output_fired_time: 4.8537526131, propagate_forward_time: 342.1674804688, propagate_backward_time: 519.8019409180, weight_update_time: 0.3220103383, other_time: 9.5375366211
epoch time: 70.3591613770s, input_fired_time: 0.7569634318, output_fired_time: 0.0000000000, propagate_forward_time: 68.0059280396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5962677002
validation[          , genome  5601] predictions:    6158/  10000 (61.58%), best:    6235/10000 (62.35%), error:     11074.26953, best error:     10890.28320 on epoch:    57, epoch:   58/100, mu: 0.9695160389, learning_rate: 0.0000269781, weight_decay: 0.0000235483

epoch time: 878.1763916016s, input_fired_time: 9.5198554993, output_fired_time: 4.8023076057, propagate_forward_time: 338.9962768555, propagate_backward_time: 515.4296875000, weight_update_time: 0.3197337687, other_time: 9.4282836914
epoch time: 70.0142364502s, input_fired_time: 0.7535631061, output_fired_time: 0.0000000000, propagate_forward_time: 67.6675338745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5931396484
validation[          , genome  5601] predictions:    6165/  10000 (61.65%), best:    6235/10000 (62.35%), error:     10975.29199, best error:     10890.28320 on epoch:    57, epoch:   59/100, mu: 0.9706037045, learning_rate: 0.0000252208, weight_decay: 0.0000223186

epoch time: 874.5429077148s, input_fired_time: 9.4559192657, output_fired_time: 4.7779068947, propagate_forward_time: 337.0189514160, propagate_backward_time: 513.9362792969, weight_update_time: 0.3148114383, other_time: 9.3538818359
epoch time: 69.6086578369s, input_fired_time: 0.7447222471, output_fired_time: 0.0000000000, propagate_forward_time: 67.2846755981, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5792617798
validation[          , genome  5601] predictions:    6225/  10000 (62.25%), best:    6235/10000 (62.35%), error:     10897.61035, best error:     10890.28320 on epoch:    57, epoch:   60/100, mu: 0.9716336131, learning_rate: 0.0000235780, weight_decay: 0.0000211531

epoch time: 866.5722045898s, input_fired_time: 9.4269790649, output_fired_time: 4.7233276367, propagate_forward_time: 334.5009155273, propagate_backward_time: 508.6407165527, weight_update_time: 0.3138429821, other_time: 9.2802429199
epoch time: 69.4686889648s, input_fired_time: 0.7478792071, output_fired_time: 0.0000000000, propagate_forward_time: 67.1671600342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5536499023
validation[          , genome  5601] predictions:    6193/  10000 (61.93%), best:    6193/10000 (61.93%), error:     10889.79492, best error:     10889.79492 on epoch:    61, epoch:   61/100, mu: 0.9726088047, learning_rate: 0.0000220422, weight_decay: 0.0000200485

epoch time: 867.6109008789s, input_fired_time: 9.4266471863, output_fired_time: 4.7361493111, propagate_forward_time: 334.4715576172, propagate_backward_time: 509.5979003906, weight_update_time: 0.3161992133, other_time: 9.3786621094
epoch time: 69.6435165405s, input_fired_time: 0.7580869198, output_fired_time: 0.0000000000, propagate_forward_time: 67.2536392212, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6317901611
validation[          , genome  5601] predictions:    6233/  10000 (62.33%), best:    6233/10000 (62.33%), error:     10773.40918, best error:     10773.40918 on epoch:    62, epoch:   62/100, mu: 0.9735322595, learning_rate: 0.0000206064, weight_decay: 0.0000190016

epoch time: 895.0250854492s, input_fired_time: 9.7448396683, output_fired_time: 4.9253363609, propagate_forward_time: 345.6765441895, propagate_backward_time: 524.6294555664, weight_update_time: 0.3315449059, other_time: 10.0488891602
epoch time: 72.2068099976s, input_fired_time: 0.7906061411, output_fired_time: 0.0000000000, propagate_forward_time: 69.7144546509, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7017517090
validation[          , genome  5601] predictions:    6052/  10000 (60.52%), best:    6233/10000 (62.33%), error:     11167.98926, best error:     10773.40918 on epoch:    62, epoch:   63/100, mu: 0.9744066596, learning_rate: 0.0000192642, weight_decay: 0.0000180094

epoch time: 897.8167114258s, input_fired_time: 9.7866010666, output_fired_time: 4.9709610939, propagate_forward_time: 346.4105529785, propagate_backward_time: 526.6285400391, weight_update_time: 0.3327127695, other_time: 10.0200195313
epoch time: 71.1650695801s, input_fired_time: 0.7749913931, output_fired_time: 0.0000000000, propagate_forward_time: 68.7184753418, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6716003418
validation[          , genome  5601] predictions:    6266/  10000 (62.66%), best:    6266/10000 (62.66%), error:     10698.37012, best error:     10698.37012 on epoch:    64, epoch:   64/100, mu: 0.9752346277, learning_rate: 0.0000180094, weight_decay: 0.0000170689

epoch time: 896.0634155273s, input_fired_time: 9.7842464447, output_fired_time: 4.9504714012, propagate_forward_time: 346.5780639648, propagate_backward_time: 524.7658081055, weight_update_time: 0.3312773407, other_time: 9.9848022461
epoch time: 73.6171112061s, input_fired_time: 0.8082111478, output_fired_time: 0.0000000000, propagate_forward_time: 71.0652923584, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7436065674
validation[          , genome  5601] predictions:    6203/  10000 (62.03%), best:    6266/10000 (62.66%), error:     10844.62109, best error:     10698.37012 on epoch:    64, epoch:   65/100, mu: 0.9760186076, learning_rate: 0.0000168363, weight_decay: 0.0000161776

epoch time: 906.1362304688s, input_fired_time: 9.9357976913, output_fired_time: 4.9789686203, propagate_forward_time: 350.5337219238, propagate_backward_time: 530.5379638672, weight_update_time: 0.3424753845, other_time: 10.1497802734
epoch time: 70.8076629639s, input_fired_time: 0.7618992925, output_fired_time: 0.0000000000, propagate_forward_time: 68.4279098511, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6178512573
validation[          , genome  5601] predictions:    6189/  10000 (61.89%), best:    6266/10000 (62.66%), error:     10957.19531, best error:     10698.37012 on epoch:    64, epoch:   66/100, mu: 0.9767609835, learning_rate: 0.0000157396, weight_decay: 0.0000153328

epoch time: 889.3245849609s, input_fired_time: 9.6659822464, output_fired_time: 4.8041176796, propagate_forward_time: 342.8224182129, propagate_backward_time: 522.5024414063, weight_update_time: 0.3298123181, other_time: 9.5296630859
epoch time: 70.1558456421s, input_fired_time: 0.7583740950, output_fired_time: 0.0000000000, propagate_forward_time: 67.8211441040, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5763244629
validation[          , genome  5601] predictions:    6306/  10000 (63.06%), best:    6266/10000 (62.66%), error:     10730.74316, best error:     10698.37012 on epoch:    64, epoch:   67/100, mu: 0.9774639606, learning_rate: 0.0000147144, weight_decay: 0.0000145322

epoch time: 883.6488037109s, input_fired_time: 9.6040697098, output_fired_time: 4.8446412086, propagate_forward_time: 341.0254821777, propagate_backward_time: 518.6671752930, weight_update_time: 0.3186325133, other_time: 9.5073852539
epoch time: 70.3270950317s, input_fired_time: 0.7603029013, output_fired_time: 0.0000000000, propagate_forward_time: 67.9953460693, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5714492798
validation[          , genome  5601] predictions:    6122/  10000 (61.22%), best:    6266/10000 (62.66%), error:     11057.59473, best error:     10698.37012 on epoch:    64, epoch:   68/100, mu: 0.9781295657, learning_rate: 0.0000137559, weight_decay: 0.0000137733

epoch time: 879.3897705078s, input_fired_time: 9.5316572189, output_fired_time: 4.7964105606, propagate_forward_time: 339.0021667480, propagate_backward_time: 516.6037597656, weight_update_time: 0.3283321857, other_time: 9.4558105469
epoch time: 69.8979797363s, input_fired_time: 0.7553074360, output_fired_time: 0.0000000000, propagate_forward_time: 67.5809860229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5616836548
validation[          , genome  5601] predictions:    6244/  10000 (62.44%), best:    6266/10000 (62.66%), error:     10757.35449, best error:     10698.37012 on epoch:    64, epoch:   69/100, mu: 0.9787598252, learning_rate: 0.0000128599, weight_decay: 0.0000130541

epoch time: 882.7806396484s, input_fired_time: 9.6550598145, output_fired_time: 4.8704762459, propagate_forward_time: 341.4007568359, propagate_backward_time: 517.0939941406, weight_update_time: 0.3292053640, other_time: 9.7603759766
epoch time: 70.9921951294s, input_fired_time: 0.7773825526, output_fired_time: 0.0000000000, propagate_forward_time: 68.5216674805, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6931457520
validation[          , genome  5601] predictions:    6307/  10000 (63.07%), best:    6266/10000 (62.66%), error:     10711.93555, best error:     10698.37012 on epoch:    64, epoch:   70/100, mu: 0.9793566465, learning_rate: 0.0000120223, weight_decay: 0.0000123724

epoch time: 885.4106445313s, input_fired_time: 9.7362585068, output_fired_time: 4.9167594910, propagate_forward_time: 342.8104858398, propagate_backward_time: 517.9947509766, weight_update_time: 0.3315516710, other_time: 9.9523925781
epoch time: 70.9889068604s, input_fired_time: 0.7874273062, output_fired_time: 0.0000000000, propagate_forward_time: 68.5210571289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6804199219
validation[          , genome  5601] predictions:    6195/  10000 (61.95%), best:    6266/10000 (62.66%), error:     10944.16211, best error:     10698.37012 on epoch:    64, epoch:   71/100, mu: 0.9799217582, learning_rate: 0.0000112392, weight_decay: 0.0000117263

epoch time: 903.8717651367s, input_fired_time: 10.0008487701, output_fired_time: 5.0004420280, propagate_forward_time: 349.3397521973, propagate_backward_time: 529.3908081055, weight_update_time: 0.3330436051, other_time: 10.1399536133
epoch time: 72.6916656494s, input_fired_time: 0.7951123118, output_fired_time: 0.0000000000, propagate_forward_time: 70.1829223633, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7136306763
validation[          , genome  5601] predictions:    6250/  10000 (62.50%), best:    6266/10000 (62.66%), error:     10801.54590, best error:     10698.37012 on epoch:    64, epoch:   72/100, mu: 0.9804568887, learning_rate: 0.0000105071, weight_decay: 0.0000111140

epoch time: 895.1301269531s, input_fired_time: 9.8127346039, output_fired_time: 4.9666996002, propagate_forward_time: 345.9232177734, propagate_backward_time: 524.4178466797, weight_update_time: 0.3373266757, other_time: 10.0096435547
epoch time: 70.6002349854s, input_fired_time: 0.7704793215, output_fired_time: 0.0000000000, propagate_forward_time: 68.2532424927, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5765151978
validation[          , genome  5601] predictions:    6246/  10000 (62.46%), best:    6266/10000 (62.66%), error:     10738.94922, best error:     10698.37012 on epoch:    64, epoch:   73/100, mu: 0.9809635878, learning_rate: 0.0000098227, weight_decay: 0.0000105336

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_189_5601.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_189_5601_0_r309903335_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50504
read mu: 0.981443
read mu_delta: 0.946903
read initial_learning_rate: 0.00134163
read learning_rate: 9.18286e-06
read learning_rate_delta: 0.934863
read initial_weight_decay: 0.000528327
read weight_decay: 9.98355e-06
read weight_decay_delta: 0.947781
read batch_size: 88
read epsilon: 1e-07
read alpha: 0.103055
read input_dropout_probability: 0.00142396
read hidden_dropout_probability: 0.100509
read velocity_reset: 1797
read epoch: 74
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 64
read number_validation_images: 10000
read best_validation_predictions: 6266
read best_validation_error: 10698.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5601
read normal distribution: '0 -1.18126010894775 0.0343537591397762'
generator_str: '1346162299'
read generator: 1346162299
read generated_by_map:
1 split_edge 1
reading 53 nodes.
reading 220 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1175.70129394531s, input_fired_time: 12.8371210098267, output_fired_time: 6.45929670333862, propagate_forward_time: 455.429382324219, propagate_backward_time: 687.858825683594, weight_update_time: 0.485497176647186, other_time: 13.1165771484375
epoch time: 95.0941848754883s, input_fired_time: 1.02899372577667, output_fired_time: 0, propagate_forward_time: 91.9491500854492, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.11604309082031
validation[          , genome  5601] predictions:    6249/  10000 (62.49%), best:    6266/10000 (62.66%), error:     10700.10352, best error:     10698.37012 on epoch:    64, epoch:   74/100, mu: 0.9814434052, learning_rate: 0.0000091829, weight_decay: 0.0000099836

epoch time: 1154.3353271484s, input_fired_time: 12.9413490295, output_fired_time: 6.9008603096, propagate_forward_time: 446.6781616211, propagate_backward_time: 674.9434204102, weight_update_time: 0.4431572556, other_time: 12.8714599609
epoch time: 95.0913543701s, input_fired_time: 1.1417167187, output_fired_time: 0.0000000000, propagate_forward_time: 91.8612365723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0884017944
validation[          , genome  5601] predictions:    6287/  10000 (62.87%), best:    6287/10000 (62.87%), error:     10693.80957, best error:     10693.80957 on epoch:    75, epoch:   75/100, mu: 0.9818977118, learning_rate: 0.0000085847, weight_decay: 0.0000094622

epoch time: 1166.6145019531s, input_fired_time: 12.9352836609, output_fired_time: 6.5132637024, propagate_forward_time: 451.7446594238, propagate_backward_time: 682.2169799805, weight_update_time: 0.4316560030, other_time: 13.2042846680
epoch time: 88.8425598145s, input_fired_time: 0.9738882184, output_fired_time: 0.0000000000, propagate_forward_time: 85.8859252930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9827499390
validation[          , genome  5601] predictions:    6316/  10000 (63.16%), best:    6316/10000 (63.16%), error:     10662.49316, best error:     10662.49316 on epoch:    76, epoch:   76/100, mu: 0.9823279381, learning_rate: 0.0000080255, weight_decay: 0.0000089681

epoch time: 1160.5645751953s, input_fired_time: 12.4668664932, output_fired_time: 6.2004098892, propagate_forward_time: 451.4776000977, propagate_backward_time: 677.5317382813, weight_update_time: 0.5360539556, other_time: 12.8878784180
epoch time: 90.3102645874s, input_fired_time: 1.0287525654, output_fired_time: 0.0000000000, propagate_forward_time: 87.2367172241, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0447921753
validation[          , genome  5601] predictions:    6309/  10000 (63.09%), best:    6309/10000 (63.09%), error:     10589.62207, best error:     10589.62207 on epoch:    77, epoch:   77/100, mu: 0.9827352762, learning_rate: 0.0000075028, weight_decay: 0.0000084998

epoch time: 1091.3751220703s, input_fired_time: 12.0483922958, output_fired_time: 6.2383542061, propagate_forward_time: 422.8899841309, propagate_backward_time: 637.9734497070, weight_update_time: 0.4684583843, other_time: 12.2249145508
epoch time: 88.1798248291s, input_fired_time: 0.9777246714, output_fired_time: 0.0000000000, propagate_forward_time: 85.1842651367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0178375244
validation[          , genome  5601] predictions:    6308/  10000 (63.08%), best:    6309/10000 (63.09%), error:     10619.66895, best error:     10589.62207 on epoch:    77, epoch:   78/100, mu: 0.9831210375, learning_rate: 0.0000070141, weight_decay: 0.0000080559

epoch time: 1089.6635742188s, input_fired_time: 11.7702045441, output_fired_time: 6.1270604134, propagate_forward_time: 422.4241027832, propagate_backward_time: 637.4389038086, weight_update_time: 0.4115989804, other_time: 11.9032592773
epoch time: 87.3264999390s, input_fired_time: 0.9609633684, output_fired_time: 0.0000000000, propagate_forward_time: 84.3977661133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9677734375
validation[          , genome  5601] predictions:    6313/  10000 (63.13%), best:    6313/10000 (63.13%), error:     10552.77832, best error:     10552.77832 on epoch:    79, epoch:   79/100, mu: 0.9834862947, learning_rate: 0.0000065572, weight_decay: 0.0000076353

epoch time: 1070.1667480469s, input_fired_time: 11.7094507217, output_fired_time: 6.2017598152, propagate_forward_time: 414.6960144043, propagate_backward_time: 625.8239135742, weight_update_time: 0.3808414638, other_time: 11.7355346680
epoch time: 83.1100845337s, input_fired_time: 0.8701839447, output_fired_time: 0.0000000000, propagate_forward_time: 80.2857360840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9541625977
validation[          , genome  5601] predictions:    6286/  10000 (62.86%), best:    6313/10000 (63.13%), error:     10696.88867, best error:     10552.77832 on epoch:    79, epoch:   80/100, mu: 0.9838321805, learning_rate: 0.0000061301, weight_decay: 0.0000072366

epoch time: 1079.6724853516s, input_fired_time: 12.0803928375, output_fired_time: 5.9784617424, propagate_forward_time: 419.3602294922, propagate_backward_time: 630.1330566406, weight_update_time: 0.3667332232, other_time: 12.1202392578
epoch time: 88.4759521484s, input_fired_time: 0.9670660496, output_fired_time: 0.0000000000, propagate_forward_time: 85.5376281738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9712600708
validation[          , genome  5601] predictions:    6293/  10000 (62.93%), best:    6313/10000 (63.13%), error:     10618.79883, best error:     10552.77832 on epoch:    79, epoch:   81/100, mu: 0.9841596484, learning_rate: 0.0000057308, weight_decay: 0.0000068587

epoch time: 1064.2171630859s, input_fired_time: 11.8090229034, output_fired_time: 6.0191197395, propagate_forward_time: 413.7142028809, propagate_backward_time: 620.2235717773, weight_update_time: 0.4188564420, other_time: 12.4511108398
epoch time: 81.6134719849s, input_fired_time: 0.8827205896, output_fired_time: 0.0000000000, propagate_forward_time: 78.8917160034, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8390350342
validation[          , genome  5601] predictions:    6287/  10000 (62.87%), best:    6313/10000 (63.13%), error:     10686.12305, best error:     10552.77832 on epoch:    79, epoch:   82/100, mu: 0.9844697714, learning_rate: 0.0000053575, weight_decay: 0.0000065005

epoch time: 1049.5996093750s, input_fired_time: 11.7496013641, output_fired_time: 5.7734165192, propagate_forward_time: 406.3337707520, propagate_backward_time: 614.0875854492, weight_update_time: 0.3935157955, other_time: 11.6552124023
epoch time: 84.2544250488s, input_fired_time: 0.8647806048, output_fired_time: 0.0000000000, propagate_forward_time: 81.4268417358, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9627990723
validation[          , genome  5601] predictions:    6253/  10000 (62.53%), best:    6313/10000 (63.13%), error:     10639.88574, best error:     10552.77832 on epoch:    79, epoch:   83/100, mu: 0.9847633839, learning_rate: 0.0000050085, weight_decay: 0.0000061611

epoch time: 1067.5823974609s, input_fired_time: 11.7253932953, output_fired_time: 6.0453553200, propagate_forward_time: 414.0479431152, propagate_backward_time: 623.8416137695, weight_update_time: 0.4082851112, other_time: 11.9220581055
epoch time: 77.3025512695s, input_fired_time: 0.8903105259, output_fired_time: 0.0000000000, propagate_forward_time: 74.6008682251, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8113708496
validation[          , genome  5601] predictions:    6338/  10000 (63.38%), best:    6313/10000 (63.13%), error:     10601.15918, best error:     10552.77832 on epoch:    79, epoch:   84/100, mu: 0.9850414395, learning_rate: 0.0000046823, weight_decay: 0.0000058393

epoch time: 1081.2952880859s, input_fired_time: 12.1142387390, output_fired_time: 6.0855178833, propagate_forward_time: 418.9039611816, propagate_backward_time: 631.9050903320, weight_update_time: 0.3898807466, other_time: 12.2864379883
epoch time: 85.3960342407s, input_fired_time: 1.0660958290, output_fired_time: 0.0000000000, propagate_forward_time: 82.2421798706, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0877609253
validation[          , genome  5601] predictions:    6329/  10000 (63.29%), best:    6313/10000 (63.13%), error:     10577.47266, best error:     10552.77832 on epoch:    79, epoch:   85/100, mu: 0.9853047132, learning_rate: 0.0000043773, weight_decay: 0.0000055344

epoch time: 1064.0484619141s, input_fired_time: 11.9820108414, output_fired_time: 6.0744934082, propagate_forward_time: 411.8923950195, propagate_backward_time: 622.2689819336, weight_update_time: 0.3554794788, other_time: 11.8305664063
epoch time: 89.0529632568s, input_fired_time: 1.1395255327, output_fired_time: 0.0000000000, propagate_forward_time: 85.8602142334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0532226563
validation[          , genome  5601] predictions:    6317/  10000 (63.17%), best:    6313/10000 (63.13%), error:     10589.83496, best error:     10552.77832 on epoch:    79, epoch:   86/100, mu: 0.9855540395, learning_rate: 0.0000040922, weight_decay: 0.0000052454

epoch time: 1061.8394775391s, input_fired_time: 11.8001356125, output_fired_time: 5.9689669609, propagate_forward_time: 411.7934265137, propagate_backward_time: 620.0484619141, weight_update_time: 0.4117074609, other_time: 12.2283935547
epoch time: 85.8815765381s, input_fired_time: 0.9758743644, output_fired_time: 0.0000000000, propagate_forward_time: 82.9250869751, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9806137085
validation[          , genome  5601] predictions:    6345/  10000 (63.45%), best:    6345/10000 (63.45%), error:     10543.41211, best error:     10543.41211 on epoch:    87, epoch:   87/100, mu: 0.9857901335, learning_rate: 0.0000038256, weight_decay: 0.0000049715

epoch time: 1035.1011962891s, input_fired_time: 11.7350950241, output_fired_time: 5.7601952553, propagate_forward_time: 401.8267517090, propagate_backward_time: 604.0521850586, weight_update_time: 0.3908912539, other_time: 11.7269897461
epoch time: 79.5957412720s, input_fired_time: 0.9546729326, output_fired_time: 0.0000000000, propagate_forward_time: 76.8137817383, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8272857666
validation[          , genome  5601] predictions:    6326/  10000 (63.26%), best:    6345/10000 (63.45%), error:     10546.59766, best error:     10543.41211 on epoch:    87, epoch:   88/100, mu: 0.9860136509, learning_rate: 0.0000035764, weight_decay: 0.0000047119

epoch time: 1043.3918457031s, input_fired_time: 11.4957838058, output_fired_time: 5.7319102287, propagate_forward_time: 404.5331115723, propagate_backward_time: 609.9801635742, weight_update_time: 0.3898805380, other_time: 11.6509399414
epoch time: 79.4646682739s, input_fired_time: 0.9374522567, output_fired_time: 0.0000000000, propagate_forward_time: 76.7044525146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8227615356
validation[          , genome  5601] predictions:    6324/  10000 (63.24%), best:    6324/10000 (63.24%), error:     10536.04395, best error:     10536.04395 on epoch:    89, epoch:   89/100, mu: 0.9862253070, learning_rate: 0.0000033435, weight_decay: 0.0000044658

epoch time: 1059.9935302734s, input_fired_time: 11.6013050079, output_fired_time: 6.0390625000, propagate_forward_time: 409.9978637695, propagate_backward_time: 620.2264404297, weight_update_time: 0.3687789142, other_time: 12.1288452148
epoch time: 88.2374801636s, input_fired_time: 1.0092830658, output_fired_time: 0.0000000000, propagate_forward_time: 85.0423889160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1858062744
validation[          , genome  5601] predictions:    6363/  10000 (63.63%), best:    6363/10000 (63.63%), error:     10499.75098, best error:     10499.75098 on epoch:    90, epoch:   90/100, mu: 0.9864257574, learning_rate: 0.0000031257, weight_decay: 0.0000042326

epoch time: 1045.2427978516s, input_fired_time: 11.4834976196, output_fired_time: 5.9415102005, propagate_forward_time: 404.9219360352, propagate_backward_time: 611.1264038086, weight_update_time: 0.4041053951, other_time: 11.7694091797
epoch time: 86.4052963257s, input_fired_time: 1.0468467474, output_fired_time: 0.0000000000, propagate_forward_time: 83.4271316528, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9313201904
validation[          , genome  5601] predictions:    6341/  10000 (63.41%), best:    6363/10000 (63.63%), error:     10516.94922, best error:     10499.75098 on epoch:    90, epoch:   91/100, mu: 0.9866155386, learning_rate: 0.0000029221, weight_decay: 0.0000040116

epoch time: 1054.9044189453s, input_fired_time: 11.9080162048, output_fired_time: 6.0084772110, propagate_forward_time: 408.6661682129, propagate_backward_time: 616.2894287109, weight_update_time: 0.3865896165, other_time: 12.0324707031
epoch time: 77.1884231567s, input_fired_time: 0.7931263447, output_fired_time: 0.0000000000, propagate_forward_time: 74.5380859375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8572082520
validation[          , genome  5601] predictions:    6371/  10000 (63.71%), best:    6363/10000 (63.63%), error:     10526.22266, best error:     10499.75098 on epoch:    90, epoch:   92/100, mu: 0.9867952466, learning_rate: 0.0000027317, weight_decay: 0.0000038021

epoch time: 1034.4052734375s, input_fired_time: 11.9151620865, output_fired_time: 5.5384554863, propagate_forward_time: 400.9800415039, propagate_backward_time: 604.6215209961, weight_update_time: 0.4134185612, other_time: 11.3500976563
epoch time: 92.2243270874s, input_fired_time: 1.1033921242, output_fired_time: 0.0000000000, propagate_forward_time: 89.0069808960, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1139526367
validation[          , genome  5601] predictions:    6360/  10000 (63.60%), best:    6363/10000 (63.63%), error:     10515.33691, best error:     10499.75098 on epoch:    90, epoch:   93/100, mu: 0.9869654179, learning_rate: 0.0000025538, weight_decay: 0.0000036036

epoch time: 1087.3681640625s, input_fired_time: 12.0134019852, output_fired_time: 5.9401173592, propagate_forward_time: 421.6241455078, propagate_backward_time: 635.5369262695, weight_update_time: 0.4139550328, other_time: 12.2536010742
epoch time: 85.5458679199s, input_fired_time: 0.8834888935, output_fired_time: 0.0000000000, propagate_forward_time: 82.5571212769, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1052551270
validation[          , genome  5601] predictions:    6375/  10000 (63.75%), best:    6375/10000 (63.75%), error:     10464.23535, best error:     10464.23535 on epoch:    94, epoch:   94/100, mu: 0.9871265292, learning_rate: 0.0000023875, weight_decay: 0.0000034154

epoch time: 1053.5971679688s, input_fired_time: 11.8386468887, output_fired_time: 5.6921019554, propagate_forward_time: 409.7169189453, propagate_backward_time: 614.2354736328, weight_update_time: 0.3698943257, other_time: 12.1140136719
epoch time: 82.8610839844s, input_fired_time: 0.9223120213, output_fired_time: 0.0000000000, propagate_forward_time: 79.9687042236, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9700698853
validation[          , genome  5601] predictions:    6356/  10000 (63.56%), best:    6375/10000 (63.75%), error:     10492.21484, best error:     10464.23535 on epoch:    94, epoch:   95/100, mu: 0.9872791171, learning_rate: 0.0000022320, weight_decay: 0.0000032371

epoch time: 1092.5014648438s, input_fired_time: 11.7466745377, output_fired_time: 6.0290226936, propagate_forward_time: 424.6634216309, propagate_backward_time: 637.7947998047, weight_update_time: 0.4140869677, other_time: 12.2674560547
epoch time: 90.1077117920s, input_fired_time: 1.0476971865, output_fired_time: 0.0000000000, propagate_forward_time: 86.9672622681, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0927505493
validation[          , genome  5601] predictions:    6309/  10000 (63.09%), best:    6375/10000 (63.75%), error:     10487.76563, best error:     10464.23535 on epoch:    94, epoch:   96/100, mu: 0.9874235988, learning_rate: 0.0000020866, weight_decay: 0.0000030680

epoch time: 1062.9794921875s, input_fired_time: 11.8349752426, output_fired_time: 5.8903260231, propagate_forward_time: 411.3151550293, propagate_backward_time: 621.9756469727, weight_update_time: 0.3696679473, other_time: 11.9633178711
epoch time: 87.8138809204s, input_fired_time: 0.9797396660, output_fired_time: 0.0000000000, propagate_forward_time: 84.8128356934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0213088989
validation[          , genome  5601] predictions:    6361/  10000 (63.61%), best:    6375/10000 (63.75%), error:     10487.27246, best error:     10464.23535 on epoch:    94, epoch:   97/100, mu: 0.9875603914, learning_rate: 0.0000019507, weight_decay: 0.0000029078

epoch time: 1058.2792968750s, input_fired_time: 11.6040325165, output_fired_time: 5.6646780968, propagate_forward_time: 409.9105529785, propagate_backward_time: 619.1451416016, weight_update_time: 0.3568769693, other_time: 11.9549560547
epoch time: 88.4013748169s, input_fired_time: 0.9113903642, output_fired_time: 0.0000000000, propagate_forward_time: 85.2984390259, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1915435791
validation[          , genome  5601] predictions:    6367/  10000 (63.67%), best:    6375/10000 (63.75%), error:     10502.43652, best error:     10464.23535 on epoch:    94, epoch:   98/100, mu: 0.9876899123, learning_rate: 0.0000018236, weight_decay: 0.0000027560

epoch time: 1072.0197753906s, input_fired_time: 11.7140016556, output_fired_time: 5.8410224915, propagate_forward_time: 415.5411987305, propagate_backward_time: 626.7599487305, weight_update_time: 0.4180112481, other_time: 12.1635742188
epoch time: 89.0032424927s, input_fired_time: 1.0154925585, output_fired_time: 0.0000000000, propagate_forward_time: 85.8862915039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1014556885
validation[          , genome  5601] predictions:    6351/  10000 (63.51%), best:    6375/10000 (63.75%), error:     10470.31348, best error:     10464.23535 on epoch:    94, epoch:   99/100, mu: 0.9878125787, learning_rate: 0.0000017048, weight_decay: 0.0000026121

epoch time: 1077.3608398438s, input_fired_time: 11.5150957108, output_fired_time: 6.0029277802, propagate_forward_time: 418.5464172363, propagate_backward_time: 628.9267578125, weight_update_time: 0.4412887692, other_time: 12.3696289063
epoch time: 88.2741394043s, input_fired_time: 0.8997480273, output_fired_time: 0.0000000000, propagate_forward_time: 85.2363891602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1380004883
validation[          , genome  5601] predictions:    6354/  10000 (63.54%), best:    6375/10000 (63.75%), error:     10479.15234, best error:     10464.23535 on epoch:    94, epoch:  100/100, mu: 0.9879287481, learning_rate: 0.0000015938, weight_decay: 0.0000024757

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 425.2131652832s, input_fired_time: 4.6040887833, output_fired_time: 0.0000000000, propagate_forward_time: 410.5185546875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.0905151367
best training[          , genome  5601] predictions:   34832/  50000 (69.66%), best:    6375/10000 (63.75%), error:     46000.93750, best error:     10464.23535 on epoch:    94, epoch:  101/100, mu: 0.9880387187, learning_rate: 0.0000014900, weight_decay: 0.0000023464
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 89.9523696899s, input_fired_time: 0.9952635765, output_fired_time: 0.0000000000, propagate_forward_time: 86.0797882080, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8773193359
   testing[          , genome  5601] predictions:    6375/  10000 (63.75%), best:    6375/10000 (63.75%), error:     10464.23535, best error:     10464.23535 on epoch:    94, epoch:  101/100, mu: 0.9880387187, learning_rate: 0.0000014900, weight_decay: 0.0000023464
backpropagation finished successfully!
21:47:42 (10568): called boinc_finish(0)

</stderr_txt>
]]>