Nameexact_genome_1542531604_183_6241_2
Workunit4097081
Created18 Nov 2018, 11:13:55 UTC
Sent18 Nov 2018, 11:43:38 UTC
Report deadline26 Jan 2019, 6:55:44 UTC
Received22 Nov 2018, 1:32:26 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID84153
Run time1 days 4 hours 59 min 59 sec
CPU time1 days 3 hours 27 min 36 sec
Validate stateValid
Credit17,200.02
Device peak FLOPS4.74 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.33
Peak working set size506.55 MB
Peak swap size510.27 MB
Peak disk usage4.86 MB

Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
, output_fired_time: 5.1743564606, propagate_forward_time: 387.6793823242, propagate_backward_time: 624.3986206055, weight_update_time: 0.8018548489, other_time: 11.9724121094
epoch time: 76.8636550903s, input_fired_time: 0.7703858018, output_fired_time: 0.0000000000, propagate_forward_time: 72.9111557007, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1821136475
validation[          , genome  6241] predictions:    5418/  10000 (54.18%), best:    5544/10000 (55.44%), error:     13797.85059, best error:     13490.29883 on epoch:    19, epoch:   21/100, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 1027.0941162109s, input_fired_time: 7.8367762566, output_fired_time: 6.2057180405, propagate_forward_time: 378.4053344727, propagate_backward_time: 621.7904052734, weight_update_time: 0.7682786584, other_time: 12.8558959961
epoch time: 77.5787200928s, input_fired_time: 0.7587445974, output_fired_time: 0.0000000000, propagate_forward_time: 74.7746429443, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0453338623
validation[          , genome  6241] predictions:    5610/  10000 (56.10%), best:    5544/10000 (55.44%), error:     13558.79199, best error:     13490.29883 on epoch:    19, epoch:   22/100, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 1005.6968994141s, input_fired_time: 7.6354293823, output_fired_time: 5.1077294350, propagate_forward_time: 370.8250427246, propagate_backward_time: 609.3565673828, weight_update_time: 0.7887437344, other_time: 12.7720947266
epoch time: 79.7698898315s, input_fired_time: 0.7993500233, output_fired_time: 0.0000000000, propagate_forward_time: 76.8725967407, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0979461670
validation[          , genome  6241] predictions:    5666/  10000 (56.66%), best:    5666/10000 (56.66%), error:     12999.92578, best error:     12999.92578 on epoch:    23, epoch:   23/100, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 1025.8862304688s, input_fired_time: 6.7018604279, output_fired_time: 5.1676759720, propagate_forward_time: 373.3687744141, propagate_backward_time: 627.8566284180, weight_update_time: 0.8175892234, other_time: 12.7913208008
epoch time: 80.1577377319s, input_fired_time: 0.8308702111, output_fired_time: 0.0000000000, propagate_forward_time: 77.1137771606, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2130889893
validation[          , genome  6241] predictions:    5588/  10000 (55.88%), best:    5666/10000 (56.66%), error:     13255.17676, best error:     12999.92578 on epoch:    23, epoch:   24/100, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 1019.4013061523s, input_fired_time: 6.7106156349, output_fired_time: 6.2344636917, propagate_forward_time: 369.8871459961, propagate_backward_time: 622.7568969727, weight_update_time: 0.7886087894, other_time: 13.8121948242
epoch time: 78.9867935181s, input_fired_time: 0.7836439610, output_fired_time: 0.0000000000, propagate_forward_time: 76.0941696167, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1089782715
validation[          , genome  6241] predictions:    5526/  10000 (55.26%), best:    5666/10000 (56.66%), error:     13656.02441, best error:     12999.92578 on epoch:    23, epoch:   25/100, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 1026.1815185547s, input_fired_time: 7.7459082603, output_fired_time: 5.1960477829, propagate_forward_time: 373.0120239258, propagate_backward_time: 625.2925415039, weight_update_time: 0.7694813013, other_time: 14.9349975586
epoch time: 74.9561996460s, input_fired_time: 0.7630715966, output_fired_time: 0.0000000000, propagate_forward_time: 71.1719894409, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0211410522
validation[          , genome  6241] predictions:    5437/  10000 (54.37%), best:    5666/10000 (56.66%), error:     14262.70410, best error:     12999.92578 on epoch:    23, epoch:   26/100, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 1019.4029541016s, input_fired_time: 6.7006483078, output_fired_time: 5.2002339363, propagate_forward_time: 375.7307434082, propagate_backward_time: 620.0021972656, weight_update_time: 0.7822502851, other_time: 11.7691650391
epoch time: 79.1759719849s, input_fired_time: 0.8000684977, output_fired_time: 0.0000000000, propagate_forward_time: 76.2894744873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0864257812
validation[          , genome  6241] predictions:    5624/  10000 (56.24%), best:    5666/10000 (56.66%), error:     13329.84180, best error:     12999.92578 on epoch:    23, epoch:   27/100, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 1021.5591430664s, input_fired_time: 6.6652650833, output_fired_time: 6.1929893494, propagate_forward_time: 368.8940429688, propagate_backward_time: 628.0000000000, weight_update_time: 0.7579162121, other_time: 11.8068237305
epoch time: 79.2865295410s, input_fired_time: 0.8050380349, output_fired_time: 0.0000000000, propagate_forward_time: 76.3675155640, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1139755249
validation[          , genome  6241] predictions:    5538/  10000 (55.38%), best:    5666/10000 (56.66%), error:     13644.99805, best error:     12999.92578 on epoch:    23, epoch:   28/100, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1035.2078857422s, input_fired_time: 9.8041458130, output_fired_time: 5.2247633934, propagate_forward_time: 374.3198242188, propagate_backward_time: 633.1455688477, weight_update_time: 0.7655487657, other_time: 12.7135009766
epoch time: 76.5984039307s, input_fired_time: 0.7867488861, output_fired_time: 0.0000000000, propagate_forward_time: 73.7308044434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0808486938
validation[          , genome  6241] predictions:    5651/  10000 (56.51%), best:    5666/10000 (56.66%), error:     13511.04492, best error:     12999.92578 on epoch:    23, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 1034.3220214844s, input_fired_time: 8.8714389801, output_fired_time: 5.1809296608, propagate_forward_time: 386.3393249512, propagate_backward_time: 619.8760375977, weight_update_time: 0.8010163903, other_time: 14.0542602539
epoch time: 77.6613311768s, input_fired_time: 0.8111274838, output_fired_time: 0.0000000000, propagate_forward_time: 74.6347427368, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2154617310
validation[          , genome  6241] predictions:    5750/  10000 (57.50%), best:    5750/10000 (57.50%), error:     12953.91309, best error:     12953.91309 on epoch:    30, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 1040.8405761719s, input_fired_time: 6.9566245079, output_fired_time: 5.3924298286, propagate_forward_time: 379.3853149414, propagate_backward_time: 632.4922485352, weight_update_time: 0.8252693415, other_time: 16.6138916016
epoch time: 79.8605499268s, input_fired_time: 0.9026911855, output_fired_time: 0.0000000000, propagate_forward_time: 76.6367721558, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3210830688
validation[          , genome  6241] predictions:    5675/  10000 (56.75%), best:    5750/10000 (57.50%), error:     13414.70312, best error:     12953.91309 on epoch:    30, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 1039.7327880859s, input_fired_time: 8.0864953995, output_fired_time: 5.4210863113, propagate_forward_time: 371.8339538574, propagate_backward_time: 640.9227905273, weight_update_time: 0.8262917995, other_time: 13.4683227539
epoch time: 78.8641967773s, input_fired_time: 0.7884319425, output_fired_time: 0.0000000000, propagate_forward_time: 75.8978729248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1778945923
validation[          , genome  6241] predictions:    5717/  10000 (57.17%), best:    5750/10000 (57.50%), error:     13015.48828, best error:     12953.91309 on epoch:    30, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 1039.1269531250s, input_fired_time: 7.9535841942, output_fired_time: 5.3140125275, propagate_forward_time: 373.1459045410, propagate_backward_time: 639.3960571289, weight_update_time: 0.8190253973, other_time: 13.3174438477
epoch time: 77.7264480591s, input_fired_time: 0.8238968849, output_fired_time: 0.0000000000, propagate_forward_time: 73.6444244385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2581253052
validation[          , genome  6241] predictions:    5690/  10000 (56.90%), best:    5750/10000 (57.50%), error:     13443.99902, best error:     12953.91309 on epoch:    30, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 1044.0675048828s, input_fired_time: 7.1343522072, output_fired_time: 6.4560794830, propagate_forward_time: 369.6663513184, propagate_backward_time: 643.1322631836, weight_update_time: 0.8453229070, other_time: 17.6784057617
epoch time: 83.2280044556s, input_fired_time: 0.8964290619, output_fired_time: 0.0000000000, propagate_forward_time: 79.9496307373, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3819427490
validation[          , genome  6241] predictions:    5551/  10000 (55.51%), best:    5750/10000 (57.50%), error:     13833.25391, best error:     12953.91309 on epoch:    30, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 1066.8322753906s, input_fired_time: 8.3370532990, output_fired_time: 5.6234807968, propagate_forward_time: 392.1339721680, propagate_backward_time: 647.5411376953, weight_update_time: 0.8711184859, other_time: 13.1965942383
epoch time: 80.8851470947s, input_fired_time: 1.9131453037, output_fired_time: 0.0000000000, propagate_forward_time: 76.6704330444, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3015670776
validation[          , genome  6241] predictions:    5583/  10000 (55.83%), best:    5750/10000 (57.50%), error:     13566.20410, best error:     12953.91309 on epoch:    30, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 1053.0340576172s, input_fired_time: 9.2778606415, output_fired_time: 5.5119204521, propagate_forward_time: 380.9714355469, propagate_backward_time: 643.3775024414, weight_update_time: 0.8669284582, other_time: 13.8953247070
epoch time: 80.9526672363s, input_fired_time: 0.9064181447, output_fired_time: 0.0000000000, propagate_forward_time: 77.6825790405, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3636703491
validation[          , genome  6241] predictions:    5802/  10000 (58.02%), best:    5802/10000 (58.02%), error:     12831.03125, best error:     12831.03125 on epoch:    36, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 1057.6904296875s, input_fired_time: 7.2607936859, output_fired_time: 7.6478586197, propagate_forward_time: 383.8914794922, propagate_backward_time: 645.8746948242, weight_update_time: 0.8546676040, other_time: 13.0156860352
epoch time: 80.5351715088s, input_fired_time: 0.9078480601, output_fired_time: 0.0000000000, propagate_forward_time: 77.2251586914, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4021682739
validation[          , genome  6241] predictions:    5690/  10000 (56.90%), best:    5802/10000 (58.02%), error:     12856.35742, best error:     12831.03125 on epoch:    36, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 1052.0437011719s, input_fired_time: 8.2383899689, output_fired_time: 6.4952449799, propagate_forward_time: 380.0574645996, propagate_backward_time: 643.3045043945, weight_update_time: 0.8664528728, other_time: 13.9480590820
epoch time: 78.6290588379s, input_fired_time: 0.8656445146, output_fired_time: 0.0000000000, propagate_forward_time: 75.5408935547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2225189209
validation[          , genome  6241] predictions:    5700/  10000 (57.00%), best:    5802/10000 (58.02%), error:     13079.95410, best error:     12831.03125 on epoch:    36, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 1043.7989501953s, input_fired_time: 7.1394319534, output_fired_time: 5.5260348320, propagate_forward_time: 378.7971191406, propagate_backward_time: 635.5518188477, weight_update_time: 0.8470469117, other_time: 16.7846069336
epoch time: 80.7445220947s, input_fired_time: 0.8786360621, output_fired_time: 0.0000000000, propagate_forward_time: 77.5753402710, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2905426025
validation[          , genome  6241] predictions:    5889/  10000 (58.89%), best:    5889/10000 (58.89%), error:     12429.35840, best error:     12429.35840 on epoch:    39, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 1043.7779541016s, input_fired_time: 8.1666927338, output_fired_time: 5.4638195038, propagate_forward_time: 380.6738281250, propagate_backward_time: 636.6417846680, weight_update_time: 0.9123798609, other_time: 12.8317260742
epoch time: 81.6039581299s, input_fired_time: 0.9231603146, output_fired_time: 0.0000000000, propagate_forward_time: 78.3473358154, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3334655762
validation[          , genome  6241] predictions:    5885/  10000 (58.85%), best:    5889/10000 (58.89%), error:     12487.21973, best error:     12429.35840 on epoch:    39, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 1055.3222656250s, input_fired_time: 8.2263231277, output_fired_time: 6.5145463943, propagate_forward_time: 386.3698425293, propagate_backward_time: 641.2143554688, weight_update_time: 0.8675263524, other_time: 12.9971923828
epoch time: 82.6633529663s, input_fired_time: 0.9265155792, output_fired_time: 0.0000000000, propagate_forward_time: 79.4013824463, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3354568481
validation[          , genome  6241] predictions:    5955/  10000 (59.55%), best:    5955/10000 (59.55%), error:     12195.79883, best error:     12195.79883 on epoch:    41, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 1047.5345458984s, input_fired_time: 8.1951646805, output_fired_time: 5.4577927589, propagate_forward_time: 378.0537414551, propagate_backward_time: 641.8579711914, weight_update_time: 0.8690654635, other_time: 13.9699096680
epoch time: 80.8384857178s, input_fired_time: 0.9230030179, output_fired_time: 0.0000000000, propagate_forward_time: 75.5638427734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3516387939
validation[          , genome  6241] predictions:    5802/  10000 (58.02%), best:    5955/10000 (59.55%), error:     12689.83887, best error:     12195.79883 on epoch:    41, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 1054.2351074219s, input_fired_time: 7.3016676903, output_fired_time: 5.6066713333, propagate_forward_time: 388.3478393555, propagate_backward_time: 640.0750732422, weight_update_time: 0.8575857878, other_time: 12.9038696289
epoch time: 77.3078613281s, input_fired_time: 0.8562416434, output_fired_time: 0.0000000000, propagate_forward_time: 73.2551116943, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1965103149
validation[          , genome  6241] predictions:    5888/  10000 (58.88%), best:    5955/10000 (59.55%), error:     12378.87891, best error:     12195.79883 on epoch:    41, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 1051.1862792969s, input_fired_time: 7.2424292564, output_fired_time: 6.4888029099, propagate_forward_time: 379.1878356934, propagate_backward_time: 642.2975463867, weight_update_time: 0.8977367878, other_time: 15.9696655273
epoch time: 78.7308578491s, input_fired_time: 0.9052886367, output_fired_time: 0.0000000000, propagate_forward_time: 75.5550689697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2705001831
validation[          , genome  6241] predictions:    5866/  10000 (58.66%), best:    5955/10000 (59.55%), error:     12672.17773, best error:     12195.79883 on epoch:    41, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 1057.5003662109s, input_fired_time: 8.3203897476, output_fired_time: 6.5214138031, propagate_forward_time: 384.7801818848, propagate_backward_time: 644.8606567383, weight_update_time: 0.8666863441, other_time: 13.0177612305
epoch time: 78.4300079346s, input_fired_time: 0.9020039439, output_fired_time: 0.0000000000, propagate_forward_time: 75.3035354614, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2244720459
validation[          , genome  6241] predictions:    5960/  10000 (59.60%), best:    5955/10000 (59.55%), error:     12338.87012, best error:     12195.79883 on epoch:    41, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 1055.5976562500s, input_fired_time: 8.2564754486, output_fired_time: 6.5721898079, propagate_forward_time: 382.0783081055, propagate_backward_time: 644.6082153320, weight_update_time: 0.9046211243, other_time: 14.0825195312
epoch time: 80.3419418335s, input_fired_time: 0.8865458369, output_fired_time: 0.0000000000, propagate_forward_time: 77.1648635864, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2905349731
validation[          , genome  6241] predictions:    5855/  10000 (58.55%), best:    5955/10000 (59.55%), error:     12476.94922, best error:     12195.79883 on epoch:    41, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 1048.6778564453s, input_fired_time: 8.1960268021, output_fired_time: 5.4266767502, propagate_forward_time: 375.8248901367, propagate_backward_time: 644.2091064453, weight_update_time: 0.8431665301, other_time: 15.0211791992
epoch time: 78.8350830078s, input_fired_time: 0.8588027358, output_fired_time: 0.0000000000, propagate_forward_time: 74.6744308472, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3018493652
validation[          , genome  6241] predictions:    5820/  10000 (58.20%), best:    5955/10000 (59.55%), error:     12827.17480, best error:     12195.79883 on epoch:    41, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 1036.7152099609s, input_fired_time: 8.0260181427, output_fired_time: 5.3627867699, propagate_forward_time: 376.9071960449, propagate_backward_time: 632.9032592773, weight_update_time: 0.8273724318, other_time: 13.5159301758
epoch time: 80.8598403931s, input_fired_time: 0.8824645281, output_fired_time: 0.0000000000, propagate_forward_time: 75.7735824585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2037963867
validation[          , genome  6241] predictions:    5884/  10000 (58.84%), best:    5955/10000 (59.55%), error:     12555.02832, best error:     12195.79883 on epoch:    41, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 1024.6176757812s, input_fired_time: 7.9537501335, output_fired_time: 5.2922062874, propagate_forward_time: 368.4147033691, propagate_backward_time: 630.8376464844, weight_update_time: 0.8138788939, other_time: 12.1193847656
epoch time: 78.6972808838s, input_fired_time: 0.9038497806, output_fired_time: 0.0000000000, propagate_forward_time: 75.6193008423, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1741333008
validation[          , genome  6241] predictions:    5861/  10000 (58.61%), best:    5955/10000 (59.55%), error:     12860.82129, best error:     12195.79883 on epoch:    41, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 1031.4807128906s, input_fired_time: 7.9714202881, output_fired_time: 6.3400006294, propagate_forward_time: 372.0468139648, propagate_backward_time: 632.6766967773, weight_update_time: 0.8309946656, other_time: 12.4457397461
epoch time: 73.4883956909s, input_fired_time: 0.7852243185, output_fired_time: 0.0000000000, propagate_forward_time: 70.6525878906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0505828857
validation[          , genome  6241] predictions:    5848/  10000 (58.48%), best:    5955/10000 (59.55%), error:     12771.85352, best error:     12195.79883 on epoch:    41, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 1048.0655517578s, input_fired_time: 8.2190275192, output_fired_time: 5.5951957703, propagate_forward_time: 380.1185913086, propagate_backward_time: 640.3383178711, weight_update_time: 0.8532760739, other_time: 13.7944335938
epoch time: 78.3587875366s, input_fired_time: 0.8962457180, output_fired_time: 0.0000000000, propagate_forward_time: 75.2112884521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2512512207
validation[          , genome  6241] predictions:    5763/  10000 (57.63%), best:    5955/10000 (59.55%), error:     12542.45020, best error:     12195.79883 on epoch:    41, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 1046.1229248047s, input_fired_time: 7.3224873543, output_fired_time: 6.4828772545, propagate_forward_time: 378.9588012695, propagate_backward_time: 639.5673828125, weight_update_time: 0.8494832516, other_time: 13.7913208008
epoch time: 80.6013641357s, input_fired_time: 0.8963543773, output_fired_time: 0.0000000000, propagate_forward_time: 77.3757400513, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3292694092
validation[          , genome  6241] predictions:    5973/  10000 (59.73%), best:    5955/10000 (59.55%), error:     12373.33008, best error:     12195.79883 on epoch:    41, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 1050.1711425781s, input_fired_time: 8.3921003342, output_fired_time: 7.4697690010, propagate_forward_time: 386.9861450195, propagate_backward_time: 633.5422973633, weight_update_time: 0.8610422611, other_time: 13.7808837891
epoch time: 78.6212692261s, input_fired_time: 1.9110733271, output_fired_time: 0.0000000000, propagate_forward_time: 74.4579467773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2522506714
validation[          , genome  6241] predictions:    5941/  10000 (59.41%), best:    5955/10000 (59.55%), error:     12388.53613, best error:     12195.79883 on epoch:    41, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 1032.6768798828s, input_fired_time: 7.1605467796, output_fired_time: 5.3670954704, propagate_forward_time: 374.2402343750, propagate_backward_time: 633.4327392578, weight_update_time: 0.8853020072, other_time: 12.4763183594
epoch time: 79.5505447388s, input_fired_time: 0.8905571103, output_fired_time: 0.0000000000, propagate_forward_time: 76.4187011719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2412872314
validation[          , genome  6241] predictions:    5982/  10000 (59.82%), best:    5982/10000 (59.82%), error:     12087.25195, best error:     12087.25195 on epoch:    54, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 1060.6375732422s, input_fired_time: 10.3701820374, output_fired_time: 6.5601820946, propagate_forward_time: 381.7357788086, propagate_backward_time: 646.8694458008, weight_update_time: 0.8495886326, other_time: 15.1019287109
epoch time: 81.2978591919s, input_fired_time: 0.9581407905, output_fired_time: 0.0000000000, propagate_forward_time: 78.1117477417, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2279739380
validation[          , genome  6241] predictions:    6009/  10000 (60.09%), best:    5982/10000 (59.82%), error:     12118.45215, best error:     12087.25195 on epoch:    54, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 1059.7044677734s, input_fired_time: 7.4264993668, output_fired_time: 7.4308872223, propagate_forward_time: 386.3173522949, propagate_backward_time: 644.5000000000, weight_update_time: 0.8571395874, other_time: 14.0296630859
epoch time: 79.9363479614s, input_fired_time: 0.9002856612, output_fired_time: 0.0000000000, propagate_forward_time: 75.7798156738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2562484741
validation[          , genome  6241] predictions:    5948/  10000 (59.48%), best:    5982/10000 (59.82%), error:     12439.44238, best error:     12087.25195 on epoch:    54, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 1060.4056396484s, input_fired_time: 7.3304872513, output_fired_time: 7.5721635818, propagate_forward_time: 387.8668518066, propagate_backward_time: 643.6320190430, weight_update_time: 0.8908007145, other_time: 14.0042114258
epoch time: 79.4210662842s, input_fired_time: 0.9065226912, output_fired_time: 0.0000000000, propagate_forward_time: 76.2284851074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2860565186
validation[          , genome  6241] predictions:    5976/  10000 (59.76%), best:    5982/10000 (59.82%), error:     12146.68262, best error:     12087.25195 on epoch:    54, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 1061.5379638672s, input_fired_time: 8.4294023514, output_fired_time: 5.5476946831, propagate_forward_time: 388.0542297363, propagate_backward_time: 644.2468872070, weight_update_time: 0.8869646788, other_time: 15.2597045898
epoch time: 81.1190261841s, input_fired_time: 0.8902093768, output_fired_time: 0.0000000000, propagate_forward_time: 77.9645843506, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2642288208
validation[          , genome  6241] predictions:    6078/  10000 (60.78%), best:    6078/10000 (60.78%), error:     11928.85156, best error:     11928.85156 on epoch:    58, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 1041.3972167969s, input_fired_time: 8.2056217194, output_fired_time: 7.3794603348, propagate_forward_time: 380.7961730957, propagate_backward_time: 631.4617309570, weight_update_time: 0.8263823986, other_time: 13.5542602539
epoch time: 80.7771606445s, input_fired_time: 1.8910614252, output_fired_time: 0.0000000000, propagate_forward_time: 75.6508255005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2352752686
validation[          , genome  6241] predictions:    6036/  10000 (60.36%), best:    6036/10000 (60.36%), error:     11818.97754, best error:     11818.97754 on epoch:    59, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 1029.5477294922s, input_fired_time: 7.1854705811, output_fired_time: 6.3163442612, propagate_forward_time: 374.7569580078, propagate_backward_time: 628.8199462891, weight_update_time: 0.8110679984, other_time: 12.4689941406
epoch time: 79.5022659302s, input_fired_time: 0.8737899065, output_fired_time: 0.0000000000, propagate_forward_time: 76.4241485596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2043304443
validation[          , genome  6241] predictions:    6010/  10000 (60.10%), best:    6036/10000 (60.36%), error:     12118.22266, best error:     11818.97754 on epoch:    59, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 1028.8121337891s, input_fired_time: 8.0377264023, output_fired_time: 5.2878131866, propagate_forward_time: 380.3676147461, propagate_backward_time: 619.6917724609, weight_update_time: 0.8287035823, other_time: 15.4271850586
epoch time: 79.6664199829s, input_fired_time: 0.8902809024, output_fired_time: 0.0000000000, propagate_forward_time: 76.5204849243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2556533813
validation[          , genome  6241] predictions:    5988/  10000 (59.88%), best:    6036/10000 (60.36%), error:     12147.33594, best error:     11818.97754 on epoch:    59, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 1029.9890136719s, input_fired_time: 8.0730257034, output_fired_time: 5.3067908287, propagate_forward_time: 378.4972534180, propagate_backward_time: 625.5819702148, weight_update_time: 0.8326014280, other_time: 12.5300292969
epoch time: 76.3794403076s, input_fired_time: 1.8258769512, output_fired_time: 0.0000000000, propagate_forward_time: 72.4185256958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1350402832
validation[          , genome  6241] predictions:    5948/  10000 (59.48%), best:    6036/10000 (60.36%), error:     12225.95898, best error:     11818.97754 on epoch:    59, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 1005.5175781250s, input_fired_time: 8.0763063431, output_fired_time: 5.2345290184, propagate_forward_time: 372.0512695312, propagate_backward_time: 607.7021484375, weight_update_time: 0.8521795869, other_time: 12.4533081055
epoch time: 81.1581039429s, input_fired_time: 0.9315485358, output_fired_time: 0.0000000000, propagate_forward_time: 78.0180969238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2084579468
validation[          , genome  6241] predictions:    5996/  10000 (59.96%), best:    6036/10000 (60.36%), error:     11951.36914, best error:     11818.97754 on epoch:    59, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 1028.1357421875s, input_fired_time: 7.0933523178, output_fired_time: 5.2402071953, propagate_forward_time: 375.4566955566, propagate_backward_time: 624.8532104492, weight_update_time: 0.8816530704, other_time: 15.4922485352
epoch time: 79.7366790771s, input_fired_time: 0.9047973156, output_fired_time: 0.0000000000, propagate_forward_time: 76.5593566895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2725219727
validation[          , genome  6241] predictions:    6095/  10000 (60.95%), best:    6095/10000 (60.95%), error:     11790.40332, best error:     11790.40332 on epoch:    64, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 1023.3492431641s, input_fired_time: 7.0899901390, output_fired_time: 5.2999005318, propagate_forward_time: 379.5795288086, propagate_backward_time: 618.0927734375, weight_update_time: 0.8393548727, other_time: 13.2870483398
epoch time: 80.0461959839s, input_fired_time: 0.8768117428, output_fired_time: 0.0000000000, propagate_forward_time: 76.9369506836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2324371338
validation[          , genome  6241] predictions:    6097/  10000 (60.97%), best:    6095/10000 (60.95%), error:     11794.97266, best error:     11790.40332 on epoch:    64, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 1038.8395996094s, input_fired_time: 8.2071504593, output_fired_time: 5.3970675468, propagate_forward_time: 385.6674804688, propagate_backward_time: 625.0572509766, weight_update_time: 0.8085668087, other_time: 14.5106201172
epoch time: 79.0315017700s, input_fired_time: 0.8707168698, output_fired_time: 0.0000000000, propagate_forward_time: 75.8655242920, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2952575684
validation[          , genome  6241] predictions:    6010/  10000 (60.10%), best:    6095/10000 (60.95%), error:     12135.28516, best error:     11790.40332 on epoch:    64, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 1021.6235961914s, input_fired_time: 7.1724119186, output_fired_time: 5.3188366890, propagate_forward_time: 369.0238647461, propagate_backward_time: 626.6292724609, weight_update_time: 0.8226858974, other_time: 13.4791870117
epoch time: 77.6820068359s, input_fired_time: 0.8679229617, output_fired_time: 0.0000000000, propagate_forward_time: 73.6690750122, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1450119019
validation[          , genome  6241] predictions:    6136/  10000 (61.36%), best:    6136/10000 (61.36%), error:     11747.63281, best error:     11747.63281 on epoch:    67, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 1042.5629882812s, input_fired_time: 7.1254181862, output_fired_time: 6.3393616676, propagate_forward_time: 390.4570922852, propagate_backward_time: 626.1196899414, weight_update_time: 0.8427684307, other_time: 12.5214843750
epoch time: 78.2620620728s, input_fired_time: 0.8979496360, output_fired_time: 0.0000000000, propagate_forward_time: 75.0961532593, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2679595947
validation[          , genome  6241] predictions:    6037/  10000 (60.37%), best:    6136/10000 (61.36%), error:     11952.90430, best error:     11747.63281 on epoch:    67, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 1038.5150146484s, input_fired_time: 9.2432069778, output_fired_time: 6.4015955925, propagate_forward_time: 377.5832519531, propagate_backward_time: 631.8796997070, weight_update_time: 0.8368143439, other_time: 13.4072875977
epoch time: 78.4634475708s, input_fired_time: 0.8815891743, output_fired_time: 0.0000000000, propagate_forward_time: 75.4261398315, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1557159424
validation[          , genome  6241] predictions:    6109/  10000 (61.09%), best:    6136/10000 (61.36%), error:     11771.66797, best error:     11747.63281 on epoch:    67, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 1032.1890869141s, input_fired_time: 8.3284530640, output_fired_time: 5.3654508591, propagate_forward_time: 371.6105346680, propagate_backward_time: 631.3807983398, weight_update_time: 0.8384268880, other_time: 15.5038452148
epoch time: 79.3356552124s, input_fired_time: 0.9120107293, output_fired_time: 0.0000000000, propagate_forward_time: 76.1235809326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3000640869
validation[          , genome  6241] predictions:    6117/  10000 (61.17%), best:    6136/10000 (61.36%), error:     12013.94434, best error:     11747.63281 on epoch:    67, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 1044.9667968750s, input_fired_time: 8.2742996216, output_fired_time: 5.3436541557, propagate_forward_time: 383.2604675293, propagate_backward_time: 634.5435791016, weight_update_time: 0.8323537707, other_time: 13.5447998047
epoch time: 78.8468780518s, input_fired_time: 0.8655753732, output_fired_time: 0.0000000000, propagate_forward_time: 73.7505111694, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2307891846
validation[          , genome  6241] predictions:    6096/  10000 (60.96%), best:    6136/10000 (61.36%), error:     11813.65625, best error:     11747.63281 on epoch:    67, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 1021.4106445312s, input_fired_time: 7.2015142441, output_fired_time: 5.3535122871, propagate_forward_time: 372.2498474121, propagate_backward_time: 624.2775878906, weight_update_time: 0.8511547446, other_time: 12.3281250000
epoch time: 80.1852645874s, input_fired_time: 0.9079009295, output_fired_time: 0.0000000000, propagate_forward_time: 77.0203094482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2570571899
validation[          , genome  6241] predictions:    6011/  10000 (60.11%), best:    6136/10000 (61.36%), error:     12041.13672, best error:     11747.63281 on epoch:    67, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 1041.8437500000s, input_fired_time: 7.2393236160, output_fired_time: 5.2732648849, propagate_forward_time: 379.1743469238, propagate_backward_time: 635.6159667969, weight_update_time: 0.8289529681, other_time: 14.5407714844
epoch time: 76.5513839722s, input_fired_time: 0.8764174581, output_fired_time: 0.0000000000, propagate_forward_time: 73.4562988281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2186660767
validation[          , genome  6241] predictions:    6034/  10000 (60.34%), best:    6136/10000 (61.36%), error:     12154.73926, best error:     11747.63281 on epoch:    67, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 1041.2720947266s, input_fired_time: 8.3584690094, output_fired_time: 5.3732109070, propagate_forward_time: 384.9692077637, propagate_backward_time: 628.4333496094, weight_update_time: 0.8998093605, other_time: 14.1378173828
epoch time: 80.7888488770s, input_fired_time: 0.9502589107, output_fired_time: 0.0000000000, propagate_forward_time: 77.4492111206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3893814087
validation[          , genome  6241] predictions:    6114/  10000 (61.14%), best:    6136/10000 (61.36%), error:     11789.36035, best error:     11747.63281 on epoch:    67, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 1057.7341308594s, input_fired_time: 7.5438833237, output_fired_time: 5.6120738983, propagate_forward_time: 392.3565368652, propagate_backward_time: 639.1806640625, weight_update_time: 0.8889310360, other_time: 13.0410156250
epoch time: 79.8268203735s, input_fired_time: 0.9196519852, output_fired_time: 0.0000000000, propagate_forward_time: 76.5880432129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3191223145
validation[          , genome  6241] predictions:    6175/  10000 (61.75%), best:    6175/10000 (61.75%), error:     11669.91895, best error:     11669.91895 on epoch:    75, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 1053.1231689453s, input_fired_time: 8.4486541748, output_fired_time: 8.5693368912, propagate_forward_time: 379.3669738770, propagate_backward_time: 643.7403564453, weight_update_time: 0.8604223132, other_time: 12.9979248047
epoch time: 77.9214172363s, input_fired_time: 0.8950166106, output_fired_time: 0.0000000000, propagate_forward_time: 74.7592468262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2671508789
validation[          , genome  6241] predictions:    6097/  10000 (60.97%), best:    6175/10000 (61.75%), error:     11750.24219, best error:     11669.91895 on epoch:    75, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 1039.2971191406s, input_fired_time: 7.5094814301, output_fired_time: 5.5624074936, propagate_forward_time: 367.4211730957, propagate_backward_time: 643.8797607422, weight_update_time: 0.8643066883, other_time: 14.9243164062
epoch time: 80.3179092407s, input_fired_time: 0.9645185471, output_fired_time: 0.0000000000, propagate_forward_time: 77.0104904175, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3429031372
validation[          , genome  6241] predictions:    6180/  10000 (61.80%), best:    6180/10000 (61.80%), error:     11655.39160, best error:     11655.39160 on epoch:    77, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 1062.1999511719s, input_fired_time: 8.6673221588, output_fired_time: 6.7252645493, propagate_forward_time: 387.7001647949, propagate_backward_time: 645.3333129883, weight_update_time: 0.9377591014, other_time: 13.7738647461
epoch time: 81.0548934937s, input_fired_time: 0.9738811851, output_fired_time: 0.0000000000, propagate_forward_time: 77.6530532837, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4279556274
validation[          , genome  6241] predictions:    6119/  10000 (61.19%), best:    6180/10000 (61.80%), error:     11693.55566, best error:     11655.39160 on epoch:    77, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 1054.2666015625s, input_fired_time: 8.6889524460, output_fired_time: 5.6903133392, propagate_forward_time: 374.1726684570, propagate_backward_time: 647.8701171875, weight_update_time: 0.9403347969, other_time: 17.8445434570
epoch time: 78.5123748779s, input_fired_time: 0.9307101965, output_fired_time: 0.0000000000, propagate_forward_time: 75.2014770508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3801879883
validation[          , genome  6241] predictions:    6156/  10000 (61.56%), best:    6180/10000 (61.80%), error:     11659.28516, best error:     11655.39160 on epoch:    77, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 1048.9494628906s, input_fired_time: 7.6378064156, output_fired_time: 5.6185798645, propagate_forward_time: 381.0370483398, propagate_backward_time: 638.1163940430, weight_update_time: 0.9210180640, other_time: 16.5396728516
epoch time: 79.4113616943s, input_fired_time: 1.0158330202, output_fired_time: 0.0000000000, propagate_forward_time: 75.8838424683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5116882324
validation[          , genome  6241] predictions:    6169/  10000 (61.69%), best:    6169/10000 (61.69%), error:     11612.24805, best error:     11612.24805 on epoch:    80, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 1061.6444091797s, input_fired_time: 7.6930270195, output_fired_time: 6.7516293526, propagate_forward_time: 393.0852355957, propagate_backward_time: 639.3939208984, weight_update_time: 0.9141266346, other_time: 14.7207031250
epoch time: 78.9731521606s, input_fired_time: 0.9549933672, output_fired_time: 0.0000000000, propagate_forward_time: 75.5824966431, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4356613159
validation[          , genome  6241] predictions:    6113/  10000 (61.13%), best:    6169/10000 (61.69%), error:     11957.28418, best error:     11612.24805 on epoch:    80, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 1058.3874511719s, input_fired_time: 8.7136840820, output_fired_time: 5.6756300926, propagate_forward_time: 388.4790344238, propagate_backward_time: 639.7294311523, weight_update_time: 0.9473988414, other_time: 15.7897338867
epoch time: 81.4596176147s, input_fired_time: 0.9958984256, output_fired_time: 0.0000000000, propagate_forward_time: 76.0664062500, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3973159790
validation[          , genome  6241] predictions:    6113/  10000 (61.13%), best:    6169/10000 (61.69%), error:     11778.94238, best error:     11612.24805 on epoch:    80, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 1055.2584228516s, input_fired_time: 9.5758085251, output_fired_time: 7.6859602928, propagate_forward_time: 380.1424255371, propagate_backward_time: 641.2210083008, weight_update_time: 0.9084537625, other_time: 16.6332397461
epoch time: 82.3992996216s, input_fired_time: 1.0346907377, output_fired_time: 0.0000000000, propagate_forward_time: 78.8363189697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5282897949
validation[          , genome  6241] predictions:    6190/  10000 (61.90%), best:    6190/10000 (61.90%), error:     11605.61719, best error:     11605.61719 on epoch:    83, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 1060.3593750000s, input_fired_time: 9.7975597382, output_fired_time: 7.8434262276, propagate_forward_time: 386.9299621582, propagate_backward_time: 639.8211669922, weight_update_time: 0.9208517671, other_time: 15.9672851562
epoch time: 82.2479400635s, input_fired_time: 1.0364239216, output_fired_time: 0.0000000000, propagate_forward_time: 77.7359695435, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4755477905
validation[          , genome  6241] predictions:    6176/  10000 (61.76%), best:    6190/10000 (61.90%), error:     11724.59766, best error:     11605.61719 on epoch:    83, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 1053.7677001953s, input_fired_time: 7.7762341499, output_fired_time: 8.7318687439, propagate_forward_time: 378.5277099609, propagate_backward_time: 643.9072875977, weight_update_time: 0.9033618569, other_time: 14.8246459961
epoch time: 79.2720718384s, input_fired_time: 0.9575552940, output_fired_time: 0.0000000000, propagate_forward_time: 74.9213104248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3932037354
validation[          , genome  6241] predictions:    6149/  10000 (61.49%), best:    6190/10000 (61.90%), error:     11701.59082, best error:     11605.61719 on epoch:    83, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 1050.3566894531s, input_fired_time: 8.6250391006, output_fired_time: 7.6917963028, propagate_forward_time: 382.6024780273, propagate_backward_time: 635.7913208008, weight_update_time: 0.9460918307, other_time: 15.6461181641
epoch time: 78.5222702026s, input_fired_time: 0.9580884576, output_fired_time: 0.0000000000, propagate_forward_time: 75.2141342163, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3500442505
validation[          , genome  6241] predictions:    6122/  10000 (61.22%), best:    6190/10000 (61.90%), error:     11903.74609, best error:     11605.61719 on epoch:    83, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 1058.4520263672s, input_fired_time: 9.7306737900, output_fired_time: 5.8547124863, propagate_forward_time: 387.4853210449, propagate_backward_time: 639.5753784180, weight_update_time: 0.9685266614, other_time: 15.8058471680
epoch time: 79.2241363525s, input_fired_time: 0.9566290975, output_fired_time: 0.0000000000, propagate_forward_time: 75.9000930786, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3674163818
validation[          , genome  6241] predictions:    6146/  10000 (61.46%), best:    6190/10000 (61.90%), error:     11690.22754, best error:     11605.61719 on epoch:    83, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 1066.4200439453s, input_fired_time: 7.9861760139, output_fired_time: 6.8789668083, propagate_forward_time: 388.2474670410, propagate_backward_time: 649.4459838867, weight_update_time: 0.9280394912, other_time: 13.8615112305
epoch time: 79.5770721436s, input_fired_time: 0.9789685607, output_fired_time: 0.0000000000, propagate_forward_time: 76.1230316162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4750747681
validation[          , genome  6241] predictions:    6168/  10000 (61.68%), best:    6190/10000 (61.90%), error:     11743.47754, best error:     11605.61719 on epoch:    83, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 1051.7327880859s, input_fired_time: 7.6401281357, output_fired_time: 5.7221179008, propagate_forward_time: 380.3365173340, propagate_backward_time: 644.3604736328, weight_update_time: 0.9410604835, other_time: 13.6734619141
epoch time: 80.3888092041s, input_fired_time: 0.9911370873, output_fired_time: 0.0000000000, propagate_forward_time: 76.9869155884, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4107589722
validation[          , genome  6241] predictions:    6166/  10000 (61.66%), best:    6166/10000 (61.66%), error:     11588.62891, best error:     11588.62891 on epoch:    89, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 1056.9821777344s, input_fired_time: 8.7333383560, output_fired_time: 6.6850485802, propagate_forward_time: 383.4753723145, propagate_backward_time: 642.3102416992, weight_update_time: 0.9199265838, other_time: 15.7781372070
epoch time: 81.9411010742s, input_fired_time: 1.0059332848, output_fired_time: 0.0000000000, propagate_forward_time: 78.4594879150, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4756774902
validation[          , genome  6241] predictions:    6140/  10000 (61.40%), best:    6166/10000 (61.66%), error:     11763.02734, best error:     11588.62891 on epoch:    89, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 1051.7637939453s, input_fired_time: 7.7294898033, output_fired_time: 6.7165188789, propagate_forward_time: 375.1036987305, propagate_backward_time: 648.6321411133, weight_update_time: 0.8982542157, other_time: 13.5819091797
epoch time: 80.0526885986s, input_fired_time: 1.0003216267, output_fired_time: 0.0000000000, propagate_forward_time: 75.5646362305, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4877319336
validation[          , genome  6241] predictions:    6186/  10000 (61.86%), best:    6166/10000 (61.66%), error:     11615.44238, best error:     11588.62891 on epoch:    89, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 1049.3889160156s, input_fired_time: 8.7388963699, output_fired_time: 5.6485738754, propagate_forward_time: 388.7951660156, propagate_backward_time: 629.4562377930, weight_update_time: 0.9304322600, other_time: 16.7500610352
epoch time: 78.4415817261s, input_fired_time: 0.9776048064, output_fired_time: 0.0000000000, propagate_forward_time: 75.1042861938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3596878052
validation[          , genome  6241] predictions:    6161/  10000 (61.61%), best:    6166/10000 (61.66%), error:     11614.54199, best error:     11588.62891 on epoch:    89, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 1049.2429199219s, input_fired_time: 7.6591548920, output_fired_time: 9.6775255203, propagate_forward_time: 377.6034240723, propagate_backward_time: 640.5421752930, weight_update_time: 0.9454039931, other_time: 13.7606811523
epoch time: 79.4597396851s, input_fired_time: 1.0135239363, output_fired_time: 0.0000000000, propagate_forward_time: 76.0396194458, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4065933228
validation[          , genome  6241] predictions:    6232/  10000 (62.32%), best:    6232/10000 (62.32%), error:     11550.72656, best error:     11550.72656 on epoch:    93, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 1062.4415283203s, input_fired_time: 8.8086585999, output_fired_time: 5.7836952209, propagate_forward_time: 382.2817993164, propagate_backward_time: 650.7931518555, weight_update_time: 0.9270461798, other_time: 14.7741699219
epoch time: 78.5887451172s, input_fired_time: 1.9665309191, output_fired_time: 0.0000000000, propagate_forward_time: 74.2812728882, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3409423828
validation[          , genome  6241] predictions:    6190/  10000 (61.90%), best:    6232/10000 (62.32%), error:     11623.12207, best error:     11550.72656 on epoch:    93, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 1062.5993652344s, input_fired_time: 7.8135743141, output_fired_time: 6.8376750946, propagate_forward_time: 385.1785278320, propagate_backward_time: 648.7987060547, weight_update_time: 0.9235311151, other_time: 13.9708862305
epoch time: 80.9159545898s, input_fired_time: 0.9990589023, output_fired_time: 0.0000000000, propagate_forward_time: 77.4871520996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4297409058
validation[          , genome  6241] predictions:    6167/  10000 (61.67%), best:    6232/10000 (62.32%), error:     11691.05762, best error:     11550.72656 on epoch:    93, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 1062.8425292969s, input_fired_time: 8.7428607941, output_fired_time: 6.7419571877, propagate_forward_time: 384.3554382324, propagate_backward_time: 647.6022949219, weight_update_time: 0.9373211861, other_time: 15.3999023438
epoch time: 81.7594833374s, input_fired_time: 0.9963214397, output_fired_time: 0.0000000000, propagate_forward_time: 78.2883758545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4747848511
validation[          , genome  6241] predictions:    6193/  10000 (61.93%), best:    6232/10000 (62.32%), error:     11578.52637, best error:     11550.72656 on epoch:    93, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

arguments:
	'../../projects/csgrid.org_csg/exact_client_0.33_x86_64-pc-linux-gnu'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_183_6241.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_183_6241_2_r259152049_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.986616
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 8.63176e-05
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 3.4527e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 97
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 93
read number_validation_images: 10000
read best_validation_predictions: 6232
read best_validation_error: 11550.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6241
read normal distribution: '1 0 0'
generator_str: '1523640780'
read generator: 1523640780
read generated_by_map:
1 crossover 1
reading 72 nodes.
reading 278 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507812
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1061.19030761719s, input_fired_time: 8.75223922729492, output_fired_time: 5.72699213027954, propagate_forward_time: 395.792266845703, propagate_backward_time: 636.966430664062, weight_update_time: 0.926141619682312, other_time: 13.952392578125
epoch time: 80.3711471557617s, input_fired_time: 0.969946026802063, output_fired_time: 0, propagate_forward_time: 75.9790802001953, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.422119140625
validation[          , genome  6241] predictions:    6188/  10000 (61.88%), best:    6232/10000 (62.32%), error:     11634.18164, best error:     11550.72656 on epoch:    93, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 1048.6058349609s, input_fired_time: 7.6160516739, output_fired_time: 5.7700614929, propagate_forward_time: 386.2145080566, propagate_backward_time: 633.2282714844, weight_update_time: 0.9157684445, other_time: 15.7769775391
epoch time: 80.2505035400s, input_fired_time: 0.9883324504, output_fired_time: 0.0000000000, propagate_forward_time: 76.8903427124, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3718261719
validation[          , genome  6241] predictions:    6167/  10000 (61.67%), best:    6232/10000 (62.32%), error:     11661.73047, best error:     11550.72656 on epoch:    93, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 1056.3272705078s, input_fired_time: 7.7824959755, output_fired_time: 5.7130303383, propagate_forward_time: 382.1677856445, propagate_backward_time: 644.9783325195, weight_update_time: 0.9446256757, other_time: 15.6856689453
epoch time: 79.5886001587s, input_fired_time: 0.9804481864, output_fired_time: 0.0000000000, propagate_forward_time: 76.1399230957, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4682312012
validation[          , genome  6241] predictions:    6111/  10000 (61.11%), best:    6232/10000 (62.32%), error:     11723.38477, best error:     11550.72656 on epoch:    93, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 1051.7850341797s, input_fired_time: 7.6786947250, output_fired_time: 5.7768673897, propagate_forward_time: 381.0889892578, propagate_backward_time: 640.4076538086, weight_update_time: 0.9385684729, other_time: 16.8328247070
epoch time: 79.1968460083s, input_fired_time: 1.0158770084, output_fired_time: 0.0000000000, propagate_forward_time: 75.6966781616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4842910767
validation[          , genome  6241] predictions:    6175/  10000 (61.75%), best:    6232/10000 (62.32%), error:     11614.69727, best error:     11550.72656 on epoch:    93, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 397.7518005371s, input_fired_time: 4.9422421455, output_fired_time: 0.0000000000, propagate_forward_time: 380.6615905762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.1479797363
best training[          , genome  6241] predictions:   40148/  50000 (80.30%), best:    6232/10000 (62.32%), error:     30339.32422, best error:     11550.72656 on epoch:    93, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 77.4580688477s, input_fired_time: 1.9464545250, output_fired_time: 0.0000000000, propagate_forward_time: 73.1484527588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3631591797
   testing[          , genome  6241] predictions:    6232/  10000 (62.32%), best:    6232/10000 (62.32%), error:     11550.72656, best error:     11550.72656 on epoch:    93, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
19:31:07 (27752): called boinc_finish(0)

</stderr_txt>
]]>