Nameexact_genome_1542495604_186_6189_0
Workunit4095601
Created18 Nov 2018, 2:05:23 UTC
Sent18 Nov 2018, 2:09:38 UTC
Report deadline21 Feb 2019, 1:45:39 UTC
Received20 Nov 2018, 17:20:53 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time1 days 20 hours 41 min 37 sec
CPU time1 days 18 hours 20 min 30 sec
Validate stateValid
Credit23,745.84
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size544.35 MB
Peak swap size564.15 MB
Peak disk usage10.86 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
 1.1945619583, output_fired_time: 0.0000000000, propagate_forward_time: 117.4014663696, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9535522461
validation[          , genome  6189] predictions:    5651/  10000 (56.51%), best:    5749/10000 (57.49%), error:     12624.41113, best error:     12394.81934 on epoch:    33, epoch:   35/100, mu: 0.9151125550, learning_rate: 0.0000153740, weight_decay: 0.0000871339

epoch time: 1449.9848632813s, input_fired_time: 16.0459976196, output_fired_time: 7.3079380989, propagate_forward_time: 570.1431884766, propagate_backward_time: 841.0120849609, weight_update_time: 0.3986231089, other_time: 15.4755859375
epoch time: 130.6076049805s, input_fired_time: 1.3258264065, output_fired_time: 0.0000000000, propagate_forward_time: 126.2219848633, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0597991943
validation[          , genome  6189] predictions:    5807/  10000 (58.07%), best:    5807/10000 (58.07%), error:     12290.64258, best error:     12290.64258 on epoch:    36, epoch:   36/100, mu: 0.9189735055, learning_rate: 0.0000143883, weight_decay: 0.0000825860

epoch time: 1459.1784667969s, input_fired_time: 16.0370445251, output_fired_time: 7.3770771027, propagate_forward_time: 573.9024658203, propagate_backward_time: 846.4196166992, weight_update_time: 0.4120959044, other_time: 15.4423217773
epoch time: 140.8069000244s, input_fired_time: 1.5084155798, output_fired_time: 0.0000000000, propagate_forward_time: 136.0714874268, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2269897461
validation[          , genome  6189] predictions:    5748/  10000 (57.48%), best:    5807/10000 (58.07%), error:     12400.03027, best error:     12290.64258 on epoch:    36, epoch:   37/100, mu: 0.9226353765, learning_rate: 0.0000134658, weight_decay: 0.0000782754

epoch time: 1465.5560302734s, input_fired_time: 16.1263027191, output_fired_time: 7.3869085312, propagate_forward_time: 575.8580322266, propagate_backward_time: 850.3732299805, weight_update_time: 0.3908748627, other_time: 15.8114624023
epoch time: 117.9566650391s, input_fired_time: 1.1963926554, output_fired_time: 0.0000000000, propagate_forward_time: 114.0253295898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7349395752
validation[          , genome  6189] predictions:    5827/  10000 (58.27%), best:    5827/10000 (58.27%), error:     12124.69043, best error:     12124.69043 on epoch:    38, epoch:   38/100, mu: 0.9261084795, learning_rate: 0.0000126024, weight_decay: 0.0000741898

epoch time: 1441.1873779297s, input_fired_time: 15.8032417297, output_fired_time: 7.2416310310, propagate_forward_time: 566.8690795898, propagate_backward_time: 835.8560180664, weight_update_time: 0.3922275901, other_time: 15.4174804688
epoch time: 116.1854095459s, input_fired_time: 1.1682552099, output_fired_time: 0.0000000000, propagate_forward_time: 112.3395385742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6776123047
validation[          , genome  6189] predictions:    5866/  10000 (58.66%), best:    5827/10000 (58.27%), error:     12255.15430, best error:     12124.69043 on epoch:    38, epoch:   39/100, mu: 0.9294025302, learning_rate: 0.0000117944, weight_decay: 0.0000703175

epoch time: 1473.1624755859s, input_fired_time: 16.1510848999, output_fired_time: 7.4930243492, propagate_forward_time: 578.3095703125, propagate_backward_time: 855.2649536133, weight_update_time: 0.4169454277, other_time: 15.9437866211
epoch time: 116.6949996948s, input_fired_time: 1.1662837267, output_fired_time: 0.0000000000, propagate_forward_time: 112.8063507080, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7223663330
validation[          , genome  6189] predictions:    5660/  10000 (56.60%), best:    5827/10000 (58.27%), error:     12624.61328, best error:     12124.69043 on epoch:    38, epoch:   40/100, mu: 0.9325267076, learning_rate: 0.0000110382, weight_decay: 0.0000666472

epoch time: 1493.8646240234s, input_fired_time: 16.4045257568, output_fired_time: 7.5894136429, propagate_forward_time: 587.0484008789, propagate_backward_time: 866.8670043945, weight_update_time: 0.4031439126, other_time: 15.9553222656
epoch time: 119.6890945435s, input_fired_time: 1.1956914663, output_fired_time: 0.0000000000, propagate_forward_time: 115.6687545776, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8246459961
validation[          , genome  6189] predictions:    5828/  10000 (58.28%), best:    5827/10000 (58.27%), error:     12212.65234, best error:     12124.69043 on epoch:    38, epoch:   41/100, mu: 0.9354898334, learning_rate: 0.0000103305, weight_decay: 0.0000631686

epoch time: 1455.1528320313s, input_fired_time: 15.9090652466, output_fired_time: 7.3147859573, propagate_forward_time: 571.9939575195, propagate_backward_time: 844.3070678711, weight_update_time: 0.4100303352, other_time: 15.6279296875
epoch time: 121.9941711426s, input_fired_time: 1.1823177338, output_fired_time: 0.0000000000, propagate_forward_time: 117.9106063843, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9012451172
validation[          , genome  6189] predictions:    5862/  10000 (58.62%), best:    5862/10000 (58.62%), error:     12087.43750, best error:     12087.43750 on epoch:    42, epoch:   42/100, mu: 0.9383001924, learning_rate: 0.0000096681, weight_decay: 0.0000598715

epoch time: 1472.3698730469s, input_fired_time: 16.2996387482, output_fired_time: 7.3683700562, propagate_forward_time: 578.4544067383, propagate_backward_time: 854.4062500000, weight_update_time: 0.3947913051, other_time: 15.8411254883
epoch time: 118.9967575073s, input_fired_time: 1.2052681446, output_fired_time: 0.0000000000, propagate_forward_time: 114.9363861084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8551025391
validation[          , genome  6189] predictions:    5973/  10000 (59.73%), best:    5973/10000 (59.73%), error:     11966.78711, best error:     11966.78711 on epoch:    43, epoch:   43/100, mu: 0.9409656525, learning_rate: 0.0000090483, weight_decay: 0.0000567465

epoch time: 1469.3549804688s, input_fired_time: 16.2487888336, output_fired_time: 7.4133820534, propagate_forward_time: 577.2401733398, propagate_backward_time: 852.6164550781, weight_update_time: 0.4019722044, other_time: 15.8362426758
epoch time: 124.3677673340s, input_fired_time: 1.2143640518, output_fired_time: 0.0000000000, propagate_forward_time: 120.2051162720, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9482879639
validation[          , genome  6189] predictions:    5860/  10000 (58.60%), best:    5973/10000 (59.73%), error:     12182.97363, best error:     11966.78711 on epoch:    43, epoch:   44/100, mu: 0.9434937239, learning_rate: 0.0000084681, weight_decay: 0.0000537846

epoch time: 1488.0927734375s, input_fired_time: 16.3723754883, output_fired_time: 7.5137758255, propagate_forward_time: 585.0623168945, propagate_backward_time: 863.3679199219, weight_update_time: 0.4493300915, other_time: 15.7764282227
epoch time: 118.5595016479s, input_fired_time: 1.1902555227, output_fired_time: 0.0000000000, propagate_forward_time: 114.6874160767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6818313599
validation[          , genome  6189] predictions:    5899/  10000 (58.99%), best:    5973/10000 (59.73%), error:     12106.40820, best error:     11966.78711 on epoch:    43, epoch:   45/100, mu: 0.9458914399, learning_rate: 0.0000079252, weight_decay: 0.0000509773

epoch time: 1498.1907958984s, input_fired_time: 16.6879062653, output_fired_time: 7.5047864914, propagate_forward_time: 588.4321289063, propagate_backward_time: 869.3364868164, weight_update_time: 0.4216322601, other_time: 16.2295532227
epoch time: 119.7352676392s, input_fired_time: 1.1975651979, output_fired_time: 0.0000000000, propagate_forward_time: 115.7919006348, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7458038330
validation[          , genome  6189] predictions:    5847/  10000 (58.47%), best:    5973/10000 (59.73%), error:     12129.18750, best error:     11966.78711 on epoch:    43, epoch:   46/100, mu: 0.9481655359, learning_rate: 0.0000074170, weight_decay: 0.0000483166

epoch time: 1473.4313964844s, input_fired_time: 16.1062526703, output_fired_time: 7.4331741333, propagate_forward_time: 579.3223266602, propagate_backward_time: 854.9068603516, weight_update_time: 0.4035098255, other_time: 15.6627807617
epoch time: 118.3948135376s, input_fired_time: 1.2174745798, output_fired_time: 0.0000000000, propagate_forward_time: 114.4913253784, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6860122681
validation[          , genome  6189] predictions:    5785/  10000 (57.85%), best:    5973/10000 (59.73%), error:     12217.75391, best error:     11966.78711 on epoch:    43, epoch:   47/100, mu: 0.9503223896, learning_rate: 0.0000069415, weight_decay: 0.0000457947

epoch time: 1505.6228027344s, input_fired_time: 16.6625614166, output_fired_time: 7.6027960777, propagate_forward_time: 592.2178344727, propagate_backward_time: 872.8302612305, weight_update_time: 0.3976385891, other_time: 16.3093261719
epoch time: 116.4890594482s, input_fired_time: 1.1724736691, output_fired_time: 0.0000000000, propagate_forward_time: 112.5736389160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7429504395
validation[          , genome  6189] predictions:    5932/  10000 (59.32%), best:    5932/10000 (59.32%), error:     11941.84863, best error:     11941.84863 on epoch:    48, epoch:   48/100, mu: 0.9523680210, learning_rate: 0.0000064964, weight_decay: 0.0000434044

epoch time: 1498.1309814453s, input_fired_time: 16.6965465546, output_fired_time: 7.6069245338, propagate_forward_time: 588.6069946289, propagate_backward_time: 868.9721679688, weight_update_time: 0.4054142237, other_time: 16.2483520508
epoch time: 129.2014465332s, input_fired_time: 1.3568031788, output_fired_time: 0.0000000000, propagate_forward_time: 124.7798767090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0647659302
validation[          , genome  6189] predictions:    5935/  10000 (59.35%), best:    5935/10000 (59.35%), error:     11904.05273, best error:     11904.05273 on epoch:    49, epoch:   49/100, mu: 0.9543082118, learning_rate: 0.0000060799, weight_decay: 0.0000411389

epoch time: 1464.6929931641s, input_fired_time: 16.0078487396, output_fired_time: 7.3278174400, propagate_forward_time: 575.9385375977, propagate_backward_time: 849.5793457031, weight_update_time: 0.3921951950, other_time: 15.8395385742
epoch time: 125.9277420044s, input_fired_time: 1.3646090031, output_fired_time: 0.0000000000, propagate_forward_time: 121.6176223755, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9455108643
validation[          , genome  6189] predictions:    5958/  10000 (59.58%), best:    5935/10000 (59.35%), error:     11935.37598, best error:     11904.05273 on epoch:    49, epoch:   50/100, mu: 0.9561483860, learning_rate: 0.0000056901, weight_decay: 0.0000389917

epoch time: 1450.5794677734s, input_fired_time: 16.0283451080, output_fired_time: 7.3301596642, propagate_forward_time: 569.3993530273, propagate_backward_time: 842.1579589844, weight_update_time: 0.4275605083, other_time: 15.6636352539
epoch time: 125.2660064697s, input_fired_time: 1.2949583530, output_fired_time: 0.0000000000, propagate_forward_time: 121.0190200806, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9520263672
validation[          , genome  6189] predictions:    5921/  10000 (59.21%), best:    5935/10000 (59.35%), error:     12013.50684, best error:     11904.05273 on epoch:    49, epoch:   51/100, mu: 0.9578936696, learning_rate: 0.0000053253, weight_decay: 0.0000369565

epoch time: 1457.6207275391s, input_fired_time: 16.1736221313, output_fired_time: 7.3789234161, propagate_forward_time: 573.1765136719, propagate_backward_time: 845.2589721680, weight_update_time: 0.4093945622, other_time: 15.6327514648
epoch time: 120.9344711304s, input_fired_time: 1.2673342228, output_fired_time: 0.0000000000, propagate_forward_time: 116.9209594727, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7461776733
validation[          , genome  6189] predictions:    5934/  10000 (59.34%), best:    5935/10000 (59.35%), error:     12044.42773, best error:     11904.05273 on epoch:    49, epoch:   52/100, mu: 0.9595489502, learning_rate: 0.0000049838, weight_decay: 0.0000350276

epoch time: 1454.7835693359s, input_fired_time: 16.1015586853, output_fired_time: 7.4308319092, propagate_forward_time: 571.2180175781, propagate_backward_time: 844.3955078125, weight_update_time: 0.4056766629, other_time: 15.6376953125
epoch time: 128.2964172363s, input_fired_time: 1.2279473543, output_fired_time: 0.0000000000, propagate_forward_time: 123.9461898804, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1222763062
validation[          , genome  6189] predictions:    5957/  10000 (59.57%), best:    5935/10000 (59.35%), error:     11923.48145, best error:     11904.05273 on epoch:    49, epoch:   53/100, mu: 0.9611188769, learning_rate: 0.0000046643, weight_decay: 0.0000331993

epoch time: 1485.0555419922s, input_fired_time: 16.4409904480, output_fired_time: 7.4188518524, propagate_forward_time: 583.8716430664, propagate_backward_time: 861.4767456055, weight_update_time: 0.3927532136, other_time: 15.8472900391
epoch time: 119.2672271729s, input_fired_time: 1.1932040453, output_fired_time: 0.0000000000, propagate_forward_time: 115.2393646240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8346557617
validation[          , genome  6189] predictions:    5892/  10000 (58.92%), best:    5935/10000 (59.35%), error:     12004.13770, best error:     11904.05273 on epoch:    49, epoch:   54/100, mu: 0.9626079202, learning_rate: 0.0000043652, weight_decay: 0.0000314665

epoch time: 1448.6420898438s, input_fired_time: 16.0324783325, output_fired_time: 7.3513436317, propagate_forward_time: 569.2279052734, propagate_backward_time: 839.5828857422, weight_update_time: 0.4206320643, other_time: 16.4475097656
epoch time: 126.8335494995s, input_fired_time: 1.2591159344, output_fired_time: 0.0000000000, propagate_forward_time: 121.9463806152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6280517578
validation[          , genome  6189] predictions:    5884/  10000 (58.84%), best:    5935/10000 (59.35%), error:     12078.21094, best error:     11904.05273 on epoch:    49, epoch:   55/100, mu: 0.9640201926, learning_rate: 0.0000040854, weight_decay: 0.0000298241

epoch time: 1590.4437255859s, input_fired_time: 18.1256160736, output_fired_time: 7.8350048065, propagate_forward_time: 630.8386230469, propagate_backward_time: 898.4427490234, weight_update_time: 0.4922089279, other_time: 35.2017822266
epoch time: 126.0850067139s, input_fired_time: 1.2601323128, output_fired_time: 0.0000000000, propagate_forward_time: 120.6382446289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1866302490
validation[          , genome  6189] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11807.66113, best error:     11807.66113 on epoch:    56, epoch:   56/100, mu: 0.9653596282, learning_rate: 0.0000038234, weight_decay: 0.0000282674

epoch time: 1535.5214843750s, input_fired_time: 17.5408401489, output_fired_time: 7.6094555855, propagate_forward_time: 597.5820922852, propagate_backward_time: 863.4919433594, weight_update_time: 0.4152812362, other_time: 49.2970581055
epoch time: 127.2513961792s, input_fired_time: 1.3703352213, output_fired_time: 0.0000000000, propagate_forward_time: 120.2772903442, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6037673950
validation[          , genome  6189] predictions:    5957/  10000 (59.57%), best:    6049/10000 (60.49%), error:     11944.59961, best error:     11807.66113 on epoch:    56, epoch:   57/100, mu: 0.9666299820, learning_rate: 0.0000035783, weight_decay: 0.0000267920

epoch time: 1543.2944335938s, input_fired_time: 17.5897712708, output_fired_time: 7.6195750237, propagate_forward_time: 601.4895019531, propagate_backward_time: 868.8490600586, weight_update_time: 0.4090489149, other_time: 47.7465209961
epoch time: 126.4949951172s, input_fired_time: 1.2535424232, output_fired_time: 0.0000000000, propagate_forward_time: 121.2556610107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9857940674
validation[          , genome  6189] predictions:    5992/  10000 (59.92%), best:    6049/10000 (60.49%), error:     11816.32422, best error:     11807.66113 on epoch:    56, epoch:   58/100, mu: 0.9678348899, learning_rate: 0.0000033489, weight_decay: 0.0000253936

epoch time: 1511.9285888672s, input_fired_time: 17.0238151550, output_fired_time: 7.5971550941, propagate_forward_time: 603.0032958984, propagate_backward_time: 860.6983642578, weight_update_time: 0.3976270258, other_time: 23.6059570313
epoch time: 123.7680358887s, input_fired_time: 1.2895665169, output_fired_time: 0.0000000000, propagate_forward_time: 119.2324218750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2460479736
validation[          , genome  6189] predictions:    5879/  10000 (58.79%), best:    6049/10000 (60.49%), error:     12071.30273, best error:     11807.66113 on epoch:    56, epoch:   59/100, mu: 0.9689776301, learning_rate: 0.0000031341, weight_decay: 0.0000240682

epoch time: 1545.3759765625s, input_fired_time: 17.7791118622, output_fired_time: 7.8044328690, propagate_forward_time: 611.2068481445, propagate_backward_time: 873.8784790039, weight_update_time: 0.4364921749, other_time: 34.7071533203
epoch time: 124.5217208862s, input_fired_time: 1.2389594316, output_fired_time: 0.0000000000, propagate_forward_time: 115.2745590210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.0082015991
validation[          , genome  6189] predictions:    5906/  10000 (59.06%), best:    6049/10000 (60.49%), error:     11977.10352, best error:     11807.66113 on epoch:    56, epoch:   60/100, mu: 0.9700614810, learning_rate: 0.0000029332, weight_decay: 0.0000228119

epoch time: 1475.6798095703s, input_fired_time: 16.5489559174, output_fired_time: 7.3889322281, propagate_forward_time: 570.7131347656, propagate_backward_time: 845.9367065430, weight_update_time: 0.4098399878, other_time: 35.0921020508
epoch time: 119.9430618286s, input_fired_time: 1.2330379486, output_fired_time: 0.0000000000, propagate_forward_time: 114.9785232544, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7314987183
validation[          , genome  6189] predictions:    5894/  10000 (58.94%), best:    6049/10000 (60.49%), error:     12023.81738, best error:     11807.66113 on epoch:    56, epoch:   61/100, mu: 0.9710894227, learning_rate: 0.0000027451, weight_decay: 0.0000216213

epoch time: 1470.5314941406s, input_fired_time: 16.4107131958, output_fired_time: 7.3703589439, propagate_forward_time: 571.6804199219, propagate_backward_time: 843.1530761719, weight_update_time: 0.4126711786, other_time: 31.9168701172
epoch time: 118.3068542480s, input_fired_time: 1.2044085264, output_fired_time: 0.0000000000, propagate_forward_time: 113.4480819702, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6543655396
validation[          , genome  6189] predictions:    6011/  10000 (60.11%), best:    6011/10000 (60.11%), error:     11783.09180, best error:     11783.09180 on epoch:    62, epoch:   62/100, mu: 0.9720643759, learning_rate: 0.0000025691, weight_decay: 0.0000204927

epoch time: 1527.9104003906s, input_fired_time: 17.0838317871, output_fired_time: 7.7424507141, propagate_forward_time: 586.5009765625, propagate_backward_time: 861.3967285156, weight_update_time: 0.4274072647, other_time: 55.1864013672
epoch time: 123.7033767700s, input_fired_time: 1.3454935551, output_fired_time: 0.0000000000, propagate_forward_time: 118.1481475830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2097320557
validation[          , genome  6189] predictions:    5994/  10000 (59.94%), best:    5994/10000 (59.94%), error:     11782.02148, best error:     11782.02148 on epoch:    63, epoch:   63/100, mu: 0.9729890823, learning_rate: 0.0000024044, weight_decay: 0.0000194231

epoch time: 1524.9333496094s, input_fired_time: 17.1731224060, output_fired_time: 7.6311798096, propagate_forward_time: 594.3251953125, propagate_backward_time: 860.7495117188, weight_update_time: 0.4554898441, other_time: 45.0543212891
epoch time: 123.1423645020s, input_fired_time: 1.2157245874, output_fired_time: 0.0000000000, propagate_forward_time: 118.0057678223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9208755493
validation[          , genome  6189] predictions:    6002/  10000 (60.02%), best:    6002/10000 (60.02%), error:     11720.87695, best error:     11720.87695 on epoch:    64, epoch:   64/100, mu: 0.9738661051, learning_rate: 0.0000022502, weight_decay: 0.0000184093

epoch time: 1540.2932128906s, input_fired_time: 17.4396476746, output_fired_time: 7.8203916550, propagate_forward_time: 601.4907836914, propagate_backward_time: 871.1218261719, weight_update_time: 0.4597323537, other_time: 42.4204711914
epoch time: 127.9595489502s, input_fired_time: 1.3678396940, output_fired_time: 0.0000000000, propagate_forward_time: 122.6293716431, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9623413086
validation[          , genome  6189] predictions:    5976/  10000 (59.76%), best:    6002/10000 (60.02%), error:     11821.43652, best error:     11720.87695 on epoch:    64, epoch:   65/100, mu: 0.9746978879, learning_rate: 0.0000021060, weight_decay: 0.0000174485

epoch time: 1534.4517822266s, input_fired_time: 17.7225761414, output_fired_time: 7.7133622169, propagate_forward_time: 605.2869262695, propagate_backward_time: 870.6962890625, weight_update_time: 0.4408024848, other_time: 33.0326538086
epoch time: 127.9998626709s, input_fired_time: 1.3023329973, output_fired_time: 0.0000000000, propagate_forward_time: 122.8135147095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8840179443
validation[          , genome  6189] predictions:    6014/  10000 (60.14%), best:    6002/10000 (60.02%), error:     11788.47363, best error:     11720.87695 on epoch:    64, epoch:   66/100, mu: 0.9754868150, learning_rate: 0.0000019709, weight_decay: 0.0000165377

epoch time: 1532.1040039063s, input_fired_time: 17.3028621674, output_fired_time: 7.6516222954, propagate_forward_time: 581.8069458008, propagate_backward_time: 859.2309570313, weight_update_time: 0.4591437578, other_time: 66.1116333008
epoch time: 120.8980484009s, input_fired_time: 1.2536363602, output_fired_time: 0.0000000000, propagate_forward_time: 114.5888595581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0555496216
validation[          , genome  6189] predictions:    6071/  10000 (60.71%), best:    6071/10000 (60.71%), error:     11719.94336, best error:     11719.94336 on epoch:    67, epoch:   67/100, mu: 0.9762350917, learning_rate: 0.0000018446, weight_decay: 0.0000156745

epoch time: 1487.1038818359s, input_fired_time: 16.7641391754, output_fired_time: 7.5283870697, propagate_forward_time: 572.8611450195, propagate_backward_time: 849.3516845703, weight_update_time: 0.4413530529, other_time: 40.5984497070
epoch time: 121.1476669312s, input_fired_time: 1.2441635132, output_fired_time: 0.0000000000, propagate_forward_time: 115.3357238770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5677795410
validation[          , genome  6189] predictions:    6047/  10000 (60.47%), best:    6071/10000 (60.71%), error:     11738.26367, best error:     11719.94336 on epoch:    67, epoch:   68/100, mu: 0.9769447446, learning_rate: 0.0000017263, weight_decay: 0.0000148564

epoch time: 1482.3349609375s, input_fired_time: 16.6529026031, output_fired_time: 7.5638437271, propagate_forward_time: 576.2425537109, propagate_backward_time: 853.4497070313, weight_update_time: 0.4555004239, other_time: 28.4259033203
epoch time: 121.7052459717s, input_fired_time: 1.2603561878, output_fired_time: 0.0000000000, propagate_forward_time: 116.5159912109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9289016724
validation[          , genome  6189] predictions:    6059/  10000 (60.59%), best:    6059/10000 (60.59%), error:     11671.88574, best error:     11671.88574 on epoch:    69, epoch:   69/100, mu: 0.9776178002, learning_rate: 0.0000016156, weight_decay: 0.0000140810

epoch time: 1497.1401367188s, input_fired_time: 16.7989368439, output_fired_time: 7.5838923454, propagate_forward_time: 580.2642822266, propagate_backward_time: 858.1425781250, weight_update_time: 0.4738740027, other_time: 34.3504638672
epoch time: 122.9517364502s, input_fired_time: 1.3165884018, output_fired_time: 0.0000000000, propagate_forward_time: 117.4272918701, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2078552246
validation[          , genome  6189] predictions:    6083/  10000 (60.83%), best:    6083/10000 (60.83%), error:     11652.86816, best error:     11652.86816 on epoch:    70, epoch:   70/100, mu: 0.9782561660, learning_rate: 0.0000015120, weight_decay: 0.0000133460

epoch time: 1504.3377685547s, input_fired_time: 16.8795585632, output_fired_time: 7.5207662582, propagate_forward_time: 572.5641479492, propagate_backward_time: 851.3758544922, weight_update_time: 0.4849202335, other_time: 55.9974975586
epoch time: 120.3693161011s, input_fired_time: 1.2546970844, output_fired_time: 0.0000000000, propagate_forward_time: 115.4583358765, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6562805176
validation[          , genome  6189] predictions:    6008/  10000 (60.08%), best:    6083/10000 (60.83%), error:     11740.33301, best error:     11652.86816 on epoch:    70, epoch:   71/100, mu: 0.9788616300, learning_rate: 0.0000014151, weight_decay: 0.0000126494

epoch time: 1497.7419433594s, input_fired_time: 16.9268245697, output_fired_time: 7.5635161400, propagate_forward_time: 576.3114624023, propagate_backward_time: 858.0858764648, weight_update_time: 0.4616301656, other_time: 38.8542480469
epoch time: 121.2103958130s, input_fired_time: 1.2806096077, output_fired_time: 0.0000000000, propagate_forward_time: 115.4409561157, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4888305664
validation[          , genome  6189] predictions:    6016/  10000 (60.16%), best:    6083/10000 (60.83%), error:     11757.89844, best error:     11652.86816 on epoch:    70, epoch:   72/100, mu: 0.9794358611, learning_rate: 0.0000013244, weight_decay: 0.0000119892

epoch time: 1502.3393554688s, input_fired_time: 16.8628616333, output_fired_time: 7.5271205902, propagate_forward_time: 574.7644042969, propagate_backward_time: 851.1505737305, weight_update_time: 0.4561433196, other_time: 52.0343627930
epoch time: 119.5719757080s, input_fired_time: 1.2601782084, output_fired_time: 0.0000000000, propagate_forward_time: 114.7252883911, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5865097046
validation[          , genome  6189] predictions:    6042/  10000 (60.42%), best:    6083/10000 (60.83%), error:     11656.80566, best error:     11652.86816 on epoch:    70, epoch:   73/100, mu: 0.9799805284, learning_rate: 0.0000012394, weight_decay: 0.0000113634

epoch time: 1458.9094238281s, input_fired_time: 16.6829032898, output_fired_time: 7.5304436684, propagate_forward_time: 571.2344970703, propagate_backward_time: 844.9584350586, weight_update_time: 0.4554756880, other_time: 18.5032348633
epoch time: 118.6539764404s, input_fired_time: 1.2401564121, output_fired_time: 0.0000000000, propagate_forward_time: 114.6998443604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7139739990
validation[          , genome  6189] predictions:    6035/  10000 (60.35%), best:    6083/10000 (60.83%), error:     11696.42871, best error:     11652.86816 on epoch:    70, epoch:   74/100, mu: 0.9804971218, learning_rate: 0.0000011600, weight_decay: 0.0000107703

epoch time: 1462.1402587891s, input_fired_time: 16.6273002625, output_fired_time: 7.4905166626, propagate_forward_time: 572.5359497070, propagate_backward_time: 847.2474365234, weight_update_time: 0.4452111721, other_time: 18.2390747070
epoch time: 117.9306106567s, input_fired_time: 1.2401225567, output_fired_time: 0.0000000000, propagate_forward_time: 114.0818405151, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6086502075
validation[          , genome  6189] predictions:    6068/  10000 (60.68%), best:    6068/10000 (60.68%), error:     11650.38379, best error:     11650.38379 on epoch:    75, epoch:   75/100, mu: 0.9809870720, learning_rate: 0.0000010856, weight_decay: 0.0000102081

epoch time: 1521.2883300781s, input_fired_time: 16.8590526581, output_fired_time: 7.5441870689, propagate_forward_time: 574.9437866211, propagate_backward_time: 852.6522216797, weight_update_time: 0.4585526586, other_time: 69.2891235352
epoch time: 118.1606750488s, input_fired_time: 1.2493333817, output_fired_time: 0.0000000000, propagate_forward_time: 113.4140167236, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4973220825
validation[          , genome  6189] predictions:    6044/  10000 (60.44%), best:    6068/10000 (60.68%), error:     11656.60352, best error:     11650.38379 on epoch:    75, epoch:   76/100, mu: 0.9814517498, learning_rate: 0.0000010160, weight_decay: 0.0000096753

epoch time: 1464.5457763672s, input_fired_time: 16.5846519470, output_fired_time: 7.5090799332, propagate_forward_time: 569.0861206055, propagate_backward_time: 842.3567504883, weight_update_time: 0.4682403505, other_time: 29.0092773438
epoch time: 119.9239501953s, input_fired_time: 1.2602115870, output_fired_time: 0.0000000000, propagate_forward_time: 114.9594039917, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7043380737
validation[          , genome  6189] predictions:    6028/  10000 (60.28%), best:    6068/10000 (60.68%), error:     11664.14160, best error:     11650.38379 on epoch:    75, epoch:   77/100, mu: 0.9818924665, learning_rate: 0.0000009509, weight_decay: 0.0000091703

epoch time: 1468.8582763672s, input_fired_time: 16.7788333893, output_fired_time: 7.4905829430, propagate_forward_time: 572.0078125000, propagate_backward_time: 848.0834960938, weight_update_time: 0.4518070817, other_time: 24.4975585938
epoch time: 118.7379302979s, input_fired_time: 1.2452276945, output_fired_time: 0.0000000000, propagate_forward_time: 113.9449005127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5478057861
validation[          , genome  6189] predictions:    6033/  10000 (60.33%), best:    6068/10000 (60.68%), error:     11722.75977, best error:     11650.38379 on epoch:    75, epoch:   78/100, mu: 0.9823104739, learning_rate: 0.0000008899, weight_decay: 0.0000086917

epoch time: 1480.1065673828s, input_fired_time: 16.7858810425, output_fired_time: 7.4921703339, propagate_forward_time: 576.5354003906, propagate_backward_time: 853.3641357422, weight_update_time: 0.4505798817, other_time: 25.9289550781
epoch time: 120.1530303955s, input_fired_time: 1.2649738789, output_fired_time: 0.0000000000, propagate_forward_time: 115.4913864136, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3966674805
validation[          , genome  6189] predictions:    6027/  10000 (60.27%), best:    6068/10000 (60.68%), error:     11694.05078, best error:     11650.38379 on epoch:    75, epoch:   79/100, mu: 0.9827069044, learning_rate: 0.0000008328, weight_decay: 0.0000082380

epoch time: 1479.7481689453s, input_fired_time: 16.8011684418, output_fired_time: 7.5787792206, propagate_forward_time: 575.1256713867, propagate_backward_time: 853.1880493164, weight_update_time: 0.4535737038, other_time: 27.0545654297
epoch time: 118.2116775513s, input_fired_time: 1.2419542074, output_fired_time: 0.0000000000, propagate_forward_time: 114.2654571533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7042694092
validation[          , genome  6189] predictions:    6067/  10000 (60.67%), best:    6067/10000 (60.67%), error:     11637.53223, best error:     11637.53223 on epoch:    80, epoch:   80/100, mu: 0.9830828905, learning_rate: 0.0000007794, weight_decay: 0.0000078080

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189_0_r588653149_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.512459
read mu: 0.98344
read mu_delta: 0.948443
read initial_learning_rate: 0.00015632
read learning_rate: 7.29464e-07
read learning_rate_delta: 0.935884
read initial_weight_decay: 0.000568883
read weight_decay: 7.40049e-06
read weight_decay_delta: 0.947805
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0871736
read input_dropout_probability: 0.00118845
read hidden_dropout_probability: 0.103497
read velocity_reset: 1821
read epoch: 81
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 80
read number_validation_images: 10000
read best_validation_predictions: 6067
read best_validation_error: 11637.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6189
read normal distribution: '0 -0.917038261890411 -2.04148006439209'
generator_str: '1760101878'
read generator: 1760101878
read generated_by_map:
1 crossover 1
reading 86 nodes.
reading 314 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1551.31665039063s, input_fired_time: 17.2099857330322, output_fired_time: 7.65790462493896, propagate_forward_time: 583.914916992188, propagate_backward_time: 856.240539550781, weight_update_time: 0.473187774419785, other_time: 86.2932739257813
epoch time: 124.186477661133s, input_fired_time: 1.31094622612, output_fired_time: 0, propagate_forward_time: 118.875785827637, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.99974822998047
validation[          , genome  6189] predictions:    5977/  10000 (59.77%), best:    6067/10000 (60.67%), error:     11813.32715, best error:     11637.53223 on epoch:    80, epoch:   81/100, mu: 0.9834395051, learning_rate: 0.0000007295, weight_decay: 0.0000074005

epoch time: 1508.4775390625s, input_fired_time: 17.1070938110, output_fired_time: 7.6872539520, propagate_forward_time: 586.9597778320, propagate_backward_time: 863.6680297852, weight_update_time: 0.4682632387, other_time: 33.0554199219
epoch time: 123.5861892700s, input_fired_time: 1.2893737555, output_fired_time: 0.0000000000, propagate_forward_time: 118.5667724609, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7300415039
validation[          , genome  6189] predictions:    6093/  10000 (60.93%), best:    6067/10000 (60.67%), error:     11645.16406, best error:     11637.53223 on epoch:    80, epoch:   82/100, mu: 0.9837777615, learning_rate: 0.0000006827, weight_decay: 0.0000070142

epoch time: 1488.8739013672s, input_fired_time: 16.9317874908, output_fired_time: 7.5512871742, propagate_forward_time: 583.4581298828, propagate_backward_time: 856.0372924805, weight_update_time: 0.4609258473, other_time: 24.8954467773
epoch time: 121.1050109863s, input_fired_time: 1.2729146481, output_fired_time: 0.0000000000, propagate_forward_time: 116.6957244873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1363754272
validation[          , genome  6189] predictions:    6099/  10000 (60.99%), best:    6099/10000 (60.99%), error:     11619.09180, best error:     11619.09180 on epoch:    83, epoch:   83/100, mu: 0.9840985537, learning_rate: 0.0000006389, weight_decay: 0.0000066481

epoch time: 1505.4962158203s, input_fired_time: 17.1721420288, output_fired_time: 7.6341967583, propagate_forward_time: 590.4553833008, propagate_backward_time: 865.5567016602, weight_update_time: 0.4589908123, other_time: 24.6778564453
epoch time: 123.6584243774s, input_fired_time: 1.3093199730, output_fired_time: 0.0000000000, propagate_forward_time: 118.7425231934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6065826416
validation[          , genome  6189] predictions:    6098/  10000 (60.98%), best:    6099/10000 (60.99%), error:     11622.63965, best error:     11619.09180 on epoch:    83, epoch:   84/100, mu: 0.9844028354, learning_rate: 0.0000005980, weight_decay: 0.0000063011

epoch time: 1501.6933593750s, input_fired_time: 17.6373214722, output_fired_time: 7.6977119446, propagate_forward_time: 592.6414184570, propagate_backward_time: 860.8013305664, weight_update_time: 0.4465937018, other_time: 22.9155273438
epoch time: 130.3992309570s, input_fired_time: 1.4963746071, output_fired_time: 0.0000000000, propagate_forward_time: 126.3055953979, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5972671509
validation[          , genome  6189] predictions:    6079/  10000 (60.79%), best:    6099/10000 (60.99%), error:     11647.85254, best error:     11619.09180 on epoch:    83, epoch:   85/100, mu: 0.9846913815, learning_rate: 0.0000005596, weight_decay: 0.0000059722

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189_0_r588653149_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.512459
read mu: 0.984965
read mu_delta: 0.948443
read initial_learning_rate: 0.00015632
read learning_rate: 5.23738e-07
read learning_rate_delta: 0.935884
read initial_weight_decay: 0.000568883
read weight_decay: 5.66051e-06
read weight_decay_delta: 0.947805
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0871736
read input_dropout_probability: 0.00118845
read hidden_dropout_probability: 0.103497
read velocity_reset: 1821
read epoch: 86
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 83
read number_validation_images: 10000
read best_validation_predictions: 6099
read best_validation_error: 11619.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6189
read normal distribution: '0 -0.917038261890411 -2.04148006439209'
generator_str: '778846902'
read generator: 778846902
read generated_by_map:
1 crossover 1
reading 86 nodes.
reading 314 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1577.75146484375s, input_fired_time: 17.9294319152832, output_fired_time: 7.8649115562439, propagate_forward_time: 612.926818847656, propagate_backward_time: 890.35107421875, weight_update_time: 0.424154251813889, other_time: 48.6792602539063
epoch time: 124.060485839844s, input_fired_time: 1.32989299297333, output_fired_time: 0, propagate_forward_time: 120.154052734375, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.5765380859375
validation[          , genome  6189] predictions:    6089/  10000 (60.89%), best:    6099/10000 (60.99%), error:     11620.13281, best error:     11619.09180 on epoch:    83, epoch:   86/100, mu: 0.9849650860, learning_rate: 0.0000005237, weight_decay: 0.0000056605

epoch time: 1460.8653564453s, input_fired_time: 16.3750171661, output_fired_time: 7.3997468948, propagate_forward_time: 574.5147094727, propagate_backward_time: 847.0185546875, weight_update_time: 0.3990469277, other_time: 15.5573120117
epoch time: 126.2452087402s, input_fired_time: 1.3874031305, output_fired_time: 0.0000000000, propagate_forward_time: 121.8893585205, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9684448242
validation[          , genome  6189] predictions:    6079/  10000 (60.79%), best:    6099/10000 (60.99%), error:     11645.74316, best error:     11619.09180 on epoch:    83, epoch:   87/100, mu: 0.9852246642, learning_rate: 0.0000004902, weight_decay: 0.0000053651

epoch time: 1507.6546630859s, input_fired_time: 17.3008174896, output_fired_time: 7.7565655708, propagate_forward_time: 592.9829711914, propagate_backward_time: 874.0422973633, weight_update_time: 0.4401450753, other_time: 15.5720214844
epoch time: 120.1420211792s, input_fired_time: 1.3337949514, output_fired_time: 0.0000000000, propagate_forward_time: 116.0029449463, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8052825928
validation[          , genome  6189] predictions:    6076/  10000 (60.76%), best:    6076/10000 (60.76%), error:     11614.97949, best error:     11614.97949 on epoch:    88, epoch:   88/100, mu: 0.9854708910, learning_rate: 0.0000004587, weight_decay: 0.0000050850

epoch time: 1487.9633789063s, input_fired_time: 17.1088027954, output_fired_time: 7.6055369377, propagate_forward_time: 584.4346923828, propagate_backward_time: 863.6710815430, weight_update_time: 0.4080332816, other_time: 15.1432495117
epoch time: 122.4885635376s, input_fired_time: 1.2948139906, output_fired_time: 0.0000000000, propagate_forward_time: 118.3240661621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8696823120
validation[          , genome  6189] predictions:    6077/  10000 (60.77%), best:    6077/10000 (60.77%), error:     11613.51953, best error:     11613.51953 on epoch:    89, epoch:   89/100, mu: 0.9857044220, learning_rate: 0.0000004293, weight_decay: 0.0000048196

epoch time: 1530.7617187500s, input_fired_time: 17.7831802368, output_fired_time: 7.8639688492, propagate_forward_time: 601.6224365234, propagate_backward_time: 887.9980468750, weight_update_time: 0.4422989488, other_time: 15.4940185547
epoch time: 117.2036437988s, input_fired_time: 1.3118771315, output_fired_time: 0.0000000000, propagate_forward_time: 113.5851516724, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3066177368
validation[          , genome  6189] predictions:    6096/  10000 (60.96%), best:    6077/10000 (60.77%), error:     11618.67188, best error:     11613.51953 on epoch:    89, epoch:   90/100, mu: 0.9859259129, learning_rate: 0.0000004018, weight_decay: 0.0000045681

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_186_6189_0_r588653149_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.512459
read mu: 0.986136
read mu_delta: 0.948443
read initial_learning_rate: 0.00015632
read learning_rate: 3.76032e-07
read learning_rate_delta: 0.935884
read initial_weight_decay: 0.000568883
read weight_decay: 4.32963e-06
read weight_decay_delta: 0.947805
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0871736
read input_dropout_probability: 0.00118845
read hidden_dropout_probability: 0.103497
read velocity_reset: 1821
read epoch: 91
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 89
read number_validation_images: 10000
read best_validation_predictions: 6077
read best_validation_error: 11613.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6189
read normal distribution: '0 -0.917038261890411 -2.04148006439209'
generator_str: '1330563928'
read generator: 1330563928
read generated_by_map:
1 crossover 1
reading 86 nodes.
reading 314 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1511.31115722656s, input_fired_time: 16.710578918457, output_fired_time: 7.3159351348877, propagate_forward_time: 611.764709472656, propagate_backward_time: 824.986389160156, weight_update_time: 0.405868887901306, other_time: 50.5335693359375
epoch time: 126.615158081055s, input_fired_time: 1.25451743602753, output_fired_time: 0, propagate_forward_time: 121.69441986084, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.66622161865234
validation[          , genome  6189] predictions:    6099/  10000 (60.99%), best:    6077/10000 (60.77%), error:     11631.83203, best error:     11613.51953 on epoch:    89, epoch:   91/100, mu: 0.9861359596, learning_rate: 0.0000003760, weight_decay: 0.0000043296

epoch time: 1481.8525390625s, input_fired_time: 16.6290130615, output_fired_time: 7.3972253799, propagate_forward_time: 614.3676757813, propagate_backward_time: 828.4223022461, weight_update_time: 0.4148500264, other_time: 15.0363159180
epoch time: 125.9276504517s, input_fired_time: 1.2551426888, output_fired_time: 0.0000000000, propagate_forward_time: 122.1768035889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4957046509
validation[          , genome  6189] predictions:    6104/  10000 (61.04%), best:    6104/10000 (61.04%), error:     11560.12500, best error:     11560.12500 on epoch:    92, epoch:   92/100, mu: 0.9863351583, learning_rate: 0.0000003519, weight_decay: 0.0000041036

epoch time: 1523.2053222656s, input_fired_time: 17.3357868195, output_fired_time: 7.4752278328, propagate_forward_time: 644.0491943359, propagate_backward_time: 835.7078247070, weight_update_time: 0.4328911304, other_time: 18.6372680664
epoch time: 136.0420532227s, input_fired_time: 1.2969759703, output_fired_time: 0.0000000000, propagate_forward_time: 130.7584228516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9866485596
validation[          , genome  6189] predictions:    6110/  10000 (61.10%), best:    6104/10000 (61.04%), error:     11633.21289, best error:     11560.12500 on epoch:    92, epoch:   93/100, mu: 0.9865241051, learning_rate: 0.0000003294, weight_decay: 0.0000038895

epoch time: 1549.2774658203s, input_fired_time: 17.4456844330, output_fired_time: 7.5871438980, propagate_forward_time: 646.0095214844, propagate_backward_time: 842.9223632813, weight_update_time: 0.4113215804, other_time: 35.3127441406
epoch time: 128.0616302490s, input_fired_time: 1.2671662569, output_fired_time: 0.0000000000, propagate_forward_time: 123.4790954590, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3153686523
validation[          , genome  6189] predictions:    6102/  10000 (61.02%), best:    6104/10000 (61.04%), error:     11630.25488, best error:     11560.12500 on epoch:    92, epoch:   94/100, mu: 0.9867033362, learning_rate: 0.0000003082, weight_decay: 0.0000036864

epoch time: 1293.5557861328s, input_fired_time: 15.6717262268, output_fired_time: 6.4861631393, propagate_forward_time: 525.0789794922, propagate_backward_time: 731.3226318359, weight_update_time: 0.3688538671, other_time: 14.9962158203
epoch time: 111.0047302246s, input_fired_time: 1.1296173334, output_fired_time: 0.0000000000, propagate_forward_time: 107.5263900757, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3487243652
validation[          , genome  6189] predictions:    6103/  10000 (61.03%), best:    6104/10000 (61.04%), error:     11626.14844, best error:     11560.12500 on epoch:    92, epoch:   95/100, mu: 0.9868733287, learning_rate: 0.0000002885, weight_decay: 0.0000034940

epoch time: 1317.0225830078s, input_fired_time: 15.9588766098, output_fired_time: 6.5995640755, propagate_forward_time: 536.9346313477, propagate_backward_time: 742.5287475586, weight_update_time: 0.3814393878, other_time: 15.0007324219
epoch time: 100.8713684082s, input_fired_time: 1.0703516006, output_fired_time: 0.0000000000, propagate_forward_time: 97.4580917358, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3429260254
validation[          , genome  6189] predictions:    6082/  10000 (60.82%), best:    6104/10000 (61.04%), error:     11626.85449, best error:     11560.12500 on epoch:    92, epoch:   96/100, mu: 0.9870344996, learning_rate: 0.0000002700, weight_decay: 0.0000033117

epoch time: 1060.8239746094s, input_fired_time: 14.7391719818, output_fired_time: 5.2967901230, propagate_forward_time: 418.4051513672, propagate_backward_time: 607.7703857422, weight_update_time: 0.3127696812, other_time: 14.6125488281
epoch time: 79.2499771118s, input_fired_time: 0.8844184875, output_fired_time: 0.0000000000, propagate_forward_time: 75.8520812988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5134735107
validation[          , genome  6189] predictions:    6111/  10000 (61.11%), best:    6104/10000 (61.04%), error:     11577.81738, best error:     11560.12500 on epoch:    92, epoch:   97/100, mu: 0.9871873856, learning_rate: 0.0000002527, weight_decay: 0.0000031388

epoch time: 975.3602294922s, input_fired_time: 14.3632097244, output_fired_time: 4.9381852150, propagate_forward_time: 379.4422912598, propagate_backward_time: 562.7438964844, weight_update_time: 0.2982094884, other_time: 13.8726806641
epoch time: 78.0114974976s, input_fired_time: 0.8873487115, output_fired_time: 0.0000000000, propagate_forward_time: 74.7127685547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4113769531
validation[          , genome  6189] predictions:    6114/  10000 (61.14%), best:    6104/10000 (61.04%), error:     11593.50488, best error:     11560.12500 on epoch:    92, epoch:   98/100, mu: 0.9873324037, learning_rate: 0.0000002365, weight_decay: 0.0000029750

epoch time: 967.9616699219s, input_fired_time: 14.3209476471, output_fired_time: 4.8667955399, propagate_forward_time: 376.1581420898, propagate_backward_time: 558.8868408203, weight_update_time: 0.2947463393, other_time: 13.7289428711
epoch time: 79.8915252686s, input_fired_time: 0.8996385932, output_fired_time: 0.0000000000, propagate_forward_time: 76.5222854614, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4696044922
validation[          , genome  6189] predictions:    6063/  10000 (60.63%), best:    6104/10000 (61.04%), error:     11608.86914, best error:     11560.12500 on epoch:    92, epoch:   99/100, mu: 0.9874699116, learning_rate: 0.0000002213, weight_decay: 0.0000028197

epoch time: 972.9093017578s, input_fired_time: 14.3474769592, output_fired_time: 4.8458046913, propagate_forward_time: 380.8563232422, propagate_backward_time: 559.0748901367, weight_update_time: 0.2979723215, other_time: 13.7847900391
epoch time: 78.7390136719s, input_fired_time: 0.8874815106, output_fired_time: 0.0000000000, propagate_forward_time: 75.3869094849, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4646224976
validation[          , genome  6189] predictions:    6099/  10000 (60.99%), best:    6104/10000 (61.04%), error:     11616.28516, best error:     11560.12500 on epoch:    92, epoch:  100/100, mu: 0.9876003265, learning_rate: 0.0000002071, weight_decay: 0.0000026725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 397.4890441895s, input_fired_time: 4.4616842270, output_fired_time: 0.0000000000, propagate_forward_time: 380.6513671875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.3760070801
best training[          , genome  6189] predictions:   31571/  50000 (63.14%), best:    6104/10000 (61.04%), error:     55990.66406, best error:     11560.12500 on epoch:    92, epoch:  101/100, mu: 0.9877240658, learning_rate: 0.0000001938, weight_decay: 0.0000025330
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 80.7036819458s, input_fired_time: 0.8896118999, output_fired_time: 0.0000000000, propagate_forward_time: 76.5627517700, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2513198853
   testing[          , genome  6189] predictions:    6104/  10000 (61.04%), best:    6104/10000 (61.04%), error:     11560.12500, best error:     11560.12500 on epoch:    92, epoch:  101/100, mu: 0.9877240658, learning_rate: 0.0000001938, weight_decay: 0.0000025330
backpropagation finished successfully!
16:27:36 (17188): called boinc_finish(0)

</stderr_txt>
]]>