Nameexact_genome_1542692703_187_6390_3
Workunit4100618
Created20 Nov 2018, 9:25:41 UTC
Sent20 Nov 2018, 9:58:45 UTC
Report deadline14 Mar 2019, 4:36:19 UTC
Received28 Nov 2018, 8:22:50 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88000
Run time2 days 7 hours 57 min 23 sec
CPU time2 days 7 hours 55 min 37 sec
Validate stateValid
Credit28,444.02
Device peak FLOPS3.82 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size540.83 MB
Peak swap size539.16 MB
Peak disk usage12.48 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
_time: 15.4135742188
epoch time: 154.1524810791s, input_fired_time: 1.9263347387, output_fired_time: 0.0000000000, propagate_forward_time: 149.5761413574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6500091553
validation[          , genome  6390] predictions:    6273/  10000 (62.73%), best:    6273/10000 (62.73%), error:     10800.95215, best error:     10800.95215 on epoch:    29, epoch:   29/100, mu: 0.8452879786, learning_rate: 0.0005826206, weight_decay: 0.0001498666

epoch time: 1830.5461425781s, input_fired_time: 22.3478565216, output_fired_time: 12.0496597290, propagate_forward_time: 743.8838500977, propagate_backward_time: 1036.8999023438, weight_update_time: 0.8651496172, other_time: 15.3647460938
epoch time: 153.4738159180s, input_fired_time: 1.9100192785, output_fired_time: 0.0000000000, propagate_forward_time: 148.9520721436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6117248535
validation[          , genome  6390] predictions:    6174/  10000 (61.74%), best:    6273/10000 (62.73%), error:     10813.81152, best error:     10800.95215 on epoch:    29, epoch:   30/100, mu: 0.8512710929, learning_rate: 0.0005363012, weight_decay: 0.0001427124

epoch time: 1815.3302001953s, input_fired_time: 22.2366638184, output_fired_time: 12.0256490707, propagate_forward_time: 735.9057006836, propagate_backward_time: 1029.3448486328, weight_update_time: 0.8606229424, other_time: 15.8172607422
epoch time: 152.6729278564s, input_fired_time: 1.9170316458, output_fired_time: 0.0000000000, propagate_forward_time: 148.1818695068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5740203857
validation[          , genome  6390] predictions:    6266/  10000 (62.66%), best:    6273/10000 (62.73%), error:     10818.79980, best error:     10800.95215 on epoch:    29, epoch:   31/100, mu: 0.8570068479, learning_rate: 0.0004936642, weight_decay: 0.0001358997

epoch time: 1829.6292724609s, input_fired_time: 22.2820568085, output_fired_time: 12.0563144684, propagate_forward_time: 742.7248535156, propagate_backward_time: 1037.0457763672, weight_update_time: 0.8609943986, other_time: 15.5202636719
epoch time: 152.6313323975s, input_fired_time: 1.9258974791, output_fired_time: 0.0000000000, propagate_forward_time: 148.0865478516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6188812256
validation[          , genome  6390] predictions:    6153/  10000 (61.53%), best:    6273/10000 (62.73%), error:     10939.47363, best error:     10800.95215 on epoch:    29, epoch:   32/100, mu: 0.8625054359, learning_rate: 0.0004544171, weight_decay: 0.0001294122

epoch time: 1834.2706298828s, input_fired_time: 22.4219894409, output_fired_time: 12.1029253006, propagate_forward_time: 744.8272094727, propagate_backward_time: 1039.3330078125, weight_update_time: 0.8707811236, other_time: 15.5854492188
epoch time: 151.9061889648s, input_fired_time: 1.8799861670, output_fired_time: 0.0000000000, propagate_forward_time: 147.0930633545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9331359863
validation[          , genome  6390] predictions:    6322/  10000 (63.22%), best:    6322/10000 (63.22%), error:     10539.84863, best error:     10539.84863 on epoch:    33, epoch:   33/100, mu: 0.8677766919, learning_rate: 0.0004182901, weight_decay: 0.0001232345

epoch time: 1824.0711669922s, input_fired_time: 22.2800788879, output_fired_time: 12.0139188766, propagate_forward_time: 740.4804687500, propagate_backward_time: 1034.1350097656, weight_update_time: 0.8586758375, other_time: 15.1617431641
epoch time: 152.8211975098s, input_fired_time: 1.8577628136, output_fired_time: 0.0000000000, propagate_forward_time: 148.2519378662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7115020752
validation[          , genome  6390] predictions:    6285/  10000 (62.85%), best:    6322/10000 (63.22%), error:     10742.79883, best error:     10539.84863 on epoch:    33, epoch:   34/100, mu: 0.8728300333, learning_rate: 0.0003850353, weight_decay: 0.0001173516

epoch time: 1828.8535156250s, input_fired_time: 22.3889999390, output_fired_time: 12.1019458771, propagate_forward_time: 740.7297973633, propagate_backward_time: 1037.1143798828, weight_update_time: 0.8730993867, other_time: 16.5184326172
epoch time: 151.6037445068s, input_fired_time: 1.8810747862, output_fired_time: 0.0000000000, propagate_forward_time: 146.5684814453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1541900635
validation[          , genome  6390] predictions:    6173/  10000 (61.73%), best:    6322/10000 (63.22%), error:     11233.96484, best error:     10539.84863 on epoch:    33, epoch:   35/100, mu: 0.8776744604, learning_rate: 0.0003544243, weight_decay: 0.0001117496

epoch time: 1811.7077636719s, input_fired_time: 22.3286228180, output_fired_time: 11.9800281525, propagate_forward_time: 733.5913696289, propagate_backward_time: 1027.9326171875, weight_update_time: 0.8630150557, other_time: 15.8752441406
epoch time: 153.3886413574s, input_fired_time: 1.9077798128, output_fired_time: 0.0000000000, propagate_forward_time: 148.9750671387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5057983398
validation[          , genome  6390] predictions:    6348/  10000 (63.48%), best:    6348/10000 (63.48%), error:     10426.45605, best error:     10426.45605 on epoch:    36, epoch:   36/100, mu: 0.8823185563, learning_rate: 0.0003262469, weight_decay: 0.0001064150

epoch time: 1821.2025146484s, input_fired_time: 22.3973846436, output_fired_time: 12.0493087769, propagate_forward_time: 738.9702758789, propagate_backward_time: 1031.6124267578, weight_update_time: 0.8676384687, other_time: 16.1732177734
epoch time: 155.1989898682s, input_fired_time: 1.8977999687, output_fired_time: 0.0000000000, propagate_forward_time: 150.7466125488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5545806885
validation[          , genome  6390] predictions:    6323/  10000 (63.23%), best:    6348/10000 (63.48%), error:     10475.12402, best error:     10426.45605 on epoch:    36, epoch:   37/100, mu: 0.8867706656, learning_rate: 0.0003003097, weight_decay: 0.0001013350

epoch time: 1814.1674804688s, input_fired_time: 22.2747631073, output_fired_time: 12.0308284760, propagate_forward_time: 735.3402099609, propagate_backward_time: 1028.6635742188, weight_update_time: 0.8588549495, other_time: 15.8580322266
epoch time: 153.6219940186s, input_fired_time: 1.9451907873, output_fired_time: 0.0000000000, propagate_forward_time: 148.9739379883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7028656006
validation[          , genome  6390] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     10406.58594, best error:     10406.58594 on epoch:    38, epoch:   38/100, mu: 0.8910387158, learning_rate: 0.0002764345, weight_decay: 0.0000964976

epoch time: 1818.5013427734s, input_fired_time: 22.4541988373, output_fired_time: 12.0428628922, propagate_forward_time: 736.5878906250, propagate_backward_time: 1030.9606933594, weight_update_time: 0.8701767325, other_time: 16.4556884766
epoch time: 153.4365386963s, input_fired_time: 1.8970098495, output_fired_time: 0.0000000000, propagate_forward_time: 148.7657165527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7738189697
validation[          , genome  6390] predictions:    6348/  10000 (63.48%), best:    6340/10000 (63.40%), error:     10510.34766, best error:     10406.58594 on epoch:    38, epoch:   39/100, mu: 0.8951302767, learning_rate: 0.0002544575, weight_decay: 0.0000918910

epoch time: 1827.9810791016s, input_fired_time: 22.4840583801, output_fired_time: 12.0870265961, propagate_forward_time: 741.8127441406, propagate_backward_time: 1035.3568115234, weight_update_time: 0.8730245233, other_time: 16.2404785156
epoch time: 152.6500854492s, input_fired_time: 1.9293607473, output_fired_time: 0.0000000000, propagate_forward_time: 147.9146118164, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8061065674
validation[          , genome  6390] predictions:    6300/  10000 (63.00%), best:    6340/10000 (63.40%), error:     10585.87988, best error:     10406.58594 on epoch:    38, epoch:   40/100, mu: 0.8990526795, learning_rate: 0.0002342276, weight_decay: 0.0000875044

epoch time: 1830.9029541016s, input_fired_time: 22.5377960205, output_fired_time: 12.1251783371, propagate_forward_time: 742.3430175781, propagate_backward_time: 1037.3532714844, weight_update_time: 0.8745844364, other_time: 16.5437011719
epoch time: 153.2937469482s, input_fired_time: 1.9354296923, output_fired_time: 0.0000000000, propagate_forward_time: 148.5554199219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8029022217
validation[          , genome  6390] predictions:    6344/  10000 (63.44%), best:    6344/10000 (63.44%), error:     10375.52344, best error:     10375.52344 on epoch:    41, epoch:   41/100, mu: 0.9028128982, learning_rate: 0.0002156061, weight_decay: 0.0000833272

epoch time: 1813.6881103516s, input_fired_time: 22.4123630524, output_fired_time: 12.0215635300, propagate_forward_time: 733.9097290039, propagate_backward_time: 1028.5792236328, weight_update_time: 0.8685705662, other_time: 16.7652587891
epoch time: 153.4113464355s, input_fired_time: 1.9645652771, output_fired_time: 0.0000000000, propagate_forward_time: 148.8166503906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6301269531
validation[          , genome  6390] predictions:    6287/  10000 (62.87%), best:    6344/10000 (63.44%), error:     10520.21973, best error:     10375.52344 on epoch:    41, epoch:   42/100, mu: 0.9064176679, learning_rate: 0.0001984650, weight_decay: 0.0000793494

epoch time: 1821.6435546875s, input_fired_time: 22.5843124390, output_fired_time: 12.0486431122, propagate_forward_time: 739.3178100586, propagate_backward_time: 1031.5408935547, weight_update_time: 0.8756601214, other_time: 16.1519775391
epoch time: 148.6565399170s, input_fired_time: 1.8330954313, output_fired_time: 0.0000000000, propagate_forward_time: 143.5258789063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2975616455
validation[          , genome  6390] predictions:    6430/  10000 (64.30%), best:    6430/10000 (64.30%), error:     10175.86816, best error:     10175.86816 on epoch:    43, epoch:   43/100, mu: 0.9098733664, learning_rate: 0.0001826867, weight_decay: 0.0000755615

epoch time: 1814.7648925781s, input_fired_time: 22.4376544952, output_fired_time: 12.0274486542, propagate_forward_time: 734.9058227539, propagate_backward_time: 1028.5905761719, weight_update_time: 0.8694483638, other_time: 16.8034667969
epoch time: 153.4199218750s, input_fired_time: 1.9711989164, output_fired_time: 0.0000000000, propagate_forward_time: 148.8198394775, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6288909912
validation[          , genome  6390] predictions:    6401/  10000 (64.01%), best:    6430/10000 (64.30%), error:     10311.11328, best error:     10175.86816 on epoch:    43, epoch:   44/100, mu: 0.9131861925, learning_rate: 0.0001681628, weight_decay: 0.0000719544

epoch time: 1831.6950683594s, input_fired_time: 22.5702877045, output_fired_time: 12.1076421738, propagate_forward_time: 742.2881469727, propagate_backward_time: 1038.5070800781, weight_update_time: 0.8750561476, other_time: 16.2219238281
epoch time: 152.1065368652s, input_fired_time: 1.8974168301, output_fired_time: 0.0000000000, propagate_forward_time: 147.7310638428, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4780578613
validation[          , genome  6390] predictions:    6459/  10000 (64.59%), best:    6459/10000 (64.59%), error:     10157.73535, best error:     10157.73535 on epoch:    45, epoch:   45/100, mu: 0.9163620472, learning_rate: 0.0001547935, weight_decay: 0.0000685195

epoch time: 1823.4934082031s, input_fired_time: 22.3958301544, output_fired_time: 12.0655088425, propagate_forward_time: 739.0670776367, propagate_backward_time: 1033.8117675781, weight_update_time: 0.8650442362, other_time: 16.1530761719
epoch time: 149.3208770752s, input_fired_time: 1.8454912901, output_fired_time: 0.0000000000, propagate_forward_time: 144.1550903320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3202972412
validation[          , genome  6390] predictions:    6447/  10000 (64.47%), best:    6459/10000 (64.59%), error:     10158.88672, best error:     10157.73535 on epoch:    45, epoch:   46/100, mu: 0.9194065928, learning_rate: 0.0001424872, weight_decay: 0.0000652486

epoch time: 1826.5554199219s, input_fired_time: 22.5658836365, output_fired_time: 12.0814132690, propagate_forward_time: 740.2337036133, propagate_backward_time: 1035.3791503906, weight_update_time: 0.8770502806, other_time: 16.2951660156
epoch time: 152.6940765381s, input_fired_time: 1.9305617809, output_fired_time: 0.0000000000, propagate_forward_time: 148.1271057129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6364135742
validation[          , genome  6390] predictions:    6512/  10000 (65.12%), best:    6512/10000 (65.12%), error:     10043.93848, best error:     10043.93848 on epoch:    47, epoch:   47/100, mu: 0.9223253131, learning_rate: 0.0001311592, weight_decay: 0.0000621338

epoch time: 1825.9760742188s, input_fired_time: 22.3573780060, output_fired_time: 12.0293827057, propagate_forward_time: 740.9047851563, propagate_backward_time: 1035.4924316406, weight_update_time: 0.8557447195, other_time: 15.1920166016
epoch time: 156.0673065186s, input_fired_time: 1.9313038588, output_fired_time: 0.0000000000, propagate_forward_time: 151.5182647705, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6177368164
validation[          , genome  6390] predictions:    6519/  10000 (65.19%), best:    6512/10000 (65.12%), error:     10069.63086, best error:     10043.93848 on epoch:    47, epoch:   48/100, mu: 0.9251233339, learning_rate: 0.0001207318, weight_decay: 0.0000591677

epoch time: 1835.3258056641s, input_fired_time: 22.5220146179, output_fired_time: 12.1132678986, propagate_forward_time: 745.0465698242, propagate_backward_time: 1039.7879638672, weight_update_time: 0.8686455488, other_time: 15.8560791016
epoch time: 155.9772338867s, input_fired_time: 1.9128926992, output_fired_time: 0.0000000000, propagate_forward_time: 151.0545959473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0097503662
validation[          , genome  6390] predictions:    6540/  10000 (65.40%), best:    6540/10000 (65.40%), error:     10025.77148, best error:     10025.77148 on epoch:    49, epoch:   49/100, mu: 0.9278056622, learning_rate: 0.0001111334, weight_decay: 0.0000563432

epoch time: 1827.8590087891s, input_fired_time: 22.4630889893, output_fired_time: 12.0851993561, propagate_forward_time: 741.7619018555, propagate_backward_time: 1035.3712158203, weight_update_time: 0.8671399355, other_time: 16.1776123047
epoch time: 152.2473297119s, input_fired_time: 1.9075055122, output_fired_time: 0.0000000000, propagate_forward_time: 147.8660430908, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4737854004
validation[          , genome  6390] predictions:    6517/  10000 (65.17%), best:    6540/10000 (65.40%), error:     10037.86621, best error:     10025.77148 on epoch:    49, epoch:   50/100, mu: 0.9303770661, learning_rate: 0.0001022981, weight_decay: 0.0000536535

epoch time: 1806.7858886719s, input_fired_time: 22.3751316071, output_fired_time: 12.0411472321, propagate_forward_time: 729.2111206055, propagate_backward_time: 1025.8553466797, weight_update_time: 0.8713163733, other_time: 17.3031005859
epoch time: 152.3957519531s, input_fired_time: 1.9737856388, output_fired_time: 0.0000000000, propagate_forward_time: 147.7981872559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6237792969
validation[          , genome  6390] predictions:    6497/  10000 (64.97%), best:    6540/10000 (65.40%), error:     10259.20703, best error:     10025.77148 on epoch:    49, epoch:   51/100, mu: 0.9328421950, learning_rate: 0.0000941652, weight_decay: 0.0000510923

epoch time: 1825.9816894531s, input_fired_time: 22.5600738525, output_fired_time: 12.0776681900, propagate_forward_time: 740.2540893555, propagate_backward_time: 1034.8116455078, weight_update_time: 0.8704708815, other_time: 16.2781982422
epoch time: 154.6249694824s, input_fired_time: 1.9496580362, output_fired_time: 0.0000000000, propagate_forward_time: 150.0014190674, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6738891602
validation[          , genome  6390] predictions:    6535/  10000 (65.35%), best:    6535/10000 (65.35%), error:      9908.40918, best error:      9908.40918 on epoch:    52, epoch:   52/100, mu: 0.9352054000, learning_rate: 0.0000866789, weight_decay: 0.0000486533

epoch time: 1832.4934082031s, input_fired_time: 22.6368484497, output_fired_time: 12.1082334518, propagate_forward_time: 743.9951782227, propagate_backward_time: 1037.6013183594, weight_update_time: 0.8723342419, other_time: 16.1518554688
epoch time: 152.9105224609s, input_fired_time: 1.9664461613, output_fired_time: 0.0000000000, propagate_forward_time: 148.3155059814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6285705566
validation[          , genome  6390] predictions:    6421/  10000 (64.21%), best:    6535/10000 (65.35%), error:     10269.35938, best error:      9908.40918 on epoch:    52, epoch:   53/100, mu: 0.9374709129, learning_rate: 0.0000797878, weight_decay: 0.0000463307

epoch time: 1805.9050292969s, input_fired_time: 22.5339221954, output_fired_time: 12.0302619934, propagate_forward_time: 729.9813842773, propagate_backward_time: 1023.7347412109, weight_update_time: 0.8731364012, other_time: 17.6246337891
epoch time: 147.0557250977s, input_fired_time: 1.8278057575, output_fired_time: 0.0000000000, propagate_forward_time: 142.0529479980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1749725342
validation[          , genome  6390] predictions:    6457/  10000 (64.57%), best:    6535/10000 (65.35%), error:     10237.23926, best error:      9908.40918 on epoch:    52, epoch:   54/100, mu: 0.9396427274, learning_rate: 0.0000734445, weight_decay: 0.0000441190

epoch time: 1831.2600097656s, input_fired_time: 22.6892776489, output_fired_time: 12.1172237396, propagate_forward_time: 743.8508300781, propagate_backward_time: 1036.3679199219, weight_update_time: 0.8771125674, other_time: 16.2347412109
epoch time: 154.1783447266s, input_fired_time: 1.9276165962, output_fired_time: 0.0000000000, propagate_forward_time: 149.7889709473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4617614746
validation[          , genome  6390] predictions:    6521/  10000 (65.21%), best:    6535/10000 (65.35%), error:     10046.65137, best error:      9908.40918 on epoch:    52, epoch:   55/100, mu: 0.9417247772, learning_rate: 0.0000676055, weight_decay: 0.0000420129

epoch time: 1838.5410156250s, input_fired_time: 22.6843376160, output_fired_time: 12.1595735550, propagate_forward_time: 745.7199707031, propagate_backward_time: 1041.7695312500, weight_update_time: 0.8747367263, other_time: 16.2076416016
epoch time: 154.5190734863s, input_fired_time: 1.9318698645, output_fired_time: 0.0000000000, propagate_forward_time: 149.9072113037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6799926758
validation[          , genome  6390] predictions:    6596/  10000 (65.96%), best:    6596/10000 (65.96%), error:      9878.50293, best error:      9878.50293 on epoch:    56, epoch:   56/100, mu: 0.9437206984, learning_rate: 0.0000622308, weight_decay: 0.0000400073

epoch time: 1829.9739990234s, input_fired_time: 22.5570907593, output_fired_time: 12.0697765350, propagate_forward_time: 743.9058227539, propagate_backward_time: 1036.0635986328, weight_update_time: 0.8651217818, other_time: 15.3775634766
epoch time: 155.4745788574s, input_fired_time: 1.9618452787, output_fired_time: 0.0000000000, propagate_forward_time: 150.8661956787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6465454102
validation[          , genome  6390] predictions:    6494/  10000 (64.94%), best:    6596/10000 (65.96%), error:     10203.57129, best error:      9878.50293 on epoch:    56, epoch:   57/100, mu: 0.9456341267, learning_rate: 0.0000572833, weight_decay: 0.0000380975

epoch time: 1837.9449462891s, input_fired_time: 22.7260169983, output_fired_time: 12.1215391159, propagate_forward_time: 748.0490722656, propagate_backward_time: 1039.4426269531, weight_update_time: 0.8718246222, other_time: 15.6055908203
epoch time: 153.6419525146s, input_fired_time: 1.9857329130, output_fired_time: 0.0000000000, propagate_forward_time: 149.0326995850, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6235198975
validation[          , genome  6390] predictions:    6560/  10000 (65.60%), best:    6560/10000 (65.60%), error:      9855.23730, best error:      9855.23730 on epoch:    58, epoch:   58/100, mu: 0.9474684596, learning_rate: 0.0000527292, weight_decay: 0.0000362788

epoch time: 1847.0965576172s, input_fired_time: 22.6327819824, output_fired_time: 12.1918563843, propagate_forward_time: 751.4237670898, propagate_backward_time: 1045.1451416016, weight_update_time: 0.8745932579, other_time: 15.7030029297
epoch time: 155.0636138916s, input_fired_time: 1.9335427284, output_fired_time: 0.0000000000, propagate_forward_time: 150.6487426758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4813232422
validation[          , genome  6390] predictions:    6575/  10000 (65.75%), best:    6575/10000 (65.75%), error:      9840.65430, best error:      9840.65430 on epoch:    59, epoch:   59/100, mu: 0.9492269158, learning_rate: 0.0000485371, weight_decay: 0.0000345470

epoch time: 1834.1483154297s, input_fired_time: 22.5521564484, output_fired_time: 12.1149368286, propagate_forward_time: 745.7621459961, propagate_backward_time: 1038.1970214844, weight_update_time: 0.8637891412, other_time: 15.5219726563
epoch time: 150.9539031982s, input_fired_time: 1.9102458954, output_fired_time: 0.0000000000, propagate_forward_time: 145.3811798096, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6624755859
validation[          , genome  6390] predictions:    6610/  10000 (66.10%), best:    6610/10000 (66.10%), error:      9822.33203, best error:      9822.33203 on epoch:    60, epoch:   60/100, mu: 0.9509126544, learning_rate: 0.0000446783, weight_decay: 0.0000328978

epoch time: 1819.2480468750s, input_fired_time: 22.6259078979, output_fired_time: 12.0665168762, propagate_forward_time: 737.0313720703, propagate_backward_time: 1031.2453613281, weight_update_time: 0.8744717240, other_time: 16.2789306641
epoch time: 153.8999176025s, input_fired_time: 1.9476548433, output_fired_time: 0.0000000000, propagate_forward_time: 148.5682983398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3839569092
validation[          , genome  6390] predictions:    6544/  10000 (65.44%), best:    6610/10000 (66.10%), error:      9955.69141, best error:      9822.33203 on epoch:    60, epoch:   61/100, mu: 0.9525287151, learning_rate: 0.0000411263, weight_decay: 0.0000313273

epoch time: 1822.1174316406s, input_fired_time: 22.7037563324, output_fired_time: 12.0728025436, propagate_forward_time: 737.9868774414, propagate_backward_time: 1032.5495605469, weight_update_time: 0.8794792295, other_time: 16.8044433594
epoch time: 152.0925445557s, input_fired_time: 1.9178861380, output_fired_time: 0.0000000000, propagate_forward_time: 147.4496459961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7250061035
validation[          , genome  6390] predictions:    6545/  10000 (65.45%), best:    6610/10000 (66.10%), error:      9927.40820, best error:      9822.33203 on epoch:    60, epoch:   62/100, mu: 0.9540779591, learning_rate: 0.0000378567, weight_decay: 0.0000298319

epoch time: 1831.4787597656s, input_fired_time: 22.5997371674, output_fired_time: 12.1418724060, propagate_forward_time: 741.8092041016, propagate_backward_time: 1038.1923828125, weight_update_time: 0.8784912825, other_time: 16.7355957031
epoch time: 154.8586120605s, input_fired_time: 1.9407678843, output_fired_time: 0.0000000000, propagate_forward_time: 150.2607116699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6571350098
validation[          , genome  6390] predictions:    6598/  10000 (65.98%), best:    6610/10000 (66.10%), error:      9846.09180, best error:      9822.33203 on epoch:    60, epoch:   63/100, mu: 0.9555631876, learning_rate: 0.0000348470, weight_decay: 0.0000284078

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6390.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6390_3_r208203758_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.497622
read mu: 0.956987
read mu_delta: 0.958655
read initial_learning_rate: 0.00643755
read learning_rate: 3.20766e-05
read learning_rate_delta: 0.920498
read initial_weight_decay: 0.000619094
read weight_decay: 2.70517e-05
read weight_decay_delta: 0.952263
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.105355
read input_dropout_probability: 0.00123136
read hidden_dropout_probability: 0.0766924
read velocity_reset: 1054
read epoch: 64
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 60
read number_validation_images: 10000
read best_validation_predictions: 6610
read best_validation_error: 9822.33
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6390
read normal distribution: '1 1.04104495048523 -0.215225517749786'
generator_str: '1604791461'
read generator: 1604791461
read generated_by_map:
1 crossover 1
reading 97 nodes.
reading 425 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1854.50354003906s, input_fired_time: 22.8713150024414, output_fired_time: 12.3217144012451, propagate_forward_time: 755.702514648438, propagate_backward_time: 1047.53039550781, weight_update_time: 0.887301862239838, other_time: 16.07763671875
epoch time: 158.326171875s, input_fired_time: 1.93949604034424, output_fired_time: 0, propagate_forward_time: 153.83268737793, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.55398559570313
validation[          , genome  6390] predictions:    6631/  10000 (66.31%), best:    6631/10000 (66.31%), error:      9784.94141, best error:      9784.94141 on epoch:    64, epoch:   64/100, mu: 0.9569869637, learning_rate: 0.0000320766, weight_decay: 0.0000270517

epoch time: 1853.1732177734s, input_fired_time: 22.8481044769, output_fired_time: 12.2514295578, propagate_forward_time: 753.1212158203, propagate_backward_time: 1048.5303955078, weight_update_time: 0.8843165636, other_time: 16.4219970703
epoch time: 152.2770538330s, input_fired_time: 1.9724607468, output_fired_time: 0.0000000000, propagate_forward_time: 147.4681091309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8364868164
validation[          , genome  6390] predictions:    6637/  10000 (66.37%), best:    6637/10000 (66.37%), error:      9746.06934, best error:      9746.06934 on epoch:    65, epoch:   65/100, mu: 0.9583519101, learning_rate: 0.0000295265, weight_decay: 0.0000257603

epoch time: 1846.1093750000s, input_fired_time: 22.8210220337, output_fired_time: 12.2328681946, propagate_forward_time: 749.5640869141, propagate_backward_time: 1044.4829101563, weight_update_time: 0.8898831010, other_time: 17.0084228516
epoch time: 152.7689819336s, input_fired_time: 1.9394826889, output_fired_time: 0.0000000000, propagate_forward_time: 148.3657073975, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4637908936
validation[          , genome  6390] predictions:    6649/  10000 (66.49%), best:    6649/10000 (66.49%), error:      9680.09668, best error:      9680.09668 on epoch:    66, epoch:   66/100, mu: 0.9596604109, learning_rate: 0.0000271791, weight_decay: 0.0000245306

epoch time: 1854.3027343750s, input_fired_time: 22.7998905182, output_fired_time: 12.2923746109, propagate_forward_time: 753.7402954102, propagate_backward_time: 1049.1005859375, weight_update_time: 0.8902345300, other_time: 16.3696289063
epoch time: 154.0710601807s, input_fired_time: 1.9062403440, output_fired_time: 0.0000000000, propagate_forward_time: 149.5889587402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5758666992
validation[          , genome  6390] predictions:    6595/  10000 (65.95%), best:    6649/10000 (66.49%), error:      9893.34473, best error:      9680.09668 on epoch:    66, epoch:   67/100, mu: 0.9609147906, learning_rate: 0.0000250183, weight_decay: 0.0000233595

epoch time: 1835.7879638672s, input_fired_time: 22.6192989349, output_fired_time: 12.1529645920, propagate_forward_time: 745.0751342773, propagate_backward_time: 1039.8529052734, weight_update_time: 0.8735018969, other_time: 16.0877685547
epoch time: 152.0195922852s, input_fired_time: 1.9352926016, output_fired_time: 0.0000000000, propagate_forward_time: 147.1336364746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9506683350
validation[          , genome  6390] predictions:    6671/  10000 (66.71%), best:    6649/10000 (66.49%), error:      9722.15527, best error:      9680.09668 on epoch:    66, epoch:   68/100, mu: 0.9621173143, learning_rate: 0.0000230293, weight_decay: 0.0000222444

epoch time: 1839.6915283203s, input_fired_time: 22.7449417114, output_fired_time: 12.1729660034, propagate_forward_time: 746.5064697266, propagate_backward_time: 1042.0273437500, weight_update_time: 0.8859975934, other_time: 16.2397460938
epoch time: 156.0698547363s, input_fired_time: 1.9477843046, output_fired_time: 0.0000000000, propagate_forward_time: 151.5807342529, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5413360596
validation[          , genome  6390] predictions:    6568/  10000 (65.68%), best:    6649/10000 (66.49%), error:      9827.23828, best error:      9680.09668 on epoch:    66, epoch:   69/100, mu: 0.9632701278, learning_rate: 0.0000211984, weight_decay: 0.0000211825

epoch time: 1840.6795654297s, input_fired_time: 22.8168258667, output_fired_time: 12.2018585205, propagate_forward_time: 746.0283813477, propagate_backward_time: 1042.7624511719, weight_update_time: 0.8852761984, other_time: 16.8701171875
epoch time: 153.5694274902s, input_fired_time: 1.9387336969, output_fired_time: 0.0000000000, propagate_forward_time: 149.0439605713, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5867309570
validation[          , genome  6390] predictions:    6631/  10000 (66.31%), best:    6649/10000 (66.49%), error:      9711.87012, best error:      9680.09668 on epoch:    66, epoch:   70/100, mu: 0.9643752575, learning_rate: 0.0000195131, weight_decay: 0.0000201713

epoch time: 1840.9254150391s, input_fired_time: 22.7102603912, output_fired_time: 12.2082281113, propagate_forward_time: 745.7252197266, propagate_backward_time: 1043.8566894531, weight_update_time: 0.8758537769, other_time: 16.4250488281
epoch time: 156.6614532471s, input_fired_time: 1.9600100517, output_fired_time: 0.0000000000, propagate_forward_time: 152.1759643555, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5254821777
validation[          , genome  6390] predictions:    6652/  10000 (66.52%), best:    6649/10000 (66.49%), error:      9746.15723, best error:      9680.09668 on epoch:    66, epoch:   71/100, mu: 0.9654347301, learning_rate: 0.0000179618, weight_decay: 0.0000192084

epoch time: 1796.8803710938s, input_fired_time: 22.6041336060, output_fired_time: 12.0624361038, propagate_forward_time: 724.2561645508, propagate_backward_time: 1020.0272216797, weight_update_time: 0.8819754124, other_time: 17.9305419922
epoch time: 149.5062866211s, input_fired_time: 1.8866842985, output_fired_time: 0.0000000000, propagate_forward_time: 144.2967071533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3228912354
validation[          , genome  6390] predictions:    6660/  10000 (66.60%), best:    6649/10000 (66.49%), error:      9704.89063, best error:      9680.09668 on epoch:    66, epoch:   72/100, mu: 0.9664503932, learning_rate: 0.0000165338, weight_decay: 0.0000182915

epoch time: 1803.6759033203s, input_fired_time: 22.7655925751, output_fired_time: 12.0464000702, propagate_forward_time: 727.9077758789, propagate_backward_time: 1023.4514160156, weight_update_time: 0.8825808764, other_time: 17.5046386719
epoch time: 150.3978576660s, input_fired_time: 1.8845115900, output_fired_time: 0.0000000000, propagate_forward_time: 145.5881042480, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9252471924
validation[          , genome  6390] predictions:    6613/  10000 (66.13%), best:    6649/10000 (66.49%), error:      9817.44531, best error:      9680.09668 on epoch:    66, epoch:   73/100, mu: 0.9674240351, learning_rate: 0.0000152193, weight_decay: 0.0000174183

epoch time: 1802.0566406250s, input_fired_time: 22.6578388214, output_fired_time: 12.0620508194, propagate_forward_time: 727.1866455078, propagate_backward_time: 1022.3364868164, weight_update_time: 0.8805422187, other_time: 17.8136596680
epoch time: 151.3091735840s, input_fired_time: 1.9146891832, output_fired_time: 0.0000000000, propagate_forward_time: 146.7605895996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6338958740
validation[          , genome  6390] predictions:    6690/  10000 (66.90%), best:    6649/10000 (66.49%), error:      9686.41992, best error:      9680.09668 on epoch:    66, epoch:   74/100, mu: 0.9683574438, learning_rate: 0.0000140094, weight_decay: 0.0000165868

epoch time: 1834.0362548828s, input_fired_time: 22.7295475006, output_fired_time: 12.1890287399, propagate_forward_time: 742.0169677734, propagate_backward_time: 1040.4448242188, weight_update_time: 0.8776009083, other_time: 16.6558837891
epoch time: 154.9680175781s, input_fired_time: 1.9493552446, output_fired_time: 0.0000000000, propagate_forward_time: 150.4969787598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5216827393
validation[          , genome  6390] predictions:    6651/  10000 (66.51%), best:    6649/10000 (66.49%), error:      9696.07910, best error:      9680.09668 on epoch:    66, epoch:   75/100, mu: 0.9692522287, learning_rate: 0.0000128956, weight_decay: 0.0000157950

epoch time: 1838.3393554688s, input_fired_time: 22.7255096436, output_fired_time: 12.2041301727, propagate_forward_time: 745.8768920898, propagate_backward_time: 1041.1673583984, weight_update_time: 0.8787495494, other_time: 16.3656005859
epoch time: 158.9863586426s, input_fired_time: 1.9503191710, output_fired_time: 0.0000000000, propagate_forward_time: 154.4967346191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5393066406
validation[          , genome  6390] predictions:    6678/  10000 (66.78%), best:    6678/10000 (66.78%), error:      9675.75586, best error:      9675.75586 on epoch:    76, epoch:   76/100, mu: 0.9701100588, learning_rate: 0.0000118704, weight_decay: 0.0000150410

epoch time: 1833.1268310547s, input_fired_time: 22.9284915924, output_fired_time: 12.1949510574, propagate_forward_time: 742.9505615234, propagate_backward_time: 1037.9637451172, weight_update_time: 0.8878508806, other_time: 17.0891113281
epoch time: 157.5191192627s, input_fired_time: 1.9385989904, output_fired_time: 0.0000000000, propagate_forward_time: 153.0450286865, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5354919434
validation[          , genome  6390] predictions:    6670/  10000 (66.70%), best:    6670/10000 (66.70%), error:      9665.20605, best error:      9665.20605 on epoch:    77, epoch:   77/100, mu: 0.9709324241, learning_rate: 0.0000109266, weight_decay: 0.0000143230

epoch time: 1850.4530029297s, input_fired_time: 22.9146041870, output_fired_time: 12.2775783539, propagate_forward_time: 751.9117431641, propagate_backward_time: 1046.7513427734, weight_update_time: 0.8925698400, other_time: 16.5977783203
epoch time: 153.9044342041s, input_fired_time: 1.9526357651, output_fired_time: 0.0000000000, propagate_forward_time: 148.7206268311, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2311706543
validation[          , genome  6390] predictions:    6673/  10000 (66.73%), best:    6673/10000 (66.73%), error:      9665.17578, best error:      9665.17578 on epoch:    78, epoch:   78/100, mu: 0.9717207551, learning_rate: 0.0000100580, weight_decay: 0.0000136392

epoch time: 1856.3840332031s, input_fired_time: 22.7626113892, output_fired_time: 12.3050765991, propagate_forward_time: 755.9896240234, propagate_backward_time: 1049.2758789063, weight_update_time: 0.8829455376, other_time: 16.0509033203
epoch time: 153.7276763916s, input_fired_time: 1.9574404955, output_fired_time: 0.0000000000, propagate_forward_time: 149.2378540039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5323791504
validation[          , genome  6390] predictions:    6656/  10000 (66.56%), best:    6673/10000 (66.73%), error:      9689.38379, best error:      9665.17578 on epoch:    78, epoch:   79/100, mu: 0.9724765420, learning_rate: 0.0000092583, weight_decay: 0.0000129881

epoch time: 1841.6350097656s, input_fired_time: 22.7355480194, output_fired_time: 12.2219762802, propagate_forward_time: 748.1845703125, propagate_backward_time: 1042.1376953125, weight_update_time: 0.8799506426, other_time: 16.3552246094
epoch time: 157.8861083984s, input_fired_time: 1.9349640608, output_fired_time: 0.0000000000, propagate_forward_time: 153.4371032715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5140380859
validation[          , genome  6390] predictions:    6648/  10000 (66.48%), best:    6673/10000 (66.73%), error:      9699.87598, best error:      9665.17578 on epoch:    78, epoch:   80/100, mu: 0.9732010365, learning_rate: 0.0000085223, weight_decay: 0.0000123681

epoch time: 1839.3044433594s, input_fired_time: 22.7509765625, output_fired_time: 12.2320117950, propagate_forward_time: 746.1385498047, propagate_backward_time: 1041.4077148438, weight_update_time: 0.8850658536, other_time: 16.7751464844
epoch time: 156.3201293945s, input_fired_time: 1.9434784651, output_fired_time: 0.0000000000, propagate_forward_time: 151.8224945068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5541534424
validation[          , genome  6390] predictions:    6650/  10000 (66.50%), best:    6673/10000 (66.73%), error:      9673.49609, best error:      9665.17578 on epoch:    78, epoch:   81/100, mu: 0.9738956094, learning_rate: 0.0000078447, weight_decay: 0.0000117777

epoch time: 1868.5557861328s, input_fired_time: 22.7737712860, output_fired_time: 12.3305444717, propagate_forward_time: 763.3985595703, propagate_backward_time: 1054.3635253906, weight_update_time: 0.8793705702, other_time: 15.6893310547
epoch time: 156.4757995605s, input_fired_time: 1.9324729443, output_fired_time: 0.0000000000, propagate_forward_time: 152.0167541504, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5265655518
validation[          , genome  6390] predictions:    6636/  10000 (66.36%), best:    6673/10000 (66.73%), error:      9724.52930, best error:      9665.17578 on epoch:    78, epoch:   82/100, mu: 0.9745614529, learning_rate: 0.0000072211, weight_decay: 0.0000112155

epoch time: 1865.9311523438s, input_fired_time: 22.8833713531, output_fired_time: 12.3625335693, propagate_forward_time: 760.1725463867, propagate_backward_time: 1054.3292236328, weight_update_time: 0.8893187642, other_time: 16.1834716797
epoch time: 157.2542419434s, input_fired_time: 1.9595556259, output_fired_time: 0.0000000000, propagate_forward_time: 152.6970214844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5976715088
validation[          , genome  6390] predictions:    6685/  10000 (66.85%), best:    6685/10000 (66.85%), error:      9644.49902, best error:      9644.49902 on epoch:    83, epoch:   83/100, mu: 0.9751997590, learning_rate: 0.0000066470, weight_decay: 0.0000106801

epoch time: 1860.2016601563s, input_fired_time: 22.9346847534, output_fired_time: 12.3188447952, propagate_forward_time: 756.8759765625, propagate_backward_time: 1051.2709960938, weight_update_time: 0.8909885883, other_time: 16.8011474609
epoch time: 156.2527770996s, input_fired_time: 1.9373948574, output_fired_time: 0.0000000000, propagate_forward_time: 151.7196807861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5957031250
validation[          , genome  6390] predictions:    6684/  10000 (66.84%), best:    6685/10000 (66.85%), error:      9650.77148, best error:      9644.49902 on epoch:    83, epoch:   84/100, mu: 0.9758116603, learning_rate: 0.0000061185, weight_decay: 0.0000101702

epoch time: 1852.9915771484s, input_fired_time: 22.9311180115, output_fired_time: 12.2745275497, propagate_forward_time: 754.8940429688, propagate_backward_time: 1046.8190917969, weight_update_time: 0.8888613582, other_time: 16.0727539063
epoch time: 152.1257324219s, input_fired_time: 1.9156651497, output_fired_time: 0.0000000000, propagate_forward_time: 147.3031616211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9069061279
validation[          , genome  6390] predictions:    6643/  10000 (66.43%), best:    6685/10000 (66.85%), error:      9743.52246, best error:      9644.49902 on epoch:    83, epoch:   85/100, mu: 0.9763982892, learning_rate: 0.0000056321, weight_decay: 0.0000096847

epoch time: 1843.8298339844s, input_fired_time: 22.6709651947, output_fired_time: 12.2046813965, propagate_forward_time: 747.7540283203, propagate_backward_time: 1045.2620849609, weight_update_time: 0.8694084883, other_time: 15.9379882813
epoch time: 157.6065216064s, input_fired_time: 1.9849483967, output_fired_time: 0.0000000000, propagate_forward_time: 152.8686218262, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7529449463
validation[          , genome  6390] predictions:    6698/  10000 (66.98%), best:    6698/10000 (66.98%), error:      9631.39844, best error:      9631.39844 on epoch:    86, epoch:   86/100, mu: 0.9769606590, learning_rate: 0.0000051843, weight_decay: 0.0000092224

epoch time: 1834.4389648438s, input_fired_time: 22.8659114838, output_fired_time: 12.1982631683, propagate_forward_time: 743.4489135742, propagate_backward_time: 1038.4200439453, weight_update_time: 0.8866009712, other_time: 17.5057373047
epoch time: 153.1398315430s, input_fired_time: 1.9264034033, output_fired_time: 0.0000000000, propagate_forward_time: 148.4851684570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7282562256
validation[          , genome  6390] predictions:    6688/  10000 (66.88%), best:    6698/10000 (66.98%), error:      9657.85938, best error:      9631.39844 on epoch:    86, epoch:   87/100, mu: 0.9774997830, learning_rate: 0.0000047722, weight_decay: 0.0000087822

epoch time: 1815.1241455078s, input_fired_time: 22.7359142303, output_fired_time: 12.1507291794, propagate_forward_time: 732.7922973633, propagate_backward_time: 1030.0784912109, weight_update_time: 0.8850984573, other_time: 17.3665771484
epoch time: 155.0656890869s, input_fired_time: 1.9780809879, output_fired_time: 0.0000000000, propagate_forward_time: 150.4969024658, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5906982422
validation[          , genome  6390] predictions:    6685/  10000 (66.85%), best:    6698/10000 (66.98%), error:      9638.55566, best error:      9631.39844 on epoch:    86, epoch:   88/100, mu: 0.9780166149, learning_rate: 0.0000043928, weight_decay: 0.0000083629

epoch time: 1836.5694580078s, input_fired_time: 22.8212909698, output_fired_time: 12.2083272934, propagate_forward_time: 744.8244628906, propagate_backward_time: 1040.3001708984, weight_update_time: 0.8802919388, other_time: 16.4151611328
epoch time: 147.2248687744s, input_fired_time: 1.8687101603, output_fired_time: 0.0000000000, propagate_forward_time: 141.9956207275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3605346680
validation[          , genome  6390] predictions:    6686/  10000 (66.86%), best:    6686/10000 (66.86%), error:      9626.49512, best error:      9626.49512 on epoch:    89, epoch:   89/100, mu: 0.9785120487, learning_rate: 0.0000040435, weight_decay: 0.0000079637

epoch time: 1810.8476562500s, input_fired_time: 22.6496715546, output_fired_time: 12.1264410019, propagate_forward_time: 733.4284667969, propagate_backward_time: 1025.9636230469, weight_update_time: 0.8747732639, other_time: 16.6794433594
epoch time: 144.7717895508s, input_fired_time: 1.8533542156, output_fired_time: 0.0000000000, propagate_forward_time: 139.8316497803, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0867919922
validation[          , genome  6390] predictions:    6686/  10000 (66.86%), best:    6686/10000 (66.86%), error:      9652.09180, best error:      9626.49512 on epoch:    89, epoch:   90/100, mu: 0.9789870381, learning_rate: 0.0000037221, weight_decay: 0.0000075835

epoch time: 1840.0314941406s, input_fired_time: 22.8162059784, output_fired_time: 12.2644128799, propagate_forward_time: 747.3592529297, propagate_backward_time: 1040.4530029297, weight_update_time: 0.8862883449, other_time: 17.1386718750
epoch time: 150.6987762451s, input_fired_time: 1.9316626787, output_fired_time: 0.0000000000, propagate_forward_time: 146.1077728271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6593475342
validation[          , genome  6390] predictions:    6713/  10000 (67.13%), best:    6686/10000 (66.86%), error:      9633.49121, best error:      9626.49512 on epoch:    89, epoch:   91/100, mu: 0.9794423580, learning_rate: 0.0000034262, weight_decay: 0.0000072215

epoch time: 1810.0621337891s, input_fired_time: 22.8803844452, output_fired_time: 12.1525211334, propagate_forward_time: 731.3433227539, propagate_backward_time: 1025.8226318359, weight_update_time: 0.8894482255, other_time: 17.8634033203
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6390.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6390_3_r208203758_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.497622
read mu: 0.979879
read mu_delta: 0.958655
read initial_learning_rate: 0.00643755
read learning_rate: 3.15378e-06
read learning_rate_delta: 0.920498
read initial_weight_decay: 0.000619094
read weight_decay: 6.87678e-06
read weight_decay_delta: 0.952263
read batch_size: 76
read epsilon: 1e-07
read alpha: 0.105355
read input_dropout_probability: 0.00123136
read hidden_dropout_probability: 0.0766924
read velocity_reset: 1054
read epoch: 92
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 89
read number_validation_images: 10000
read best_validation_predictions: 6686
read best_validation_error: 9626.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6390
read normal distribution: '1 1.04104495048523 -0.215225517749786'
generator_str: '1829484702'
read generator: 1829484702
read generated_by_map:
1 crossover 1
reading 97 nodes.
reading 425 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1830.40710449219s, input_fired_time: 22.7891063690186, output_fired_time: 12.2488584518433, propagate_forward_time: 742.147827148438, propagate_backward_time: 1036.28002929688, weight_update_time: 0.884516179561615, other_time: 16.9412841796875
epoch time: 154.834106445313s, input_fired_time: 1.94778561592102, output_fired_time: 0, propagate_forward_time: 149.716125488281, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.17019653320313
validation[          , genome  6390] predictions:    6693/  10000 (66.93%), best:    6686/10000 (66.86%), error:      9631.45020, best error:      9626.49512 on epoch:    89, epoch:   92/100, mu: 0.9798788428, learning_rate: 0.0000031538, weight_decay: 0.0000068768

epoch time: 1844.1243896484s, input_fired_time: 23.0364265442, output_fired_time: 12.3068847656, propagate_forward_time: 749.8316040039, propagate_backward_time: 1041.7510986328, weight_update_time: 0.8976787329, other_time: 17.1983642578
epoch time: 157.8529205322s, input_fired_time: 1.9486365318, output_fired_time: 0.0000000000, propagate_forward_time: 153.1415252686, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7627563477
validation[          , genome  6390] predictions:    6690/  10000 (66.90%), best:    6686/10000 (66.86%), error:      9632.04492, best error:      9626.49512 on epoch:    89, epoch:   93/100, mu: 0.9802973270, learning_rate: 0.0000029030, weight_decay: 0.0000065485

epoch time: 1850.7540283203s, input_fired_time: 22.9880256653, output_fired_time: 12.3226900101, propagate_forward_time: 753.0867309570, propagate_backward_time: 1045.6446533203, weight_update_time: 0.8896326423, other_time: 16.7120361328
epoch time: 157.1858062744s, input_fired_time: 1.9858677387, output_fired_time: 0.0000000000, propagate_forward_time: 152.5219879150, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6779479980
validation[          , genome  6390] predictions:    6686/  10000 (66.86%), best:    6686/10000 (66.86%), error:      9646.18652, best error:      9626.49512 on epoch:    89, epoch:   94/100, mu: 0.9806984663, learning_rate: 0.0000026722, weight_decay: 0.0000062359

epoch time: 1833.6914062500s, input_fired_time: 22.8619174957, output_fired_time: 12.2543411255, propagate_forward_time: 742.9844970703, propagate_backward_time: 1038.8460693359, weight_update_time: 0.8862211704, other_time: 16.7445068359
epoch time: 157.3176879883s, input_fired_time: 1.9463129044, output_fired_time: 0.0000000000, propagate_forward_time: 152.7295379639, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6418304443
validation[          , genome  6390] predictions:    6673/  10000 (66.73%), best:    6686/10000 (66.86%), error:      9637.97363, best error:      9626.49512 on epoch:    89, epoch:   95/100, mu: 0.9810830355, learning_rate: 0.0000024598, weight_decay: 0.0000059382

epoch time: 1829.2880859375s, input_fired_time: 22.9028129578, output_fired_time: 12.2390346527, propagate_forward_time: 739.6862182617, propagate_backward_time: 1036.6185302734, weight_update_time: 0.8951578140, other_time: 17.8414306641
epoch time: 146.3280487061s, input_fired_time: 1.8918691874, output_fired_time: 0.0000000000, propagate_forward_time: 141.2919006348, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1442718506
validation[          , genome  6390] predictions:    6696/  10000 (66.96%), best:    6696/10000 (66.96%), error:      9607.97070, best error:      9607.97070 on epoch:    96, epoch:   96/100, mu: 0.9814516902, learning_rate: 0.0000022642, weight_decay: 0.0000056547

epoch time: 1809.9412841797s, input_fired_time: 22.9015445709, output_fired_time: 12.1230039597, propagate_forward_time: 730.4676513672, propagate_backward_time: 1026.2241210938, weight_update_time: 0.8906581402, other_time: 18.2249755859
epoch time: 150.0959930420s, input_fired_time: 1.9295411110, output_fired_time: 0.0000000000, propagate_forward_time: 144.9285278320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2379302979
validation[          , genome  6390] predictions:    6694/  10000 (66.94%), best:    6696/10000 (66.96%), error:      9631.72266, best error:      9607.97070 on epoch:    96, epoch:   97/100, mu: 0.9818051457, learning_rate: 0.0000020842, weight_decay: 0.0000053848

epoch time: 1790.5369873047s, input_fired_time: 23.0144844055, output_fired_time: 12.1096839905, propagate_forward_time: 722.7486572266, propagate_backward_time: 1014.5977172852, weight_update_time: 0.8941302299, other_time: 18.0663452148
epoch time: 155.8804626465s, input_fired_time: 1.9692451954, output_fired_time: 0.0000000000, propagate_forward_time: 151.1551666260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7560577393
validation[          , genome  6390] predictions:    6718/  10000 (67.18%), best:    6718/10000 (67.18%), error:      9604.50684, best error:      9604.50684 on epoch:    98, epoch:   98/100, mu: 0.9821439385, learning_rate: 0.0000019185, weight_decay: 0.0000051277

epoch time: 1802.1473388672s, input_fired_time: 22.7273349762, output_fired_time: 12.1388978958, propagate_forward_time: 728.8211669922, propagate_backward_time: 1020.8321533203, weight_update_time: 0.8859456182, other_time: 17.6278076172
epoch time: 147.1790008545s, input_fired_time: 1.8788050413, output_fired_time: 0.0000000000, propagate_forward_time: 142.4938507080, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8063507080
validation[          , genome  6390] predictions:    6673/  10000 (66.73%), best:    6718/10000 (67.18%), error:      9648.84570, best error:      9604.50684 on epoch:    98, epoch:   99/100, mu: 0.9824687243, learning_rate: 0.0000017660, weight_decay: 0.0000048830

epoch time: 1813.4265136719s, input_fired_time: 22.8134155273, output_fired_time: 12.1561622620, propagate_forward_time: 734.2905883789, propagate_backward_time: 1026.5322265625, weight_update_time: 0.8881886005, other_time: 17.6340332031
epoch time: 148.8429260254s, input_fired_time: 1.8954803944, output_fired_time: 0.0000000000, propagate_forward_time: 144.0306701660, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9167785645
validation[          , genome  6390] predictions:    6694/  10000 (66.94%), best:    6718/10000 (67.18%), error:      9630.21680, best error:      9604.50684 on epoch:    98, epoch:  100/100, mu: 0.9827800989, learning_rate: 0.0000016256, weight_decay: 0.0000046499

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 762.6696166992s, input_fired_time: 9.6610183716, output_fired_time: 0.0000000000, propagate_forward_time: 738.4967041016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 14.5119018555
best training[          , genome  6390] predictions:   37018/  50000 (74.04%), best:    6718/10000 (67.18%), error:     40161.09766, best error:      9604.50684 on epoch:    98, epoch:  101/100, mu: 0.9830785990, learning_rate: 0.0000014964, weight_decay: 0.0000044279
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 150.5650177002s, input_fired_time: 1.9405255318, output_fired_time: 0.0000000000, propagate_forward_time: 145.2654266357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3590698242
   testing[          , genome  6390] predictions:    6718/  10000 (67.18%), best:    6718/10000 (67.18%), error:      9604.50684, best error:      9604.50684 on epoch:    98, epoch:  101/100, mu: 0.9830785990, learning_rate: 0.0000014964, weight_decay: 0.0000044279
backpropagation finished successfully!
08:20:54 (14016): called boinc_finish(0)

</stderr_txt>
]]>