Nameexact_genome_1542031203_187_5525_3
Workunit4079881
Created27 Jan 2019, 3:09:49 UTC
Sent27 Jan 2019, 3:10:13 UTC
Report deadline12 Apr 2019, 14:17:39 UTC
Received28 Jan 2019, 16:06:18 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID23692
Run time1 days 10 hours 18 min 46 sec
CPU time1 days 7 hours 18 min 21 sec
Validate stateValid
Credit18,865.87
Device peak FLOPS4.16 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size543.79 MB
Peak swap size563.53 MB
Peak disk usage10.20 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
pagate_forward_time: 463.1506347656, propagate_backward_time: 678.8526611328, weight_update_time: 0.6629487276, other_time: 12.3024902344
epoch time: 91.2092819214s, input_fired_time: 1.1304560900, output_fired_time: 0.0000000000, propagate_forward_time: 88.1456375122, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9331893921
validation[          , genome  5525] predictions:    6355/  10000 (63.55%), best:    6355/10000 (63.55%), error:     10452.65137, best error:     10452.65137 on epoch:    35, epoch:   35/100, mu: 0.9128856063, learning_rate: 0.0001570569, weight_decay: 0.0000802114

epoch time: 1104.8802490234s, input_fired_time: 14.5915708542, output_fired_time: 7.7849149704, propagate_forward_time: 441.0594787598, propagate_backward_time: 629.7326660156, weight_update_time: 0.6278991103, other_time: 11.7116699219
epoch time: 93.4993896484s, input_fired_time: 1.2088198662, output_fired_time: 0.0000000000, propagate_forward_time: 90.2753067017, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0152664185
validation[          , genome  5525] predictions:    6266/  10000 (62.66%), best:    6355/10000 (63.55%), error:     10583.44727, best error:     10452.65137 on epoch:    35, epoch:   36/100, mu: 0.9168173671, learning_rate: 0.0001454075, weight_decay: 0.0000759489

epoch time: 1133.2620849609s, input_fired_time: 14.9774942398, output_fired_time: 8.1215028763, propagate_forward_time: 452.0219116211, propagate_backward_time: 646.2130737305, weight_update_time: 0.6056401134, other_time: 11.9281005859
epoch time: 93.9931869507s, input_fired_time: 1.1877557039, output_fired_time: 0.0000000000, propagate_forward_time: 90.8857345581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9196929932
validation[          , genome  5525] predictions:    6340/  10000 (63.40%), best:    6355/10000 (63.55%), error:     10549.30859, best error:     10452.65137 on epoch:    35, epoch:   37/100, mu: 0.9205486774, learning_rate: 0.0001346221, weight_decay: 0.0000719129

epoch time: 1163.7871093750s, input_fired_time: 15.2721366882, output_fired_time: 8.5048608780, propagate_forward_time: 462.6462707520, propagate_backward_time: 665.2488403320, weight_update_time: 0.6112635732, other_time: 12.1150512695
epoch time: 100.7904052734s, input_fired_time: 1.2966482639, output_fired_time: 0.0000000000, propagate_forward_time: 97.3597640991, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1339950562
validation[          , genome  5525] predictions:    6390/  10000 (63.90%), best:    6355/10000 (63.55%), error:     10460.07422, best error:     10452.65137 on epoch:    35, epoch:   38/100, mu: 0.9240897298, learning_rate: 0.0001246368, weight_decay: 0.0000680914

epoch time: 1174.2398681641s, input_fired_time: 15.6126184464, output_fired_time: 8.4015245438, propagate_forward_time: 467.7644348145, propagate_backward_time: 670.2110595703, weight_update_time: 0.6731124520, other_time: 12.2502441406
epoch time: 98.1600341797s, input_fired_time: 1.2227993011, output_fired_time: 0.0000000000, propagate_forward_time: 94.8467178345, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0905151367
validation[          , genome  5525] predictions:    6378/  10000 (63.78%), best:    6355/10000 (63.55%), error:     10542.42480, best error:     10452.65137 on epoch:    35, epoch:   39/100, mu: 0.9274502397, learning_rate: 0.0001153920, weight_decay: 0.0000644730

epoch time: 1210.1988525391s, input_fired_time: 16.1549053192, output_fired_time: 8.6367301941, propagate_forward_time: 478.1669616699, propagate_backward_time: 694.1660156250, weight_update_time: 0.6890071630, other_time: 13.0742187500
epoch time: 101.2443923950s, input_fired_time: 1.2674256563, output_fired_time: 0.0000000000, propagate_forward_time: 97.7846069336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1923599243
validation[          , genome  5525] predictions:    6338/  10000 (63.38%), best:    6355/10000 (63.55%), error:     10620.84961, best error:     10452.65137 on epoch:    35, epoch:   40/100, mu: 0.9306394458, learning_rate: 0.0001068330, weight_decay: 0.0000610468

epoch time: 1266.4957275391s, input_fired_time: 16.8195190430, output_fired_time: 9.1532325745, propagate_forward_time: 498.8346557617, propagate_backward_time: 727.7965087891, weight_update_time: 0.7232708335, other_time: 13.8919067383
epoch time: 101.1929779053s, input_fired_time: 1.2486835718, output_fired_time: 0.0000000000, propagate_forward_time: 97.6210861206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3232116699
validation[          , genome  5525] predictions:    6415/  10000 (64.15%), best:    6415/10000 (64.15%), error:     10331.34668, best error:     10331.34668 on epoch:    41, epoch:   41/100, mu: 0.9336659908, learning_rate: 0.0000989089, weight_decay: 0.0000578027

epoch time: 1251.3475341797s, input_fired_time: 16.4477825165, output_fired_time: 8.7156105042, propagate_forward_time: 493.9496765137, propagate_backward_time: 718.9265747070, weight_update_time: 0.7536355853, other_time: 13.3079223633
epoch time: 103.0267028809s, input_fired_time: 1.2684071064, output_fired_time: 0.0000000000, propagate_forward_time: 99.5797424316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1785507202
validation[          , genome  5525] predictions:    6432/  10000 (64.32%), best:    6415/10000 (64.15%), error:     10338.54004, best error:     10331.34668 on epoch:    41, epoch:   42/100, mu: 0.9365382791, learning_rate: 0.0000915725, weight_decay: 0.0000547310

epoch time: 1301.1688232422s, input_fired_time: 17.4966392517, output_fired_time: 9.0197296143, propagate_forward_time: 514.8755493164, propagate_backward_time: 745.9584960938, weight_update_time: 0.7755575776, other_time: 13.8184204102
epoch time: 107.3827209473s, input_fired_time: 1.2790820599, output_fired_time: 0.0000000000, propagate_forward_time: 103.7802886963, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3233489990
validation[          , genome  5525] predictions:    6367/  10000 (63.67%), best:    6415/10000 (64.15%), error:     10515.84961, best error:     10331.34668 on epoch:    41, epoch:   43/100, mu: 0.9392641187, learning_rate: 0.0000847802, weight_decay: 0.0000518226

epoch time: 1235.1920166016s, input_fired_time: 16.4424915314, output_fired_time: 8.7358684540, propagate_forward_time: 488.0876770020, propagate_backward_time: 709.1431274414, weight_update_time: 0.7419106364, other_time: 12.7828979492
epoch time: 106.6517181396s, input_fired_time: 1.3538714647, output_fired_time: 0.0000000000, propagate_forward_time: 103.0721588135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2256851196
validation[          , genome  5525] predictions:    6440/  10000 (64.40%), best:    6415/10000 (64.15%), error:     10450.54102, best error:     10331.34668 on epoch:    41, epoch:   44/100, mu: 0.9418509603, learning_rate: 0.0000784918, weight_decay: 0.0000490687

epoch time: 1238.6367187500s, input_fired_time: 16.4930686951, output_fired_time: 8.7209434509, propagate_forward_time: 489.7943420410, propagate_backward_time: 710.5588989258, weight_update_time: 0.6759163141, other_time: 13.0695190430
epoch time: 108.3936843872s, input_fired_time: 1.4077886343, output_fired_time: 0.0000000000, propagate_forward_time: 104.7239990234, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2618942261
validation[          , genome  5525] predictions:    6400/  10000 (64.00%), best:    6400/10000 (64.00%), error:     10286.45313, best error:     10286.45313 on epoch:    45, epoch:   45/100, mu: 0.9443058968, learning_rate: 0.0000726698, weight_decay: 0.0000464611

epoch time: 1302.7307128906s, input_fired_time: 17.5721015930, output_fired_time: 8.9672241211, propagate_forward_time: 514.5794067383, propagate_backward_time: 747.9707641602, weight_update_time: 0.8109793663, other_time: 13.6412353516
epoch time: 105.7705917358s, input_fired_time: 1.4161742926, output_fired_time: 0.0000000000, propagate_forward_time: 102.1335983276, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2208175659
validation[          , genome  5525] predictions:    6369/  10000 (63.69%), best:    6400/10000 (64.00%), error:     10522.87793, best error:     10286.45313 on epoch:    45, epoch:   46/100, mu: 0.9466356635, learning_rate: 0.0000672796, weight_decay: 0.0000439921

epoch time: 1276.6458740234s, input_fired_time: 17.1527709961, output_fired_time: 8.9415369034, propagate_forward_time: 501.1761169434, propagate_backward_time: 735.8353881836, weight_update_time: 0.7008417249, other_time: 13.5401000977
epoch time: 101.6160583496s, input_fired_time: 1.3965680599, output_fired_time: 0.0000000000, propagate_forward_time: 97.9970474243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2224426270
validation[          , genome  5525] predictions:    6408/  10000 (64.08%), best:    6400/10000 (64.00%), error:     10360.07617, best error:     10286.45313 on epoch:    45, epoch:   47/100, mu: 0.9488466382, learning_rate: 0.0000622893, weight_decay: 0.0000416544

epoch time: 1221.9195556641s, input_fired_time: 16.3692665100, output_fired_time: 8.5663747787, propagate_forward_time: 482.5699462891, propagate_backward_time: 701.6916503906, weight_update_time: 0.7123382092, other_time: 12.7222900391
epoch time: 88.6758575439s, input_fired_time: 1.1370096207, output_fired_time: 0.0000000000, propagate_forward_time: 85.6572341919, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8816146851
validation[          , genome  5525] predictions:    6476/  10000 (64.76%), best:    6476/10000 (64.76%), error:     10265.05469, best error:     10265.05469 on epoch:    48, epoch:   48/100, mu: 0.9509449005, learning_rate: 0.0000576691, weight_decay: 0.0000394408

epoch time: 1167.2152099609s, input_fired_time: 15.7381277084, output_fired_time: 8.2805051804, propagate_forward_time: 460.2011718750, propagate_backward_time: 670.8753051758, weight_update_time: 0.6637546420, other_time: 12.1200561523
epoch time: 100.1323699951s, input_fired_time: 1.3035589457, output_fired_time: 0.0000000000, propagate_forward_time: 96.6894836426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1393280029
validation[          , genome  5525] predictions:    6465/  10000 (64.65%), best:    6465/10000 (64.65%), error:     10224.82910, best error:     10224.82910 on epoch:    49, epoch:   49/100, mu: 0.9529361725, learning_rate: 0.0000533916, weight_decay: 0.0000373449

epoch time: 1170.0728759766s, input_fired_time: 15.8051853180, output_fired_time: 8.3284044266, propagate_forward_time: 458.6707458496, propagate_backward_time: 675.1389770508, weight_update_time: 0.6953312755, other_time: 12.1295776367
epoch time: 97.7263641357s, input_fired_time: 1.2691093683, output_fired_time: 0.0000000000, propagate_forward_time: 94.4132995605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0439529419
validation[          , genome  5525] predictions:    6499/  10000 (64.99%), best:    6499/10000 (64.99%), error:     10132.84668, best error:     10132.84668 on epoch:    50, epoch:   50/100, mu: 0.9548259377, learning_rate: 0.0000494313, weight_decay: 0.0000353603

epoch time: 1207.4976806641s, input_fired_time: 16.1615371704, output_fired_time: 8.2421665192, propagate_forward_time: 471.9375000000, propagate_backward_time: 698.7144165039, weight_update_time: 0.6830231547, other_time: 12.4420776367
epoch time: 92.1325759888s, input_fired_time: 1.1680427790, output_fired_time: 0.0000000000, propagate_forward_time: 89.1456222534, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8189086914
validation[          , genome  5525] predictions:    6348/  10000 (63.48%), best:    6499/10000 (64.99%), error:     10490.90625, best error:     10132.84668 on epoch:    50, epoch:   51/100, mu: 0.9566193223, learning_rate: 0.0000457649, weight_decay: 0.0000334813

epoch time: 1165.8294677734s, input_fired_time: 15.5302238464, output_fired_time: 8.1461181641, propagate_forward_time: 458.2231750488, propagate_backward_time: 672.1445922852, weight_update_time: 0.6771857142, other_time: 11.7853393555
epoch time: 96.0210494995s, input_fired_time: 1.2439371347, output_fired_time: 0.0000000000, propagate_forward_time: 92.8178024292, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9593124390
validation[          , genome  5525] predictions:    6520/  10000 (65.20%), best:    6499/10000 (64.99%), error:     10181.52832, best error:     10132.84668 on epoch:    50, epoch:   52/100, mu: 0.9583212733, learning_rate: 0.0000423703, weight_decay: 0.0000317020

epoch time: 1193.5477294922s, input_fired_time: 15.9738750458, output_fired_time: 8.3277225494, propagate_forward_time: 471.6658325195, propagate_backward_time: 685.2075805664, weight_update_time: 0.6813443303, other_time: 12.3726806641
epoch time: 95.7830505371s, input_fired_time: 1.2376849651, output_fired_time: 0.0000000000, propagate_forward_time: 92.5672988892, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9780654907
validation[          , genome  5525] predictions:    6522/  10000 (65.22%), best:    6499/10000 (64.99%), error:     10184.85059, best error:     10132.84668 on epoch:    50, epoch:   53/100, mu: 0.9599364400, learning_rate: 0.0000392276, weight_decay: 0.0000300174

epoch time: 1175.7410888672s, input_fired_time: 15.7837209702, output_fired_time: 8.2293758392, propagate_forward_time: 463.6014404297, propagate_backward_time: 676.0618286133, weight_update_time: 0.6980404258, other_time: 12.0647583008
epoch time: 90.9620056152s, input_fired_time: 1.2028883696, output_fired_time: 0.0000000000, propagate_forward_time: 87.8939056396, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8652114868
validation[          , genome  5525] predictions:    6463/  10000 (64.63%), best:    6499/10000 (64.99%), error:     10188.14258, best error:     10132.84668 on epoch:    50, epoch:   54/100, mu: 0.9614692926, learning_rate: 0.0000363179, weight_decay: 0.0000284222

epoch time: 1169.4467773438s, input_fired_time: 15.4478559494, output_fired_time: 8.1288738251, propagate_forward_time: 457.6341552734, propagate_backward_time: 676.2984619141, weight_update_time: 0.6744895577, other_time: 11.9373779297
epoch time: 93.7113876343s, input_fired_time: 1.1942340136, output_fired_time: 0.0000000000, propagate_forward_time: 90.5391464233, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9780044556
validation[          , genome  5525] predictions:    6398/  10000 (63.98%), best:    6499/10000 (64.99%), error:     10320.63965, best error:     10132.84668 on epoch:    50, epoch:   55/100, mu: 0.9629239440, learning_rate: 0.0000336241, weight_decay: 0.0000269118

epoch time: 1155.4591064453s, input_fired_time: 15.3694763184, output_fired_time: 7.8637676239, propagate_forward_time: 448.6625671387, propagate_backward_time: 671.6552124023, weight_update_time: 0.6504803896, other_time: 11.9080200195
epoch time: 98.3422012329s, input_fired_time: 1.2927153111, output_fired_time: 0.0000000000, propagate_forward_time: 94.9572677612, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0922164917
validation[          , genome  5525] predictions:    6442/  10000 (64.42%), best:    6499/10000 (64.99%), error:     10272.62012, best error:     10132.84668 on epoch:    50, epoch:   56/100, mu: 0.9643044472, learning_rate: 0.0000311301, weight_decay: 0.0000254817

epoch time: 1140.0399169922s, input_fired_time: 15.2376451492, output_fired_time: 7.7496805191, propagate_forward_time: 441.0389099121, propagate_backward_time: 664.1638183594, weight_update_time: 0.6697823405, other_time: 11.8498535156
epoch time: 101.1054992676s, input_fired_time: 1.4094691277, output_fired_time: 0.0000000000, propagate_forward_time: 97.5331954956, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1628341675
validation[          , genome  5525] predictions:    6538/  10000 (65.38%), best:    6538/10000 (65.38%), error:     10091.07227, best error:     10091.07227 on epoch:    57, epoch:   57/100, mu: 0.9656145573, learning_rate: 0.0000288211, weight_decay: 0.0000241276

epoch time: 1121.9259033203s, input_fired_time: 14.7799816132, output_fired_time: 7.5980935097, propagate_forward_time: 433.1658325195, propagate_backward_time: 654.6791992188, weight_update_time: 0.6186813712, other_time: 11.7026977539
epoch time: 85.4517822266s, input_fired_time: 1.1505873203, output_fired_time: 0.0000000000, propagate_forward_time: 82.3890609741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9121322632
validation[          , genome  5525] predictions:    6451/  10000 (64.51%), best:    6538/10000 (65.38%), error:     10266.19824, best error:     10091.07227 on epoch:    57, epoch:   58/100, mu: 0.9668578506, learning_rate: 0.0000266833, weight_decay: 0.0000228454

epoch time: 1133.5711669922s, input_fired_time: 15.6709365845, output_fired_time: 7.8150253296, propagate_forward_time: 433.4870300293, propagate_backward_time: 664.7222900391, weight_update_time: 0.6633737087, other_time: 11.8758544922
epoch time: 92.1328735352s, input_fired_time: 1.2210546732, output_fired_time: 0.0000000000, propagate_forward_time: 88.8438568115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0679626465
validation[          , genome  5525] predictions:    6515/  10000 (65.15%), best:    6515/10000 (65.15%), error:     10057.42188, best error:     10057.42188 on epoch:    59, epoch:   59/100, mu: 0.9680377841, learning_rate: 0.0000247041, weight_decay: 0.0000216314

epoch time: 1148.1557617188s, input_fired_time: 15.4941177368, output_fired_time: 7.8772821426, propagate_forward_time: 437.6561584473, propagate_backward_time: 675.2623291016, weight_update_time: 0.6717089415, other_time: 11.8658447266
epoch time: 89.6450119019s, input_fired_time: 1.2273238897, output_fired_time: 0.0000000000, propagate_forward_time: 86.3980255127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0196609497
validation[          , genome  5525] predictions:    6557/  10000 (65.57%), best:    6515/10000 (65.15%), error:     10115.84863, best error:     10057.42188 on epoch:    59, epoch:   60/100, mu: 0.9691575766, learning_rate: 0.0000228717, weight_decay: 0.0000204819

epoch time: 1169.3541259766s, input_fired_time: 15.7610301971, output_fired_time: 7.9996361732, propagate_forward_time: 445.5570068359, propagate_backward_time: 687.7957153320, weight_update_time: 0.6818311810, other_time: 12.2407836914
epoch time: 96.6528472900s, input_fired_time: 1.3273366690, output_fired_time: 0.0000000000, propagate_forward_time: 93.2128982544, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1126098633
validation[          , genome  5525] predictions:    6450/  10000 (64.50%), best:    6515/10000 (65.15%), error:     10230.29688, best error:     10057.42188 on epoch:    59, epoch:   61/100, mu: 0.9702202678, learning_rate: 0.0000211753, weight_decay: 0.0000193935

epoch time: 1206.4448242188s, input_fired_time: 15.9716386795, output_fired_time: 8.3210048676, propagate_forward_time: 459.5498046875, propagate_backward_time: 710.0352783203, weight_update_time: 0.6967980862, other_time: 12.5670166016
epoch time: 101.4198379517s, input_fired_time: 1.3765617609, output_fired_time: 0.0000000000, propagate_forward_time: 97.7692337036, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2740402222
validation[          , genome  5525] predictions:    6515/  10000 (65.15%), best:    6515/10000 (65.15%), error:     10174.61035, best error:     10057.42188 on epoch:    59, epoch:   62/100, mu: 0.9712287784, learning_rate: 0.0000196046, weight_decay: 0.0000183629

epoch time: 1193.0975341797s, input_fired_time: 16.0650310516, output_fired_time: 8.0906915665, propagate_forward_time: 456.0926208496, propagate_backward_time: 700.5532836914, weight_update_time: 0.6535903811, other_time: 12.2958374023
epoch time: 94.4341812134s, input_fired_time: 1.2981736660, output_fired_time: 0.0000000000, propagate_forward_time: 91.0513305664, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0846786499
validation[          , genome  5525] predictions:    6530/  10000 (65.30%), best:    6515/10000 (65.15%), error:     10152.70898, best error:     10057.42188 on epoch:    59, epoch:   63/100, mu: 0.9721858501, learning_rate: 0.0000181505, weight_decay: 0.0000173870

epoch time: 1144.0891113281s, input_fired_time: 15.5542612076, output_fired_time: 7.8825035095, propagate_forward_time: 436.8804626465, propagate_backward_time: 671.5741577148, weight_update_time: 0.6791042089, other_time: 12.1976928711
epoch time: 95.4859924316s, input_fired_time: 1.2737400532, output_fired_time: 0.0000000000, propagate_forward_time: 92.0912017822, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1210479736
validation[          , genome  5525] predictions:    6428/  10000 (64.28%), best:    6515/10000 (65.15%), error:     10245.29590, best error:     10057.42188 on epoch:    59, epoch:   64/100, mu: 0.9730941057, learning_rate: 0.0000168042, weight_decay: 0.0000164631

epoch time: 1155.8972167969s, input_fired_time: 15.6429929733, output_fired_time: 7.9534773827, propagate_forward_time: 443.3996887207, propagate_backward_time: 676.5828857422, weight_update_time: 0.6547208428, other_time: 12.3182373047
epoch time: 83.4509735107s, input_fired_time: 1.1336206198, output_fired_time: 0.0000000000, propagate_forward_time: 80.5084304810, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8089218140
validation[          , genome  5525] predictions:    6566/  10000 (65.66%), best:    6566/10000 (65.66%), error:      9981.10254, best error:      9981.10254 on epoch:    65, epoch:   65/100, mu: 0.9739560485, learning_rate: 0.0000155578, weight_decay: 0.0000155882

epoch time: 1124.6922607422s, input_fired_time: 15.1407260895, output_fired_time: 7.7818989754, propagate_forward_time: 429.6941528320, propagate_backward_time: 660.1951904297, weight_update_time: 0.6636797786, other_time: 11.8803100586
epoch time: 102.8800659180s, input_fired_time: 1.3612188101, output_fired_time: 0.0000000000, propagate_forward_time: 99.2780685425, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2407760620
validation[          , genome  5525] predictions:    6521/  10000 (65.21%), best:    6566/10000 (65.66%), error:     10104.83594, best error:      9981.10254 on epoch:    65, epoch:   66/100, mu: 0.9747740626, learning_rate: 0.0000144038, weight_decay: 0.0000147598

epoch time: 1307.7297363281s, input_fired_time: 17.6749057770, output_fired_time: 9.0664978027, propagate_forward_time: 501.2432861328, propagate_backward_time: 765.8519897461, weight_update_time: 0.7073884010, other_time: 13.8930053711
epoch time: 104.4194869995s, input_fired_time: 1.4363923073, output_fired_time: 0.0000000000, propagate_forward_time: 100.6622467041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3208465576
validation[          , genome  5525] predictions:    6556/  10000 (65.56%), best:    6566/10000 (65.66%), error:      9988.32129, best error:      9981.10254 on epoch:    65, epoch:   67/100, mu: 0.9755503535, learning_rate: 0.0000133354, weight_decay: 0.0000139755

epoch time: 1318.2937011719s, input_fired_time: 17.5721454620, output_fired_time: 8.9839477539, propagate_forward_time: 504.3217773438, propagate_backward_time: 773.6082153320, weight_update_time: 0.7024898529, other_time: 13.8075561523
epoch time: 98.7053375244s, input_fired_time: 1.4042013884, output_fired_time: 0.0000000000, propagate_forward_time: 95.1822128296, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1189270020
validation[          , genome  5525] predictions:    6593/  10000 (65.93%), best:    6593/10000 (65.93%), error:      9978.44531, best error:      9978.44531 on epoch:    68, epoch:   68/100, mu: 0.9762870669, learning_rate: 0.0000123463, weight_decay: 0.0000132328

epoch time: 1197.4996337891s, input_fired_time: 16.3010082245, output_fired_time: 8.1405229568, propagate_forward_time: 458.4695739746, propagate_backward_time: 701.9123535156, weight_update_time: 0.6334033012, other_time: 12.6761474609
epoch time: 100.4334945679s, input_fired_time: 1.3453949690, output_fired_time: 0.0000000000, propagate_forward_time: 96.8013381958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2867584229
validation[          , genome  5525] predictions:    6523/  10000 (65.23%), best:    6593/10000 (65.93%), error:     10039.87891, best error:      9978.44531 on epoch:    68, epoch:   69/100, mu: 0.9769862294, learning_rate: 0.0000114305, weight_decay: 0.0000125296

epoch time: 1119.8679199219s, input_fired_time: 15.0872859955, output_fired_time: 7.6415815353, propagate_forward_time: 429.0122985840, propagate_backward_time: 656.3782348633, weight_update_time: 0.6339898705, other_time: 11.7484741211
epoch time: 85.2516479492s, input_fired_time: 1.1619318724, output_fired_time: 0.0000000000, propagate_forward_time: 82.1649398804, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9247741699
validation[          , genome  5525] predictions:    6499/  10000 (64.99%), best:    6593/10000 (65.93%), error:     10158.39453, best error:      9978.44531 on epoch:    68, epoch:   70/100, mu: 0.9776497483, learning_rate: 0.0000105827, weight_decay: 0.0000118638

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525_3_r165493131_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508499
read mu: 0.978279
read mu_delta: 0.949014
read initial_learning_rate: 0.00233087
read learning_rate: 9.79774e-06
read learning_rate_delta: 0.925827
read initial_weight_decay: 0.000542312
read weight_decay: 1.12333e-05
read weight_decay_delta: 0.946859
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0741988
read input_dropout_probability: 0.00114977
read hidden_dropout_probability: 0.092632
read velocity_reset: 1991
read epoch: 71
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 68
read number_validation_images: 10000
read best_validation_predictions: 6593
read best_validation_error: 9978.45
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5525
read normal distribution: '1 0 0'
generator_str: '1090297904'
read generator: 1090297904
read generated_by_map:
1 enable_edge 1
reading 77 nodes.
reading 326 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525_3_r165493131_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508499
read mu: 0.978279
read mu_delta: 0.949014
read initial_learning_rate: 0.00233087
read learning_rate: 9.79774e-06
read learning_rate_delta: 0.925827
read initial_weight_decay: 0.000542312
read weight_decay: 1.12333e-05
read weight_decay_delta: 0.946859
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0741988
read input_dropout_probability: 0.00114977
read hidden_dropout_probability: 0.092632
read velocity_reset: 1991
read epoch: 71
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 68
read number_validation_images: 10000
read best_validation_predictions: 6593
read best_validation_error: 9978.45
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5525
read normal distribution: '1 0 0'
generator_str: '1090297904'
read generator: 1090297904
read generated_by_map:
1 enable_edge 1
reading 77 nodes.
reading 326 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1079.72717285156s, input_fired_time: 14.5599012374878, output_fired_time: 7.35509061813354, propagate_forward_time: 411.964080810547, propagate_backward_time: 634.631530761719, weight_update_time: 0.596714019775391, other_time: 11.2164916992188
epoch time: 81.733154296875s, input_fired_time: 1.09924602508545, output_fired_time: 0, propagate_forward_time: 78.899299621582, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.73461151123047
validation[          , genome  5525] predictions:    6572/  10000 (65.72%), best:    6593/10000 (65.93%), error:     10004.49316, best error:      9978.44531 on epoch:    68, epoch:   71/100, mu: 0.9782794118, learning_rate: 0.0000097977, weight_decay: 0.0000112333

epoch time: 1026.9339599609s, input_fired_time: 13.7520904541, output_fired_time: 7.0112853050, propagate_forward_time: 393.0501403809, propagate_backward_time: 602.5816040039, weight_update_time: 0.5726268888, other_time: 10.5388793945
epoch time: 81.8604888916s, input_fired_time: 1.0950458050, output_fired_time: 0.0000000000, propagate_forward_time: 79.0017166138, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7637252808
validation[          , genome  5525] predictions:    6521/  10000 (65.21%), best:    6593/10000 (65.93%), error:     10046.85352, best error:      9978.44531 on epoch:    68, epoch:   72/100, mu: 0.9788770080, learning_rate: 0.0000090710, weight_decay: 0.0000106364

epoch time: 1026.8421630859s, input_fired_time: 13.7680854797, output_fired_time: 6.9420938492, propagate_forward_time: 391.4038391113, propagate_backward_time: 604.2385253906, weight_update_time: 0.5719498396, other_time: 10.4896240234
epoch time: 80.7568359375s, input_fired_time: 1.0916413069, output_fired_time: 0.0000000000, propagate_forward_time: 77.9262008667, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7389907837
validation[          , genome  5525] predictions:    6582/  10000 (65.82%), best:    6582/10000 (65.82%), error:      9947.85645, best error:      9947.85645 on epoch:    73, epoch:   73/100, mu: 0.9794441462, learning_rate: 0.0000083982, weight_decay: 0.0000100712

epoch time: 1022.6133422852s, input_fired_time: 13.7124767303, output_fired_time: 6.9353322983, propagate_forward_time: 389.7363281250, propagate_backward_time: 601.7611083984, weight_update_time: 0.5693865418, other_time: 10.4681396484
epoch time: 80.7845611572s, input_fired_time: 1.1019521952, output_fired_time: 0.0000000000, propagate_forward_time: 77.9415817261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7410278320
validation[          , genome  5525] predictions:    6574/  10000 (65.74%), best:    6582/10000 (65.82%), error:      9995.48535, best error:      9947.85645 on epoch:    73, epoch:   74/100, mu: 0.9799823761, learning_rate: 0.0000077753, weight_decay: 0.0000095360

epoch time: 1024.5090332031s, input_fired_time: 13.7222890854, output_fired_time: 6.9304323196, propagate_forward_time: 389.9713134766, propagate_backward_time: 603.4291381836, weight_update_time: 0.5711116791, other_time: 10.4558715820
epoch time: 80.8620758057s, input_fired_time: 1.0966606140, output_fired_time: 0.0000000000, propagate_forward_time: 78.0163955688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7490158081
validation[          , genome  5525] predictions:    6529/  10000 (65.29%), best:    6582/10000 (65.82%), error:     10086.94922, best error:      9947.85645 on epoch:    73, epoch:   75/100, mu: 0.9804931283, learning_rate: 0.0000071985, weight_decay: 0.0000090292

epoch time: 1024.7219238281s, input_fired_time: 13.7372684479, output_fired_time: 6.9800562859, propagate_forward_time: 390.2074584961, propagate_backward_time: 603.2106323242, weight_update_time: 0.5735367537, other_time: 10.5865478516
epoch time: 80.9685516357s, input_fired_time: 1.0993541479, output_fired_time: 0.0000000000, propagate_forward_time: 78.1502609253, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7189331055
validation[          , genome  5525] predictions:    6582/  10000 (65.82%), best:    6582/10000 (65.82%), error:      9948.73340, best error:      9947.85645 on epoch:    73, epoch:   76/100, mu: 0.9809778333, learning_rate: 0.0000066646, weight_decay: 0.0000085494

epoch time: 1031.2042236328s, input_fired_time: 13.8484563828, output_fired_time: 7.1050448418, propagate_forward_time: 392.4134521484, propagate_backward_time: 607.2547607422, weight_update_time: 0.5771787763, other_time: 10.5825195313
epoch time: 81.6272659302s, input_fired_time: 1.1025692225, output_fired_time: 0.0000000000, propagate_forward_time: 78.7715377808, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7531585693
validation[          , genome  5525] predictions:    6568/  10000 (65.68%), best:    6582/10000 (65.82%), error:      9977.00293, best error:      9947.85645 on epoch:    73, epoch:   77/100, mu: 0.9814378619, learning_rate: 0.0000061703, weight_decay: 0.0000080951

epoch time: 1027.4451904297s, input_fired_time: 13.8058919907, output_fired_time: 7.0379028320, propagate_forward_time: 391.1876525879, propagate_backward_time: 604.8406372070, weight_update_time: 0.5801230073, other_time: 10.5730590820
epoch time: 81.0050048828s, input_fired_time: 1.1143386364, output_fired_time: 0.0000000000, propagate_forward_time: 78.1358489990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7548141479
validation[          , genome  5525] predictions:    6560/  10000 (65.60%), best:    6582/10000 (65.82%), error:      9987.33398, best error:      9947.85645 on epoch:    73, epoch:   78/100, mu: 0.9818744063, learning_rate: 0.0000057126, weight_decay: 0.0000076649

epoch time: 1029.7655029297s, input_fired_time: 13.8066158295, output_fired_time: 7.0836935043, propagate_forward_time: 392.3525390625, propagate_backward_time: 605.9348144531, weight_update_time: 0.5696455836, other_time: 10.5878295898
epoch time: 80.3811950684s, input_fired_time: 1.0989611149, output_fired_time: 0.0000000000, propagate_forward_time: 77.5292968750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7529373169
validation[          , genome  5525] predictions:    6604/  10000 (66.04%), best:    6604/10000 (66.04%), error:      9910.75879, best error:      9910.75879 on epoch:    79, epoch:   79/100, mu: 0.9822887182, learning_rate: 0.0000052889, weight_decay: 0.0000072576

epoch time: 1024.0991210938s, input_fired_time: 13.7372093201, output_fired_time: 6.9352807999, propagate_forward_time: 388.7106323242, propagate_backward_time: 604.2709350586, weight_update_time: 0.5723768473, other_time: 10.4450683594
epoch time: 81.1268844604s, input_fired_time: 1.0951310396, output_fired_time: 0.0000000000, propagate_forward_time: 78.3029327393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7288208008
validation[          , genome  5525] predictions:    6559/  10000 (65.59%), best:    6604/10000 (66.04%), error:     10017.97852, best error:      9910.75879 on epoch:    79, epoch:   80/100, mu: 0.9826818705, learning_rate: 0.0000048966, weight_decay: 0.0000068719

epoch time: 1029.9832763672s, input_fired_time: 13.8373994827, output_fired_time: 7.0000519753, propagate_forward_time: 391.1025390625, propagate_backward_time: 607.5096435547, weight_update_time: 0.5730854273, other_time: 10.5336303711
epoch time: 80.5964965820s, input_fired_time: 1.1063456535, output_fired_time: 0.0000000000, propagate_forward_time: 77.7617416382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7284088135
validation[          , genome  5525] predictions:    6565/  10000 (65.65%), best:    6604/10000 (66.04%), error:      9939.84766, best error:      9910.75879 on epoch:    79, epoch:   81/100, mu: 0.9830549955, learning_rate: 0.0000045334, weight_decay: 0.0000065067

epoch time: 1026.0317382813s, input_fired_time: 13.7718019485, output_fired_time: 6.9817333221, propagate_forward_time: 388.9700317383, propagate_backward_time: 605.8224487305, weight_update_time: 0.5853796005, other_time: 10.4857177734
epoch time: 80.7265014648s, input_fired_time: 1.1074054241, output_fired_time: 0.0000000000, propagate_forward_time: 77.8724975586, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7465972900
validation[          , genome  5525] predictions:    6579/  10000 (65.79%), best:    6604/10000 (66.04%), error:      9977.88574, best error:      9910.75879 on epoch:    79, epoch:   82/100, mu: 0.9834091067, learning_rate: 0.0000041971, weight_decay: 0.0000061609

epoch time: 1024.6928710938s, input_fired_time: 13.7683725357, output_fired_time: 6.9403333664, propagate_forward_time: 389.0115356445, propagate_backward_time: 604.5274047852, weight_update_time: 0.5695985556, other_time: 10.4452514648
epoch time: 80.7152862549s, input_fired_time: 1.1116456985, output_fired_time: 0.0000000000, propagate_forward_time: 77.8631134033, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7405242920
validation[          , genome  5525] predictions:    6569/  10000 (65.69%), best:    6604/10000 (66.04%), error:      9931.73535, best error:      9910.75879 on epoch:    79, epoch:   83/100, mu: 0.9837451577, learning_rate: 0.0000038858, weight_decay: 0.0000058335

epoch time: 1023.9269409180s, input_fired_time: 13.7143373489, output_fired_time: 6.9580740929, propagate_forward_time: 388.2713928223, propagate_backward_time: 604.5415649414, weight_update_time: 0.5703446269, other_time: 10.4415893555
epoch time: 81.3366012573s, input_fired_time: 1.1075559855, output_fired_time: 0.0000000000, propagate_forward_time: 78.4771957397, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7518463135
validation[          , genome  5525] predictions:    6596/  10000 (65.96%), best:    6596/10000 (65.96%), error:      9900.08887, best error:      9900.08887 on epoch:    84, epoch:   84/100, mu: 0.9840640426, learning_rate: 0.0000035976, weight_decay: 0.0000055235

epoch time: 1025.2255859375s, input_fired_time: 13.8014354706, output_fired_time: 6.9368925095, propagate_forward_time: 387.8904724121, propagate_backward_time: 606.1347656250, weight_update_time: 0.5787354112, other_time: 10.4620361328
epoch time: 80.5608596802s, input_fired_time: 1.1158162355, output_fired_time: 0.0000000000, propagate_forward_time: 77.7053222656, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7397232056
validation[          , genome  5525] predictions:    6603/  10000 (66.03%), best:    6603/10000 (66.03%), error:      9897.31152, best error:      9897.31152 on epoch:    85, epoch:   85/100, mu: 0.9843667150, learning_rate: 0.0000033307, weight_decay: 0.0000052300

epoch time: 1023.1286621094s, input_fired_time: 13.7248334885, output_fired_time: 6.9678025246, propagate_forward_time: 387.8026123047, propagate_backward_time: 604.1501464844, weight_update_time: 0.5677073598, other_time: 10.4832763672
epoch time: 80.4323730469s, input_fired_time: 1.1023178101, output_fired_time: 0.0000000000, propagate_forward_time: 77.5957412720, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7343139648
validation[          , genome  5525] predictions:    6581/  10000 (65.81%), best:    6581/10000 (65.81%), error:      9870.36719, best error:      9870.36719 on epoch:    86, epoch:   86/100, mu: 0.9846539497, learning_rate: 0.0000030837, weight_decay: 0.0000049521

epoch time: 1023.1365356445s, input_fired_time: 13.7470664978, output_fired_time: 6.9425868988, propagate_forward_time: 388.2591857910, propagate_backward_time: 603.7171630859, weight_update_time: 0.5687659979, other_time: 10.4705810547
epoch time: 80.6093826294s, input_fired_time: 1.0991045237, output_fired_time: 0.0000000000, propagate_forward_time: 77.7762985229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7339782715
validation[          , genome  5525] predictions:    6537/  10000 (65.37%), best:    6581/10000 (65.81%), error:      9966.41992, best error:      9870.36719 on epoch:    86, epoch:   87/100, mu: 0.9849265218, learning_rate: 0.0000028550, weight_decay: 0.0000046889

epoch time: 1025.9849853516s, input_fired_time: 13.7359914780, output_fired_time: 6.9134383202, propagate_forward_time: 388.5192871094, propagate_backward_time: 606.3814086914, weight_update_time: 0.5679372549, other_time: 10.4348754883
epoch time: 81.0591278076s, input_fired_time: 1.1489262581, output_fired_time: 0.0000000000, propagate_forward_time: 78.1508865356, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7593154907
validation[          , genome  5525] predictions:    6583/  10000 (65.83%), best:    6581/10000 (65.81%), error:      9930.66211, best error:      9870.36719 on epoch:    86, epoch:   88/100, mu: 0.9851852059, learning_rate: 0.0000026432, weight_decay: 0.0000044398

epoch time: 1024.5603027344s, input_fired_time: 13.8188095093, output_fired_time: 6.9784321785, propagate_forward_time: 388.9074707031, propagate_backward_time: 604.2985229492, weight_update_time: 0.5776342750, other_time: 10.5570678711
epoch time: 80.7163238525s, input_fired_time: 1.1075351238, output_fired_time: 0.0000000000, propagate_forward_time: 77.8624420166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7463455200
validation[          , genome  5525] predictions:    6602/  10000 (66.02%), best:    6581/10000 (65.81%), error:      9893.98633, best error:      9870.36719 on epoch:    86, epoch:   89/100, mu: 0.9854307175, learning_rate: 0.0000024471, weight_decay: 0.0000042038

epoch time: 1088.9191894531s, input_fired_time: 14.6799011230, output_fired_time: 7.4310240746, propagate_forward_time: 412.7774353027, propagate_backward_time: 642.6324462891, weight_update_time: 0.6123732924, other_time: 11.3983154297
epoch time: 98.6544799805s, input_fired_time: 1.3937932253, output_fired_time: 0.0000000000, propagate_forward_time: 95.1501541138, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1105346680
validation[          , genome  5525] predictions:    6646/  10000 (66.46%), best:    6581/10000 (65.81%), error:      9871.87109, best error:      9870.36719 on epoch:    86, epoch:   90/100, mu: 0.9856637120, learning_rate: 0.0000022656, weight_decay: 0.0000039804

epoch time: 1043.6257324219s, input_fired_time: 13.9725751877, output_fired_time: 7.0301122665, propagate_forward_time: 395.9476928711, propagate_backward_time: 615.8942871094, weight_update_time: 0.6180878878, other_time: 10.7811279297
epoch time: 80.3914337158s, input_fired_time: 1.1050904989, output_fired_time: 0.0000000000, propagate_forward_time: 77.5348739624, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7514724731
validation[          , genome  5525] predictions:    6582/  10000 (65.82%), best:    6581/10000 (65.81%), error:      9902.82617, best error:      9870.36719 on epoch:    86, epoch:   91/100, mu: 0.9858847857, learning_rate: 0.0000020976, weight_decay: 0.0000037689

epoch time: 1031.9172363281s, input_fired_time: 13.8984956741, output_fired_time: 6.9866509438, propagate_forward_time: 391.9932556152, propagate_backward_time: 608.3856811523, weight_update_time: 0.5741277933, other_time: 10.6531372070
epoch time: 81.1026458740s, input_fired_time: 1.1193747520, output_fired_time: 0.0000000000, propagate_forward_time: 78.2093582153, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7739105225
validation[          , genome  5525] predictions:    6619/  10000 (66.19%), best:    6581/10000 (65.81%), error:      9872.85156, best error:      9870.36719 on epoch:    86, epoch:   92/100, mu: 0.9860945940, learning_rate: 0.0000019420, weight_decay: 0.0000035686

epoch time: 1026.7360839844s, input_fired_time: 13.8124456406, output_fired_time: 6.9459466934, propagate_forward_time: 389.4734497070, propagate_backward_time: 605.9461059570, weight_update_time: 0.5716766119, other_time: 10.5581665039
epoch time: 81.4219741821s, input_fired_time: 1.1220922470, output_fired_time: 0.0000000000, propagate_forward_time: 78.5188446045, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7810363770
validation[          , genome  5525] predictions:    6561/  10000 (65.61%), best:    6581/10000 (65.81%), error:      9984.05762, best error:      9870.36719 on epoch:    86, epoch:   93/100, mu: 0.9862937331, learning_rate: 0.0000017980, weight_decay: 0.0000033790

epoch time: 1025.4708251953s, input_fired_time: 13.7717390060, output_fired_time: 6.8936305046, propagate_forward_time: 388.7201232910, propagate_backward_time: 605.5014038086, weight_update_time: 0.5774827600, other_time: 10.5839233398
epoch time: 80.4158401489s, input_fired_time: 1.0989247561, output_fired_time: 0.0000000000, propagate_forward_time: 77.5655212402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7513961792
validation[          , genome  5525] predictions:    6620/  10000 (66.20%), best:    6581/10000 (65.81%), error:      9872.59375, best error:      9870.36719 on epoch:    86, epoch:   94/100, mu: 0.9864826798, learning_rate: 0.0000016646, weight_decay: 0.0000031994

epoch time: 1026.6835937500s, input_fired_time: 13.8137874603, output_fired_time: 6.8864703178, propagate_forward_time: 389.0808715820, propagate_backward_time: 606.3399047852, weight_update_time: 0.5701080561, other_time: 10.5625610352
epoch time: 80.5776519775s, input_fired_time: 1.1007096767, output_fired_time: 0.0000000000, propagate_forward_time: 77.7168350220, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7601089478
validation[          , genome  5525] predictions:    6614/  10000 (66.14%), best:    6581/10000 (65.81%), error:      9876.74609, best error:      9870.36719 on epoch:    86, epoch:   95/100, mu: 0.9866620302, learning_rate: 0.0000015411, weight_decay: 0.0000030294

epoch time: 1025.5798339844s, input_fired_time: 13.7621717453, output_fired_time: 6.8639082909, propagate_forward_time: 389.3648071289, propagate_backward_time: 605.0337524414, weight_update_time: 0.5753836632, other_time: 10.5552368164
epoch time: 80.4571838379s, input_fired_time: 1.1043534279, output_fired_time: 0.0000000000, propagate_forward_time: 77.5904846191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7623443604
validation[          , genome  5525] predictions:    6578/  10000 (65.78%), best:    6581/10000 (65.81%), error:     10002.10352, best error:      9870.36719 on epoch:    86, epoch:   96/100, mu: 0.9868322015, learning_rate: 0.0000014268, weight_decay: 0.0000028684

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_187_5525_3_r165493131_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508499
read mu: 0.986994
read mu_delta: 0.949014
read initial_learning_rate: 0.00233087
read learning_rate: 1.32098e-06
read learning_rate_delta: 0.925827
read initial_weight_decay: 0.000542312
read weight_decay: 2.71599e-06
read weight_decay_delta: 0.946859
read batch_size: 92
read epsilon: 1e-07
read alpha: 0.0741988
read input_dropout_probability: 0.00114977
read hidden_dropout_probability: 0.092632
read velocity_reset: 1991
read epoch: 97
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 86
read number_validation_images: 10000
read best_validation_predictions: 6581
read best_validation_error: 9870.37
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5525
read normal distribution: '1 0 0'
generator_str: '46867533'
read generator: 46867533
read generated_by_map:
1 enable_edge 1
reading 77 nodes.
reading 326 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1105.13159179688s, input_fired_time: 14.979811668396, output_fired_time: 7.46507406234741, propagate_forward_time: 418.969268798828, propagate_backward_time: 652.326721191406, weight_update_time: 0.600168108940125, other_time: 11.3906860351563
epoch time: 81.5856018066406s, input_fired_time: 1.12210762500763, output_fired_time: 0, propagate_forward_time: 78.6710815429688, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.79241180419922
validation[          , genome  5525] predictions:    6606/  10000 (66.06%), best:    6581/10000 (65.81%), error:      9874.86621, best error:      9870.36719 on epoch:    86, epoch:   97/100, mu: 0.9869937301, learning_rate: 0.0000013210, weight_decay: 0.0000027160

epoch time: 1035.6188964844s, input_fired_time: 13.9538946152, output_fired_time: 7.0832400322, propagate_forward_time: 392.2933959961, propagate_backward_time: 611.6347656250, weight_update_time: 0.5699210763, other_time: 10.6535644531
epoch time: 81.1164474487s, input_fired_time: 1.1297599077, output_fired_time: 0.0000000000, propagate_forward_time: 78.2119064331, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7747802734
validation[          , genome  5525] predictions:    6631/  10000 (66.31%), best:    6631/10000 (66.31%), error:      9857.58887, best error:      9857.58887 on epoch:    98, epoch:   98/100, mu: 0.9871470332, learning_rate: 0.0000012230, weight_decay: 0.0000025717

epoch time: 1029.8887939453s, input_fired_time: 13.8695163727, output_fired_time: 6.9395327568, propagate_forward_time: 390.0950622559, propagate_backward_time: 608.4545288086, weight_update_time: 0.5698858500, other_time: 10.5302124023
epoch time: 81.6908874512s, input_fired_time: 1.1227910519, output_fired_time: 0.0000000000, propagate_forward_time: 78.8080749512, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7600250244
validation[          , genome  5525] predictions:    6629/  10000 (66.29%), best:    6631/10000 (66.31%), error:      9861.56055, best error:      9857.58887 on epoch:    98, epoch:   99/100, mu: 0.9872924685, learning_rate: 0.0000011323, weight_decay: 0.0000024350

epoch time: 1033.1755371094s, input_fired_time: 13.8823356628, output_fired_time: 7.0104060173, propagate_forward_time: 391.3675842285, propagate_backward_time: 610.2991943359, weight_update_time: 0.5723078847, other_time: 10.6160888672
epoch time: 81.1708526611s, input_fired_time: 1.1124565601, output_fired_time: 0.0000000000, propagate_forward_time: 78.3118667603, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7465286255
validation[          , genome  5525] predictions:    6648/  10000 (66.48%), best:    6631/10000 (66.31%), error:      9882.92773, best error:      9857.58887 on epoch:    98, epoch:  100/100, mu: 0.9874305129, learning_rate: 0.0000010483, weight_decay: 0.0000023056

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 407.4948730469s, input_fired_time: 5.6465034485, output_fired_time: 0.0000000000, propagate_forward_time: 392.9003906250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.9479675293
best training[          , genome  5525] predictions:   36301/  50000 (72.60%), best:    6631/10000 (66.31%), error:     41643.97266, best error:      9857.58887 on epoch:    98, epoch:  101/100, mu: 0.9875615239, learning_rate: 0.0000009705, weight_decay: 0.0000021831
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 81.1479415894s, input_fired_time: 1.1076595783, output_fired_time: 0.0000000000, propagate_forward_time: 78.2675476074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7727355957
   testing[          , genome  5525] predictions:    6631/  10000 (66.31%), best:    6631/10000 (66.31%), error:      9857.58887, best error:      9857.58887 on epoch:    98, epoch:  101/100, mu: 0.9875615239, learning_rate: 0.0000009705, weight_decay: 0.0000021831
backpropagation finished successfully!
11:05:05 (8188): called boinc_finish(0)

</stderr_txt>
]]>