Nameexact_genome_1542031203_188_5538_3
Workunit4080259
Created21 Nov 2018, 11:45:13 UTC
Sent21 Nov 2018, 11:45:24 UTC
Report deadline2 Feb 2019, 0:30:53 UTC
Received22 Nov 2018, 11:49:55 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time20 hours 17 min 42 sec
CPU time19 hours 49 min 41 sec
Validate stateValid
Credit18,132.90
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size540.66 MB
Peak swap size560.41 MB
Peak disk usage10.63 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
e: 216.9091033936, propagate_backward_time: 343.1724548340, weight_update_time: 0.2491474450, other_time: 14.1118469238
epoch time: 45.4748497009s, input_fired_time: 0.6935515404, output_fired_time: 0.0000000000, propagate_forward_time: 42.3050117493, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4762878418
validation[          , genome  5538] predictions:    6166/  10000 (61.66%), best:    6166/10000 (61.66%), error:     11025.53320, best error:     11025.53320 on epoch:    35, epoch:   35/100, mu: 0.9055929184, learning_rate: 0.0000553162, weight_decay: 0.0000495531

epoch time: 590.3452758789s, input_fired_time: 11.5409097672, output_fired_time: 3.2474496365, propagate_forward_time: 217.8440856934, propagate_backward_time: 343.6143493652, weight_update_time: 0.2534798980, other_time: 14.0985107422
epoch time: 45.9175796509s, input_fired_time: 0.6890652180, output_fired_time: 0.0000000000, propagate_forward_time: 42.6652488708, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5632667542
validation[          , genome  5538] predictions:    6223/  10000 (62.23%), best:    6223/10000 (62.23%), error:     10968.60938, best error:     10968.60938 on epoch:    36, epoch:   36/100, mu: 0.9097189903, learning_rate: 0.0000513855, weight_decay: 0.0000464659

epoch time: 587.3051757813s, input_fired_time: 11.5428237915, output_fired_time: 3.2424812317, propagate_forward_time: 214.5140838623, propagate_backward_time: 344.0714111328, weight_update_time: 0.2368968427, other_time: 13.9343261719
epoch time: 45.3862228394s, input_fired_time: 0.6868116260, output_fired_time: 0.0000000000, propagate_forward_time: 42.2271919250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4722175598
validation[          , genome  5538] predictions:    6169/  10000 (61.69%), best:    6223/10000 (62.23%), error:     11046.02148, best error:     10968.60938 on epoch:    36, epoch:   37/100, mu: 0.9136433601, learning_rate: 0.0000477341, weight_decay: 0.0000435710

epoch time: 585.1757202148s, input_fired_time: 11.5391988754, output_fired_time: 3.2143325806, propagate_forward_time: 213.2888183594, propagate_backward_time: 343.1923217773, weight_update_time: 0.2362834811, other_time: 13.9410400391
epoch time: 45.9913749695s, input_fired_time: 0.7033637762, output_fired_time: 0.0000000000, propagate_forward_time: 42.7614974976, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5265121460
validation[          , genome  5538] predictions:    6294/  10000 (62.94%), best:    6294/10000 (62.94%), error:     10842.95703, best error:     10842.95703 on epoch:    38, epoch:   38/100, mu: 0.9173759222, learning_rate: 0.0000443421, weight_decay: 0.0000408565

epoch time: 588.8942260742s, input_fired_time: 11.5543708801, output_fired_time: 3.2584202290, propagate_forward_time: 216.0850677490, propagate_backward_time: 343.8077697754, weight_update_time: 0.2509345710, other_time: 14.1885681152
epoch time: 46.3189926147s, input_fired_time: 0.7050039768, output_fired_time: 0.0000000000, propagate_forward_time: 43.0449638367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5690231323
validation[          , genome  5538] predictions:    6279/  10000 (62.79%), best:    6294/10000 (62.94%), error:     10872.76758, best error:     10842.95703 on epoch:    38, epoch:   39/100, mu: 0.9209260345, learning_rate: 0.0000411912, weight_decay: 0.0000383111

epoch time: 583.7416381836s, input_fired_time: 11.5571155548, output_fired_time: 3.2357265949, propagate_forward_time: 212.0049438477, propagate_backward_time: 342.7376708984, weight_update_time: 0.2511801422, other_time: 14.2061767578
epoch time: 45.1275100708s, input_fired_time: 0.6988173723, output_fired_time: 0.0000000000, propagate_forward_time: 41.8930130005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5356788635
validation[          , genome  5538] predictions:    6249/  10000 (62.49%), best:    6294/10000 (62.94%), error:     10923.72656, best error:     10842.95703 on epoch:    38, epoch:   40/100, mu: 0.9243025780, learning_rate: 0.0000382642, weight_decay: 0.0000359242

epoch time: 588.0872802734s, input_fired_time: 11.5619583130, output_fired_time: 3.2676107883, propagate_forward_time: 214.4199676514, propagate_backward_time: 344.6601257324, weight_update_time: 0.2530718148, other_time: 14.1775817871
epoch time: 45.9085922241s, input_fired_time: 0.6955343485, output_fired_time: 0.0000000000, propagate_forward_time: 42.6948890686, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5181694031
validation[          , genome  5538] predictions:    6095/  10000 (60.95%), best:    6294/10000 (62.94%), error:     11250.88477, best error:     10842.95703 on epoch:    38, epoch:   41/100, mu: 0.9275140762, learning_rate: 0.0000355452, weight_decay: 0.0000336861

epoch time: 592.5282592773s, input_fired_time: 11.5818462372, output_fired_time: 3.2516913414, propagate_forward_time: 220.4168853760, propagate_backward_time: 343.1567687988, weight_update_time: 0.2541056871, other_time: 14.1210632324
epoch time: 46.1937141418s, input_fired_time: 0.7085787058, output_fired_time: 0.0000000000, propagate_forward_time: 42.8963394165, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5887947083
validation[          , genome  5538] predictions:    6172/  10000 (61.72%), best:    6294/10000 (62.94%), error:     11064.12988, best error:     10842.95703 on epoch:    38, epoch:   42/100, mu: 0.9305685759, learning_rate: 0.0000330194, weight_decay: 0.0000315874

epoch time: 590.7990722656s, input_fired_time: 11.5942077637, output_fired_time: 3.2765569687, propagate_forward_time: 217.1622924805, propagate_backward_time: 344.5278625488, weight_update_time: 0.2522689998, other_time: 14.2381286621
epoch time: 46.2742309570s, input_fired_time: 0.7039188743, output_fired_time: 0.0000000000, propagate_forward_time: 43.0339584351, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5363540649
validation[          , genome  5538] predictions:    6160/  10000 (61.60%), best:    6294/10000 (62.94%), error:     11031.52832, best error:     10842.95703 on epoch:    38, epoch:   43/100, mu: 0.9334737659, learning_rate: 0.0000306730, weight_decay: 0.0000296195

epoch time: 589.6963500977s, input_fired_time: 11.5918321609, output_fired_time: 3.2864773273, propagate_forward_time: 216.2640380859, propagate_backward_time: 344.4497985840, weight_update_time: 0.2523634732, other_time: 14.1041564941
epoch time: 46.9772300720s, input_fired_time: 0.7107350826, output_fired_time: 0.0000000000, propagate_forward_time: 43.7586860657, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5078086853
validation[          , genome  5538] predictions:    6324/  10000 (63.24%), best:    6324/10000 (63.24%), error:     10766.95801, best error:     10766.95801 on epoch:    44, epoch:   44/100, mu: 0.9362369776, learning_rate: 0.0000284934, weight_decay: 0.0000277742

epoch time: 590.4133300781s, input_fired_time: 11.6028251648, output_fired_time: 3.2696976662, propagate_forward_time: 218.1618347168, propagate_backward_time: 343.1860351563, weight_update_time: 0.2559986413, other_time: 14.1929016113
epoch time: 46.0138816833s, input_fired_time: 0.7036963701, output_fired_time: 0.0000000000, propagate_forward_time: 42.7327995300, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5773849487
validation[          , genome  5538] predictions:    6211/  10000 (62.11%), best:    6324/10000 (63.24%), error:     10846.32031, best error:     10766.95801 on epoch:    44, epoch:   45/100, mu: 0.9388650656, learning_rate: 0.0000264687, weight_decay: 0.0000260438

epoch time: 591.0483398438s, input_fired_time: 11.6071043015, output_fired_time: 3.2902476788, propagate_forward_time: 216.4597167969, propagate_backward_time: 345.5181884766, weight_update_time: 0.2497681081, other_time: 14.1730957031
epoch time: 46.4469757080s, input_fired_time: 0.7108748555, output_fired_time: 0.0000000000, propagate_forward_time: 43.2282485962, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5078506470
validation[          , genome  5538] predictions:    6186/  10000 (61.86%), best:    6324/10000 (63.24%), error:     10968.15430, best error:     10766.95801 on epoch:    44, epoch:   46/100, mu: 0.9413647056, learning_rate: 0.0000245878, weight_decay: 0.0000244212

epoch time: 590.4658813477s, input_fired_time: 11.6214876175, output_fired_time: 3.3013012409, propagate_forward_time: 216.0202636719, propagate_backward_time: 345.3765258789, weight_update_time: 0.2511153519, other_time: 14.1463012695
epoch time: 45.5749282837s, input_fired_time: 0.7088139057, output_fired_time: 0.0000000000, propagate_forward_time: 42.3530578613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5130577087
validation[          , genome  5538] predictions:    6316/  10000 (63.16%), best:    6316/10000 (63.16%), error:     10704.63184, best error:     10704.63184 on epoch:    47, epoch:   47/100, mu: 0.9437421560, learning_rate: 0.0000228407, weight_decay: 0.0000228998

epoch time: 586.7550048828s, input_fired_time: 11.6017875671, output_fired_time: 3.2536456585, propagate_forward_time: 214.2639923096, propagate_backward_time: 343.4275512695, weight_update_time: 0.2520623803, other_time: 14.2080078125
epoch time: 46.3363380432s, input_fired_time: 0.7101292610, output_fired_time: 0.0000000000, propagate_forward_time: 43.0843734741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5418357849
validation[          , genome  5538] predictions:    6266/  10000 (62.66%), best:    6316/10000 (63.16%), error:     10809.83203, best error:     10704.63184 on epoch:    47, epoch:   48/100, mu: 0.9460033774, learning_rate: 0.0000212176, weight_decay: 0.0000214731

epoch time: 588.5635375977s, input_fired_time: 11.6120424271, output_fired_time: 3.2523579597, propagate_forward_time: 216.1515350342, propagate_backward_time: 343.2852783203, weight_update_time: 0.2520825565, other_time: 14.2622680664
epoch time: 47.1932334900s, input_fired_time: 0.7156991363, output_fired_time: 0.0000000000, propagate_forward_time: 43.9371604919, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5403747559
validation[          , genome  5538] predictions:    6250/  10000 (62.50%), best:    6316/10000 (63.16%), error:     10750.88477, best error:     10704.63184 on epoch:    47, epoch:   49/100, mu: 0.9481540918, learning_rate: 0.0000197099, weight_decay: 0.0000201353

epoch time: 584.7086181641s, input_fired_time: 11.5982456207, output_fired_time: 3.2144749165, propagate_forward_time: 212.3889465332, propagate_backward_time: 343.6374206543, weight_update_time: 0.2305824757, other_time: 13.8695068359
epoch time: 44.9234085083s, input_fired_time: 0.6976781487, output_fired_time: 0.0000000000, propagate_forward_time: 41.7240715027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5016593933
validation[          , genome  5538] predictions:    6316/  10000 (63.16%), best:    6316/10000 (63.16%), error:     10700.09180, best error:     10700.09180 on epoch:    50, epoch:   50/100, mu: 0.9501996636, learning_rate: 0.0000183093, weight_decay: 0.0000188808

epoch time: 585.5580444336s, input_fired_time: 11.6020793915, output_fired_time: 3.2195243835, propagate_forward_time: 213.2978820801, propagate_backward_time: 343.6472167969, weight_update_time: 0.2311375886, other_time: 13.7913513184
epoch time: 45.0054626465s, input_fired_time: 0.6968621612, output_fired_time: 0.0000000000, propagate_forward_time: 41.8379516602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4706497192
validation[          , genome  5538] predictions:    6327/  10000 (63.27%), best:    6327/10000 (63.27%), error:     10698.73047, best error:     10698.73047 on epoch:    51, epoch:   51/100, mu: 0.9521452188, learning_rate: 0.0000170083, weight_decay: 0.0000177045

epoch time: 582.1782226563s, input_fired_time: 11.5941495895, output_fired_time: 3.1922686100, propagate_forward_time: 210.9491119385, propagate_backward_time: 342.5823364258, weight_update_time: 0.2297026962, other_time: 13.8603515625
epoch time: 46.0751991272s, input_fired_time: 0.6981275678, output_fired_time: 0.0000000000, propagate_forward_time: 42.8819961548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4950752258
validation[          , genome  5538] predictions:    6283/  10000 (62.83%), best:    6327/10000 (63.27%), error:     10699.58008, best error:     10698.73047 on epoch:    51, epoch:   52/100, mu: 0.9539957047, learning_rate: 0.0000157997, weight_decay: 0.0000166015

epoch time: 587.8209838867s, input_fired_time: 11.6203041077, output_fired_time: 3.2248895168, propagate_forward_time: 213.5384979248, propagate_backward_time: 345.5690612793, weight_update_time: 0.2269568890, other_time: 13.8681945801
epoch time: 47.0376892090s, input_fired_time: 0.7054818273, output_fired_time: 0.0000000000, propagate_forward_time: 43.8982391357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4339675903
validation[          , genome  5538] predictions:    6327/  10000 (63.27%), best:    6327/10000 (63.27%), error:     10623.24219, best error:     10623.24219 on epoch:    53, epoch:   53/100, mu: 0.9557557106, learning_rate: 0.0000146770, weight_decay: 0.0000155672

epoch time: 585.4000854492s, input_fired_time: 11.6095428467, output_fired_time: 3.1968171597, propagate_forward_time: 212.9960021973, propagate_backward_time: 343.8268737793, weight_update_time: 0.2288671434, other_time: 13.7708129883
epoch time: 45.3019981384s, input_fired_time: 0.6986509562, output_fired_time: 0.0000000000, propagate_forward_time: 42.1339530945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4693946838
validation[          , genome  5538] predictions:    6319/  10000 (63.19%), best:    6327/10000 (63.27%), error:     10731.12109, best error:     10623.24219 on epoch:    53, epoch:   54/100, mu: 0.9574297071, learning_rate: 0.0000136340, weight_decay: 0.0000145974

epoch time: 585.7122802734s, input_fired_time: 11.6183166504, output_fired_time: 3.1919145584, propagate_forward_time: 213.6638031006, propagate_backward_time: 343.5222167969, weight_update_time: 0.2280389965, other_time: 13.7160644531
epoch time: 46.4982795715s, input_fired_time: 0.7012751698, output_fired_time: 0.0000000000, propagate_forward_time: 43.2944831848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5025215149
validation[          , genome  5538] predictions:    6366/  10000 (63.66%), best:    6327/10000 (63.27%), error:     10671.19043, best error:     10623.24219 on epoch:    53, epoch:   55/100, mu: 0.9590218663, learning_rate: 0.0000126652, weight_decay: 0.0000136879

epoch time: 587.7156372070s, input_fired_time: 11.6064910889, output_fired_time: 3.1980874538, propagate_forward_time: 216.0417633057, propagate_backward_time: 343.2208251953, weight_update_time: 0.2294270694, other_time: 13.6484985352
epoch time: 46.1437149048s, input_fired_time: 0.7058341503, output_fired_time: 0.0000000000, propagate_forward_time: 42.9842567444, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4536247253
validation[          , genome  5538] predictions:    6303/  10000 (63.03%), best:    6327/10000 (63.27%), error:     10676.26270, best error:     10623.24219 on epoch:    53, epoch:   56/100, mu: 0.9605361819, learning_rate: 0.0000117652, weight_decay: 0.0000128351

epoch time: 583.6885375977s, input_fired_time: 11.6291742325, output_fired_time: 3.1989755630, propagate_forward_time: 211.3363037109, propagate_backward_time: 343.7185974121, weight_update_time: 0.2294418812, other_time: 13.8055114746
epoch time: 45.6329078674s, input_fired_time: 0.7041543722, output_fired_time: 0.0000000000, propagate_forward_time: 42.4761390686, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4526138306
validation[          , genome  5538] predictions:    6360/  10000 (63.60%), best:    6360/10000 (63.60%), error:     10563.84180, best error:     10563.84180 on epoch:    57, epoch:   57/100, mu: 0.9619764686, learning_rate: 0.0000109292, weight_decay: 0.0000120355

epoch time: 584.5749511719s, input_fired_time: 11.6266927719, output_fired_time: 3.1948988438, propagate_forward_time: 211.9472656250, propagate_backward_time: 343.8855895996, weight_update_time: 0.2295338064, other_time: 13.9205017090
epoch time: 45.2288856506s, input_fired_time: 0.7017929554, output_fired_time: 0.0000000000, propagate_forward_time: 42.0350036621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4920883179
validation[          , genome  5538] predictions:    6331/  10000 (63.31%), best:    6360/10000 (63.60%), error:     10614.16699, best error:     10563.84180 on epoch:    57, epoch:   58/100, mu: 0.9633463621, learning_rate: 0.0000101526, weight_decay: 0.0000112857

epoch time: 584.7854614258s, input_fired_time: 11.6325092316, output_fired_time: 3.2331352234, propagate_forward_time: 211.9974060059, propagate_backward_time: 344.0738525391, weight_update_time: 0.2290130407, other_time: 13.8485412598
epoch time: 45.6381530762s, input_fired_time: 0.7000024319, output_fired_time: 0.0000000000, propagate_forward_time: 42.4758949280, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4622573853
validation[          , genome  5538] predictions:    6250/  10000 (62.50%), best:    6360/10000 (63.60%), error:     10672.06250, best error:     10563.84180 on epoch:    57, epoch:   59/100, mu: 0.9646492600, learning_rate: 0.0000094311, weight_decay: 0.0000105826

epoch time: 584.4205322266s, input_fired_time: 11.6299924850, output_fired_time: 3.2028341293, propagate_forward_time: 211.9424743652, propagate_backward_time: 343.8020629883, weight_update_time: 0.2272086889, other_time: 13.8431701660
epoch time: 45.6540985107s, input_fired_time: 0.7141261101, output_fired_time: 0.0000000000, propagate_forward_time: 42.4488182068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4911537170
validation[          , genome  5538] predictions:    6303/  10000 (63.03%), best:    6360/10000 (63.60%), error:     10645.74316, best error:     10563.84180 on epoch:    57, epoch:   60/100, mu: 0.9658885002, learning_rate: 0.0000087610, weight_decay: 0.0000099233

epoch time: 588.2107543945s, input_fired_time: 11.6449537277, output_fired_time: 3.2640926838, propagate_forward_time: 215.3364257813, propagate_backward_time: 343.7794189453, weight_update_time: 0.2497057468, other_time: 14.1858520508
epoch time: 46.1045150757s, input_fired_time: 0.7164840102, output_fired_time: 0.0000000000, propagate_forward_time: 42.8368682861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5511627197
validation[          , genome  5538] predictions:    6319/  10000 (63.19%), best:    6360/10000 (63.60%), error:     10610.45410, best error:     10563.84180 on epoch:    57, epoch:   61/100, mu: 0.9670671225, learning_rate: 0.0000081384, weight_decay: 0.0000093050

epoch time: 590.2589111328s, input_fired_time: 11.6518468857, output_fired_time: 3.2827007771, propagate_forward_time: 217.3453063965, propagate_backward_time: 343.7591247559, weight_update_time: 0.2516892850, other_time: 14.2199096680
epoch time: 45.6185417175s, input_fired_time: 0.7164112329, output_fired_time: 0.0000000000, propagate_forward_time: 42.3492317200, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5528984070
validation[          , genome  5538] predictions:    6288/  10000 (62.88%), best:    6360/10000 (63.60%), error:     10725.62695, best error:     10563.84180 on epoch:    57, epoch:   62/100, mu: 0.9681881666, learning_rate: 0.0000075601, weight_decay: 0.0000087253

epoch time: 588.6716308594s, input_fired_time: 11.6640729904, output_fired_time: 3.2621612549, propagate_forward_time: 216.3892211914, propagate_backward_time: 343.1529235840, weight_update_time: 0.2488449812, other_time: 14.2032775879
epoch time: 45.4274368286s, input_fired_time: 0.7200756669, output_fired_time: 0.0000000000, propagate_forward_time: 42.1466064453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5607528687
validation[          , genome  5538] predictions:    6400/  10000 (64.00%), best:    6400/10000 (64.00%), error:     10511.28516, best error:     10511.28516 on epoch:    63, epoch:   63/100, mu: 0.9692543745, learning_rate: 0.0000070229, weight_decay: 0.0000081817

epoch time: 584.8044433594s, input_fired_time: 11.6459617615, output_fired_time: 3.2742800713, propagate_forward_time: 212.6805267334, propagate_backward_time: 342.8868103027, weight_update_time: 0.2502779067, other_time: 14.3168640137
epoch time: 44.8147087097s, input_fired_time: 0.7110762000, output_fired_time: 0.0000000000, propagate_forward_time: 41.5516662598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5519676208
validation[          , genome  5538] predictions:    6312/  10000 (63.12%), best:    6400/10000 (64.00%), error:     10661.43750, best error:     10511.28516 on epoch:    63, epoch:   64/100, mu: 0.9702684879, learning_rate: 0.0000065239, weight_decay: 0.0000076720

epoch time: 589.0059814453s, input_fired_time: 11.6640405655, output_fired_time: 3.2542102337, propagate_forward_time: 216.3441162109, propagate_backward_time: 343.4964599609, weight_update_time: 0.2520520389, other_time: 14.2471313477
epoch time: 45.3052711487s, input_fired_time: 0.7231415510, output_fired_time: 0.0000000000, propagate_forward_time: 42.0499420166, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5321884155
validation[          , genome  5538] predictions:    6395/  10000 (63.95%), best:    6395/10000 (63.95%), error:     10505.50488, best error:     10505.50488 on epoch:    65, epoch:   65/100, mu: 0.9712330103, learning_rate: 0.0000060603, weight_decay: 0.0000071940

epoch time: 588.5702514648s, input_fired_time: 11.6518907547, output_fired_time: 3.2663242817, propagate_forward_time: 215.6784515381, propagate_backward_time: 343.7129821777, weight_update_time: 0.2532445192, other_time: 14.2605895996
epoch time: 46.1377868652s, input_fired_time: 0.7174092531, output_fired_time: 0.0000000000, propagate_forward_time: 42.8741493225, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5462265015
validation[          , genome  5538] predictions:    6344/  10000 (63.44%), best:    6395/10000 (63.95%), error:     10581.29883, best error:     10505.50488 on epoch:    65, epoch:   66/100, mu: 0.9721503854, learning_rate: 0.0000056296, weight_decay: 0.0000067458

epoch time: 583.5129394531s, input_fired_time: 11.6655740738, output_fired_time: 3.2540326118, propagate_forward_time: 211.5648040771, propagate_backward_time: 342.7754516602, weight_update_time: 0.2507361174, other_time: 14.2530517578
epoch time: 46.5470390320s, input_fired_time: 0.7214309573, output_fired_time: 0.0000000000, propagate_forward_time: 43.2559356689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5696716309
validation[          , genome  5538] predictions:    6402/  10000 (64.02%), best:    6395/10000 (63.95%), error:     10518.10254, best error:     10505.50488 on epoch:    65, epoch:   67/100, mu: 0.9730229378, learning_rate: 0.0000052296, weight_decay: 0.0000063255

epoch time: 587.8870239258s, input_fired_time: 11.6583328247, output_fired_time: 3.2537829876, propagate_forward_time: 214.6554260254, propagate_backward_time: 344.1102600098, weight_update_time: 0.2498663068, other_time: 14.2092285156
epoch time: 46.8671531677s, input_fired_time: 0.7221235037, output_fired_time: 0.0000000000, propagate_forward_time: 43.6014022827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5436286926
validation[          , genome  5538] predictions:    6381/  10000 (63.81%), best:    6395/10000 (63.95%), error:     10537.58008, best error:     10505.50488 on epoch:    65, epoch:   68/100, mu: 0.9738528132, learning_rate: 0.0000048580, weight_decay: 0.0000059314

epoch time: 587.2874755859s, input_fired_time: 11.6666498184, output_fired_time: 3.2637414932, propagate_forward_time: 215.3326263428, propagate_backward_time: 342.9308471680, weight_update_time: 0.2489520758, other_time: 14.0936279297
epoch time: 45.7175712585s, input_fired_time: 0.7223595977, output_fired_time: 0.0000000000, propagate_forward_time: 42.4284934998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5667190552
validation[          , genome  5538] predictions:    6414/  10000 (64.14%), best:    6414/10000 (64.14%), error:     10471.00098, best error:     10471.00098 on epoch:    69, epoch:   69/100, mu: 0.9746421576, learning_rate: 0.0000045128, weight_decay: 0.0000055619

epoch time: 579.2554321289s, input_fired_time: 11.6248397827, output_fired_time: 3.2145881653, propagate_forward_time: 210.4117431641, propagate_backward_time: 339.1037902832, weight_update_time: 0.2463680506, other_time: 14.9004821777
epoch time: 45.6190338135s, input_fired_time: 0.7113417387, output_fired_time: 0.0000000000, propagate_forward_time: 42.2253112793, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6823806763
validation[          , genome  5538] predictions:    6435/  10000 (64.35%), best:    6435/10000 (64.35%), error:     10447.59277, best error:     10447.59277 on epoch:    70, epoch:   70/100, mu: 0.9753928781, learning_rate: 0.0000041921, weight_decay: 0.0000052154

epoch time: 576.3591308594s, input_fired_time: 11.6143341064, output_fired_time: 3.1821439266, propagate_forward_time: 208.8281707764, propagate_backward_time: 337.9000854492, weight_update_time: 0.2430582643, other_time: 14.8344116211
epoch time: 45.4627304077s, input_fired_time: 0.7089232802, output_fired_time: 0.0000000000, propagate_forward_time: 42.0976448059, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6561622620
validation[          , genome  5538] predictions:    6382/  10000 (63.82%), best:    6435/10000 (64.35%), error:     10525.70020, best error:     10447.59277 on epoch:    70, epoch:   71/100, mu: 0.9761069417, learning_rate: 0.0000038942, weight_decay: 0.0000048905

epoch time: 584.5426635742s, input_fired_time: 11.6660366058, output_fired_time: 3.2091319561, propagate_forward_time: 213.4465332031, propagate_backward_time: 341.4330139160, weight_update_time: 0.2349620163, other_time: 14.7879943848
epoch time: 43.6898574829s, input_fired_time: 0.6984603405, output_fired_time: 0.0000000000, propagate_forward_time: 40.2410125732, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7503852844
validation[          , genome  5538] predictions:    6374/  10000 (63.74%), best:    6435/10000 (64.35%), error:     10490.61621, best error:     10447.59277 on epoch:    70, epoch:   72/100, mu: 0.9767860770, learning_rate: 0.0000036175, weight_decay: 0.0000045858

epoch time: 578.0136108398s, input_fired_time: 11.6083602905, output_fired_time: 3.1445980072, propagate_forward_time: 209.6088714600, propagate_backward_time: 339.0841369629, weight_update_time: 0.2247893065, other_time: 14.5676574707
epoch time: 46.1852455139s, input_fired_time: 0.7160029411, output_fired_time: 0.0000000000, propagate_forward_time: 43.0501136780, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4191284180
validation[          , genome  5538] predictions:    6374/  10000 (63.74%), best:    6435/10000 (64.35%), error:     10521.21191, best error:     10447.59277 on epoch:    70, epoch:   73/100, mu: 0.9774320126, learning_rate: 0.0000033604, weight_decay: 0.0000043001

epoch time: 596.0263671875s, input_fired_time: 11.7359590530, output_fired_time: 3.2956249714, propagate_forward_time: 216.6357879639, propagate_backward_time: 351.2389526367, weight_update_time: 0.2339211404, other_time: 13.1199951172
epoch time: 44.5208816528s, input_fired_time: 0.7085307837, output_fired_time: 0.0000000000, propagate_forward_time: 41.5616378784, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2507133484
validation[          , genome  5538] predictions:    6396/  10000 (63.96%), best:    6435/10000 (64.35%), error:     10491.93457, best error:     10447.59277 on epoch:    70, epoch:   74/100, mu: 0.9780463576, learning_rate: 0.0000031217, weight_decay: 0.0000040322

epoch time: 576.4431152344s, input_fired_time: 11.6365747452, output_fired_time: 3.1625893116, propagate_forward_time: 207.7612915039, propagate_backward_time: 340.3377990723, weight_update_time: 0.2252242565, other_time: 13.5448303223
epoch time: 44.3984298706s, input_fired_time: 0.7031155825, output_fired_time: 0.0000000000, propagate_forward_time: 41.2117309570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4835815430
validation[          , genome  5538] predictions:    6432/  10000 (64.32%), best:    6432/10000 (64.32%), error:     10426.21094, best error:     10426.21094 on epoch:    75, epoch:   75/100, mu: 0.9786306620, learning_rate: 0.0000028998, weight_decay: 0.0000037810

epoch time: 574.8057861328s, input_fired_time: 11.6211929321, output_fired_time: 3.1469810009, propagate_forward_time: 207.2350158691, propagate_backward_time: 339.2976379395, weight_update_time: 0.2263127267, other_time: 13.5049438477
epoch time: 46.1377029419s, input_fired_time: 0.7163965106, output_fired_time: 0.0000000000, propagate_forward_time: 43.1318626404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2894439697
validation[          , genome  5538] predictions:    6430/  10000 (64.30%), best:    6432/10000 (64.32%), error:     10447.86914, best error:     10426.21094 on epoch:    75, epoch:   76/100, mu: 0.9791864157, learning_rate: 0.0000026938, weight_decay: 0.0000035454

epoch time: 591.3147583008s, input_fired_time: 11.7324018478, output_fired_time: 3.2530791759, propagate_forward_time: 214.9748382568, propagate_backward_time: 348.5183105469, weight_update_time: 0.2359019965, other_time: 12.8361206055
epoch time: 44.7091064453s, input_fired_time: 0.7077497244, output_fired_time: 0.0000000000, propagate_forward_time: 41.6946220398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3067359924
validation[          , genome  5538] predictions:    6426/  10000 (64.26%), best:    6432/10000 (64.32%), error:     10426.70410, best error:     10426.21094 on epoch:    75, epoch:   77/100, mu: 0.9797149897, learning_rate: 0.0000025024, weight_decay: 0.0000033245

epoch time: 581.8335571289s, input_fired_time: 11.6580629349, output_fired_time: 3.2003262043, propagate_forward_time: 211.0306396484, propagate_backward_time: 342.8478088379, weight_update_time: 0.2290947288, other_time: 13.0967102051
epoch time: 45.5913581848s, input_fired_time: 0.7096895576, output_fired_time: 0.0000000000, propagate_forward_time: 42.4992713928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3823966980
validation[          , genome  5538] predictions:    6409/  10000 (64.09%), best:    6409/10000 (64.09%), error:     10421.29004, best error:     10421.29004 on epoch:    78, epoch:   78/100, mu: 0.9802177548, learning_rate: 0.0000023245, weight_decay: 0.0000031174

epoch time: 579.0276489258s, input_fired_time: 11.6307916641, output_fired_time: 3.1631383896, propagate_forward_time: 210.3564453125, propagate_backward_time: 340.7772216797, weight_update_time: 0.2256102413, other_time: 13.1000366211
epoch time: 44.9110908508s, input_fired_time: 0.7078860998, output_fired_time: 0.0000000000, propagate_forward_time: 41.8841514587, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3190536499
validation[          , genome  5538] predictions:    6438/  10000 (64.38%), best:    6409/10000 (64.09%), error:     10424.59473, best error:     10421.29004 on epoch:    78, epoch:   79/100, mu: 0.9806959629, learning_rate: 0.0000021594, weight_decay: 0.0000029232

epoch time: 580.5341186523s, input_fired_time: 11.6537456512, output_fired_time: 3.1746327877, propagate_forward_time: 210.3005523682, propagate_backward_time: 342.1923828125, weight_update_time: 0.2277262062, other_time: 13.2128295898
epoch time: 45.8166351318s, input_fired_time: 0.7158411741, output_fired_time: 0.0000000000, propagate_forward_time: 42.8485374451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2522583008
validation[          , genome  5538] predictions:    6415/  10000 (64.15%), best:    6409/10000 (64.09%), error:     10423.37500, best error:     10421.29004 on epoch:    78, epoch:   80/100, mu: 0.9811507463, learning_rate: 0.0000020059, weight_decay: 0.0000027411

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538_3_r456341114_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502254
read mu: 0.981583
read mu_delta: 0.951117
read initial_learning_rate: 0.000729916
read learning_rate: 1.86338e-06
read learning_rate_delta: 0.928941
read initial_weight_decay: 0.00047082
read weight_decay: 2.5703e-06
read weight_decay_delta: 0.937699
read batch_size: 89
read epsilon: 1e-07
read alpha: 0.101158
read input_dropout_probability: 0.00120195
read hidden_dropout_probability: 0.0910619
read velocity_reset: 1804
read epoch: 81
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 78
read number_validation_images: 10000
read best_validation_predictions: 6409
read best_validation_error: 10421.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5538
read normal distribution: '0 0.70513117313385 -0.782089412212372'
generator_str: '6489499'
read generator: 6489499
read generated_by_map:
1 split_edge 1
reading 72 nodes.
reading 300 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 588.024658203125s, input_fired_time: 11.7559518814087, output_fired_time: 3.29922008514404, propagate_forward_time: 210.552032470703, propagate_backward_time: 349.316436767578, weight_update_time: 0.236126720905304, other_time: 13.1010131835938
epoch time: 44.2606391906738s, input_fired_time: 0.710902035236359, output_fired_time: 0, propagate_forward_time: 41.1988220214844, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.35091400146484
validation[          , genome  5538] predictions:    6417/  10000 (64.17%), best:    6409/10000 (64.09%), error:     10443.65332, best error:     10421.29004 on epoch:    78, epoch:   81/100, mu: 0.9815832973, learning_rate: 0.0000018634, weight_decay: 0.0000025703

epoch time: 584.4170532227s, input_fired_time: 11.6959476471, output_fired_time: 3.2542173862, propagate_forward_time: 209.7329406738, propagate_backward_time: 346.3379516602, weight_update_time: 0.2317495346, other_time: 13.3960266113
epoch time: 44.5230941772s, input_fired_time: 0.7065142989, output_fired_time: 0.0000000000, propagate_forward_time: 41.3699378967, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4466438293
validation[          , genome  5538] predictions:    6402/  10000 (64.02%), best:    6409/10000 (64.09%), error:     10424.82422, best error:     10421.29004 on epoch:    78, epoch:   82/100, mu: 0.9819947481, learning_rate: 0.0000017310, weight_decay: 0.0000024102

epoch time: 577.5687255859s, input_fired_time: 11.6170139313, output_fired_time: 3.1748509407, propagate_forward_time: 207.5929412842, propagate_backward_time: 341.5544738770, weight_update_time: 0.2251622230, other_time: 13.6294250488
epoch time: 43.6941452026s, input_fired_time: 0.6974873543, output_fired_time: 0.0000000000, propagate_forward_time: 40.5073356628, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4893226624
validation[          , genome  5538] predictions:    6411/  10000 (64.11%), best:    6409/10000 (64.09%), error:     10450.70898, best error:     10421.29004 on epoch:    78, epoch:   83/100, mu: 0.9823860526, learning_rate: 0.0000016080, weight_decay: 0.0000022600

epoch time: 570.3125000000s, input_fired_time: 11.5606899261, output_fired_time: 3.1249902248, propagate_forward_time: 205.3927459717, propagate_backward_time: 336.3096923828, weight_update_time: 0.2202184647, other_time: 13.9243774414
epoch time: 44.1341438293s, input_fired_time: 0.6982824206, output_fired_time: 0.0000000000, propagate_forward_time: 40.9252014160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5106582642
validation[          , genome  5538] predictions:    6404/  10000 (64.04%), best:    6409/10000 (64.09%), error:     10448.65430, best error:     10421.29004 on epoch:    78, epoch:   84/100, mu: 0.9827582240, learning_rate: 0.0000014937, weight_decay: 0.0000021192

epoch time: 566.3280639648s, input_fired_time: 11.5908374786, output_fired_time: 3.1297788620, propagate_forward_time: 201.9016723633, propagate_backward_time: 335.7659912109, weight_update_time: 0.2233276367, other_time: 13.9398193359
epoch time: 44.4530715942s, input_fired_time: 0.6983532310, output_fired_time: 0.0000000000, propagate_forward_time: 41.2328224182, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5218963623
validation[          , genome  5538] predictions:    6436/  10000 (64.36%), best:    6409/10000 (64.09%), error:     10428.19238, best error:     10421.29004 on epoch:    78, epoch:   85/100, mu: 0.9831122160, learning_rate: 0.0000013876, weight_decay: 0.0000019872

epoch time: 572.2436523438s, input_fired_time: 11.5895147324, output_fired_time: 3.1384494305, propagate_forward_time: 207.9252014160, propagate_backward_time: 335.4677124023, weight_update_time: 0.2308599800, other_time: 14.1227722168
epoch time: 43.3714447021s, input_fired_time: 0.7124180198, output_fired_time: 0.0000000000, propagate_forward_time: 39.9812126160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6778144836
validation[          , genome  5538] predictions:    6427/  10000 (64.27%), best:    6409/10000 (64.09%), error:     10453.95703, best error:     10421.29004 on epoch:    78, epoch:   86/100, mu: 0.9834489226, learning_rate: 0.0000012890, weight_decay: 0.0000018634

epoch time: 571.1337890625s, input_fired_time: 11.6040735245, output_fired_time: 3.1744647026, propagate_forward_time: 206.4993591309, propagate_backward_time: 335.4548645020, weight_update_time: 0.2395482659, other_time: 14.4010620117
epoch time: 44.1883010864s, input_fired_time: 0.7123128772, output_fired_time: 0.0000000000, propagate_forward_time: 40.8991966248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5767898560
validation[          , genome  5538] predictions:    6426/  10000 (64.26%), best:    6426/10000 (64.26%), error:     10404.96484, best error:     10404.96484 on epoch:    87, epoch:   87/100, mu: 0.9837691784, learning_rate: 0.0000011974, weight_decay: 0.0000017473

epoch time: 568.4653930664s, input_fired_time: 11.6017141342, output_fired_time: 3.1776635647, propagate_forward_time: 203.7490844727, propagate_backward_time: 335.4949951172, weight_update_time: 0.2424267083, other_time: 14.4419555664
epoch time: 43.9623908997s, input_fired_time: 0.7109467983, output_fired_time: 0.0000000000, propagate_forward_time: 40.5818977356, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6695480347
validation[          , genome  5538] predictions:    6402/  10000 (64.02%), best:    6426/10000 (64.26%), error:     10438.17480, best error:     10404.96484 on epoch:    87, epoch:   88/100, mu: 0.9840737581, learning_rate: 0.0000011123, weight_decay: 0.0000016384

epoch time: 570.8956909180s, input_fired_time: 11.6262617111, output_fired_time: 3.1938698292, propagate_forward_time: 205.7581634521, propagate_backward_time: 335.9531250000, weight_update_time: 0.2413433939, other_time: 14.3642578125
epoch time: 48.0963554382s, input_fired_time: 0.7557419538, output_fired_time: 0.0000000000, propagate_forward_time: 44.8993911743, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4412231445
validation[          , genome  5538] predictions:    6412/  10000 (64.12%), best:    6426/10000 (64.26%), error:     10405.25195, best error:     10404.96484 on epoch:    87, epoch:   89/100, mu: 0.9843634367, learning_rate: 0.0000010332, weight_decay: 0.0000015364

epoch time: 1181.8535156250s, input_fired_time: 16.4666252136, output_fired_time: 7.1946163177, propagate_forward_time: 462.7956237793, propagate_backward_time: 679.0347290039, weight_update_time: 0.5197228789, other_time: 16.3618774414
epoch time: 104.0596847534s, input_fired_time: 1.4009702206, output_fired_time: 0.0000000000, propagate_forward_time: 99.2185134888, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4402008057
validation[          , genome  5538] predictions:    6433/  10000 (64.33%), best:    6426/10000 (64.26%), error:     10418.79785, best error:     10404.96484 on epoch:    87, epoch:   90/100, mu: 0.9846389890, learning_rate: 0.0000009598, weight_decay: 0.0000014406

epoch time: 1158.3022460938s, input_fired_time: 16.4827594757, output_fired_time: 7.1072411537, propagate_forward_time: 454.7363586426, propagate_backward_time: 664.1873779297, weight_update_time: 0.5279343724, other_time: 15.7884521484
epoch time: 102.1798934937s, input_fired_time: 1.4534236193, output_fired_time: 0.0000000000, propagate_forward_time: 97.6633071899, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0631637573
validation[          , genome  5538] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10395.25195, best error:     10395.25195 on epoch:    91, epoch:   91/100, mu: 0.9849010706, learning_rate: 0.0000008916, weight_decay: 0.0000013509

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538_3_r456341114_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502254
read mu: 0.98515
read mu_delta: 0.951117
read initial_learning_rate: 0.000729916
read learning_rate: 8.28263e-07
read learning_rate_delta: 0.928941
read initial_weight_decay: 0.00047082
read weight_decay: 1.26672e-06
read weight_decay_delta: 0.937699
read batch_size: 89
read epsilon: 1e-07
read alpha: 0.101158
read input_dropout_probability: 0.00120195
read hidden_dropout_probability: 0.0910619
read velocity_reset: 1804
read epoch: 92
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 91
read number_validation_images: 10000
read best_validation_predictions: 6437
read best_validation_error: 10395.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5538
read normal distribution: '0 0.70513117313385 -0.782089412212372'
generator_str: '100120175'
read generator: 100120175
read generated_by_map:
1 split_edge 1
reading 72 nodes.
reading 300 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 568.809936523438s, input_fired_time: 11.8554544448853, output_fired_time: 3.17272996902466, propagate_forward_time: 202.446578979492, propagate_backward_time: 337.290802001953, weight_update_time: 0.228099673986435, other_time: 14.0443420410156
epoch time: 43.2663497924805s, input_fired_time: 0.7169548869133, output_fired_time: 0, propagate_forward_time: 40.0148048400879, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.53459167480469
validation[          , genome  5538] predictions:    6463/  10000 (64.63%), best:    6463/10000 (64.63%), error:     10387.53027, best error:     10387.53027 on epoch:    92, epoch:   92/100, mu: 0.9851503372, learning_rate: 0.0000008283, weight_decay: 0.0000012667

epoch time: 568.6234741211s, input_fired_time: 11.8939876556, output_fired_time: 3.1911892891, propagate_forward_time: 202.6124725342, propagate_backward_time: 336.1701965332, weight_update_time: 0.2381942272, other_time: 14.7556457520
epoch time: 43.6707038879s, input_fired_time: 0.7249633074, output_fired_time: 0.0000000000, propagate_forward_time: 40.2989273071, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6468124390
validation[          , genome  5538] predictions:    6437/  10000 (64.37%), best:    6437/10000 (64.37%), error:     10378.06934, best error:     10378.06934 on epoch:    93, epoch:   93/100, mu: 0.9853873849, learning_rate: 0.0000007694, weight_decay: 0.0000011878

epoch time: 958.5347290039s, input_fired_time: 14.4479122162, output_fired_time: 5.6664214134, propagate_forward_time: 370.4874877930, propagate_backward_time: 553.5996704102, weight_update_time: 0.4307158589, other_time: 14.3331909180
epoch time: 90.4993591309s, input_fired_time: 1.2306929827, output_fired_time: 0.0000000000, propagate_forward_time: 86.7097091675, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5589599609
validation[          , genome  5538] predictions:    6450/  10000 (64.50%), best:    6450/10000 (64.50%), error:     10366.10449, best error:     10366.10449 on epoch:    94, epoch:   94/100, mu: 0.9856128693, learning_rate: 0.0000007147, weight_decay: 0.0000011138

epoch time: 1070.6790771484s, input_fired_time: 14.8700113297, output_fired_time: 6.4314665794, propagate_forward_time: 418.3696289063, propagate_backward_time: 617.2304077148, weight_update_time: 0.4662178457, other_time: 13.7775268555
epoch time: 88.6356506348s, input_fired_time: 1.1902906895, output_fired_time: 0.0000000000, propagate_forward_time: 85.0006790161, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4446792603
validation[          , genome  5538] predictions:    6451/  10000 (64.51%), best:    6450/10000 (64.50%), error:     10369.18164, best error:     10366.10449 on epoch:    94, epoch:   95/100, mu: 0.9858273268, learning_rate: 0.0000006639, weight_decay: 0.0000010444

epoch time: 1074.4495849609s, input_fired_time: 14.9832534790, output_fired_time: 6.5034003258, propagate_forward_time: 419.8335876465, propagate_backward_time: 619.0239868164, weight_update_time: 0.4749126136, other_time: 14.1052856445
epoch time: 90.0143432617s, input_fired_time: 1.2097095251, output_fired_time: 0.0000000000, propagate_forward_time: 86.4181671143, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3864669800
validation[          , genome  5538] predictions:    6436/  10000 (64.36%), best:    6450/10000 (64.50%), error:     10379.84082, best error:     10366.10449 on epoch:    94, epoch:   96/100, mu: 0.9860312939, learning_rate: 0.0000006168, weight_decay: 0.0000009793

epoch time: 1120.8696289063s, input_fired_time: 15.6984500885, output_fired_time: 6.8350868225, propagate_forward_time: 439.2614440918, propagate_backward_time: 643.8433837891, weight_update_time: 0.5070523620, other_time: 15.2312011719
epoch time: 94.5610122681s, input_fired_time: 1.2498631477, output_fired_time: 0.0000000000, propagate_forward_time: 90.5863494873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7248001099
validation[          , genome  5538] predictions:    6450/  10000 (64.50%), best:    6450/10000 (64.50%), error:     10377.09375, best error:     10366.10449 on epoch:    94, epoch:   97/100, mu: 0.9862253070, learning_rate: 0.0000005729, weight_decay: 0.0000009183

epoch time: 1158.5139160156s, input_fired_time: 16.3373641968, output_fired_time: 7.1174144745, propagate_forward_time: 453.7887878418, propagate_backward_time: 665.2872314453, weight_update_time: 0.5262597799, other_time: 15.9830322266
epoch time: 95.2556686401s, input_fired_time: 1.2782032490, output_fired_time: 0.0000000000, propagate_forward_time: 91.2845535278, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6929092407
validation[          , genome  5538] predictions:    6454/  10000 (64.54%), best:    6450/10000 (64.50%), error:     10372.94824, best error:     10366.10449 on epoch:    94, epoch:   98/100, mu: 0.9864098430, learning_rate: 0.0000005322, weight_decay: 0.0000008611

epoch time: 1099.9699707031s, input_fired_time: 15.3915052414, output_fired_time: 6.6347985268, propagate_forward_time: 432.1127929688, propagate_backward_time: 631.9225463867, weight_update_time: 0.4919144809, other_time: 13.9083862305
epoch time: 86.3387374878s, input_fired_time: 1.1716451645, output_fired_time: 0.0000000000, propagate_forward_time: 82.9259033203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2411880493
validation[          , genome  5538] predictions:    6460/  10000 (64.60%), best:    6450/10000 (64.50%), error:     10388.65723, best error:     10366.10449 on epoch:    94, epoch:   99/100, mu: 0.9865853190, learning_rate: 0.0000004944, weight_decay: 0.0000008075

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5538_3_r456341114_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502254
read mu: 0.986752
read mu_delta: 0.951117
read initial_learning_rate: 0.000729916
read learning_rate: 4.59275e-07
read learning_rate_delta: 0.928941
read initial_weight_decay: 0.00047082
read weight_decay: 7.57159e-07
read weight_decay_delta: 0.937699
read batch_size: 89
read epsilon: 1e-07
read alpha: 0.101158
read input_dropout_probability: 0.00120195
read hidden_dropout_probability: 0.0910619
read velocity_reset: 1804
read epoch: 100
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 94
read number_validation_images: 10000
read best_validation_predictions: 6450
read best_validation_error: 10366.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5538
read normal distribution: '0 0.70513117313385 -0.782089412212372'
generator_str: '507252458'
read generator: 507252458
read generated_by_map:
1 split_edge 1
reading 72 nodes.
reading 300 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1082.60424804688s, input_fired_time: 15.3941869735718, output_fired_time: 6.31242752075195, propagate_forward_time: 426.144714355469, propagate_backward_time: 621.829406738281, weight_update_time: 0.426892876625061, other_time: 12.923583984375
epoch time: 88.7528686523438s, input_fired_time: 1.16884887218475, output_fired_time: 0, propagate_forward_time: 85.3995056152344, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.18451690673828
validation[          , genome  5538] predictions:    6461/  10000 (64.61%), best:    6450/10000 (64.50%), error:     10374.56152, best error:     10366.10449 on epoch:    94, epoch:  100/100, mu: 0.9867522120, learning_rate: 0.0000004593, weight_decay: 0.0000007572

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 445.0343017578s, input_fired_time: 6.0384039879, output_fired_time: 0.0000000000, propagate_forward_time: 427.9390869141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.0568237305
best training[          , genome  5538] predictions:   34658/  50000 (69.32%), best:    6450/10000 (64.50%), error:     46579.16016, best error:     10366.10449 on epoch:    94, epoch:  101/100, mu: 0.9869109988, learning_rate: 0.0000004266, weight_decay: 0.0000007100
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 87.8200759888s, input_fired_time: 1.1975467205, output_fired_time: 0.0000000000, propagate_forward_time: 84.4214324951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2010955811
   testing[          , genome  5538] predictions:    6450/  10000 (64.50%), best:    6450/10000 (64.50%), error:     10366.10449, best error:     10366.10449 on epoch:    94, epoch:  101/100, mu: 0.9869109988, learning_rate: 0.0000004266, weight_decay: 0.0000007100
backpropagation finished successfully!
14:48:56 (15176): called boinc_finish(0)

</stderr_txt>
]]>