Nameexact_genome_1542727803_186_6422_1
Workunit4101102
Created20 Nov 2018, 18:06:15 UTC
Sent20 Nov 2018, 18:07:47 UTC
Report deadline18 Feb 2019, 12:47:03 UTC
Received28 Nov 2018, 6:10:41 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88000
Run time1 days 20 hours 24 min 50 sec
CPU time1 days 20 hours 23 min 11 sec
Validate stateValid
Credit22,444.32
Device peak FLOPS3.82 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size536.82 MB
Peak swap size535.12 MB
Peak disk usage10.61 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
ed_time: 0.0000000000, propagate_forward_time: 117.4102935791, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4562683105
validation[          , genome  6422] predictions:    5808/  10000 (58.08%), best:    5990/10000 (59.90%), error:     12217.77148, best error:     11925.81348 on epoch:    33, epoch:   35/100, mu: 0.9261611104, learning_rate: 0.0000182366, weight_decay: 0.0000848984

epoch time: 1441.4985351563s, input_fired_time: 16.8380146027, output_fired_time: 8.7643661499, propagate_forward_time: 584.3352661133, propagate_backward_time: 817.9332885742, weight_update_time: 0.5876548886, other_time: 13.6275634766
epoch time: 119.1560211182s, input_fired_time: 1.3768523932, output_fired_time: 0.0000000000, propagate_forward_time: 113.8324050903, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9467620850
validation[          , genome  6422] predictions:    6001/  10000 (60.01%), best:    5990/10000 (59.90%), error:     11928.50098, best error:     11925.81348 on epoch:    33, epoch:   36/100, mu: 0.9296987653, learning_rate: 0.0000170849, weight_decay: 0.0000805753

epoch time: 1446.2524414063s, input_fired_time: 17.0377864838, output_fired_time: 8.8133945465, propagate_forward_time: 584.9091796875, propagate_backward_time: 819.7814941406, weight_update_time: 0.6010632515, other_time: 15.7105712891
epoch time: 122.6249160767s, input_fired_time: 1.4544012547, output_fired_time: 0.0000000000, propagate_forward_time: 118.8981781006, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2723388672
validation[          , genome  6422] predictions:    6078/  10000 (60.78%), best:    6078/10000 (60.78%), error:     11765.16113, best error:     11765.16113 on epoch:    37, epoch:   37/100, mu: 0.9330403805, learning_rate: 0.0000160060, weight_decay: 0.0000764723

epoch time: 1447.7331542969s, input_fired_time: 16.7887592316, output_fired_time: 8.7854404449, propagate_forward_time: 587.0255737305, propagate_backward_time: 821.7827758789, weight_update_time: 0.5827862620, other_time: 13.3505859375
epoch time: 121.3036499023s, input_fired_time: 1.3833581209, output_fired_time: 0.0000000000, propagate_forward_time: 117.6085891724, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3116989136
validation[          , genome  6422] predictions:    6040/  10000 (60.40%), best:    6078/10000 (60.78%), error:     11823.70508, best error:     11765.16113 on epoch:    37, epoch:   38/100, mu: 0.9361968040, learning_rate: 0.0000149952, weight_decay: 0.0000725783

epoch time: 1447.2945556641s, input_fired_time: 16.8701610565, output_fired_time: 8.7879199982, propagate_forward_time: 587.4611816406, propagate_backward_time: 820.7770385742, weight_update_time: 0.5855749249, other_time: 13.3982543945
epoch time: 122.4782867432s, input_fired_time: 1.3942538500, output_fired_time: 0.0000000000, propagate_forward_time: 118.8476486206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2363815308
validation[          , genome  6422] predictions:    5992/  10000 (59.92%), best:    6078/10000 (60.78%), error:     11883.33789, best error:     11765.16113 on epoch:    37, epoch:   39/100, mu: 0.9391782880, learning_rate: 0.0000140483, weight_decay: 0.0000688826

epoch time: 1454.4199218750s, input_fired_time: 16.8593692780, output_fired_time: 8.8161926270, propagate_forward_time: 590.1569213867, propagate_backward_time: 825.2783203125, weight_update_time: 0.5862435699, other_time: 13.3091430664
epoch time: 122.5556030273s, input_fired_time: 1.3862006664, output_fired_time: 0.0000000000, propagate_forward_time: 118.6356124878, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5337905884
validation[          , genome  6422] predictions:    6073/  10000 (60.73%), best:    6078/10000 (60.78%), error:     11787.28223, best error:     11765.16113 on epoch:    37, epoch:   40/100, mu: 0.9419945478, learning_rate: 0.0000131611, weight_decay: 0.0000653750

epoch time: 1443.3057861328s, input_fired_time: 16.7924556732, output_fired_time: 8.7726812363, propagate_forward_time: 584.0629272461, propagate_backward_time: 819.5778808594, weight_update_time: 0.5872169137, other_time: 14.0997924805
epoch time: 121.0202026367s, input_fired_time: 1.4216625690, output_fired_time: 0.0000000000, propagate_forward_time: 116.9591217041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6394195557
validation[          , genome  6422] predictions:    6013/  10000 (60.13%), best:    6078/10000 (60.78%), error:     11839.21191, best error:     11765.16113 on epoch:    37, epoch:   41/100, mu: 0.9446547627, learning_rate: 0.0000123300, weight_decay: 0.0000620461

epoch time: 1446.0400390625s, input_fired_time: 16.9146003723, output_fired_time: 8.7978153229, propagate_forward_time: 584.9815673828, propagate_backward_time: 820.4380493164, weight_update_time: 0.5954506397, other_time: 14.9080200195
epoch time: 118.2446594238s, input_fired_time: 1.3529917002, output_fired_time: 0.0000000000, propagate_forward_time: 113.8472976685, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0443725586
validation[          , genome  6422] predictions:    6087/  10000 (60.87%), best:    6087/10000 (60.87%), error:     11631.89160, best error:     11631.89160 on epoch:    42, epoch:   42/100, mu: 0.9471675754, learning_rate: 0.0000115514, weight_decay: 0.0000588866

epoch time: 1443.6107177734s, input_fired_time: 16.8382091522, output_fired_time: 8.7816104889, propagate_forward_time: 583.9545288086, propagate_backward_time: 819.6612548828, weight_update_time: 0.5891966820, other_time: 14.3750610352
epoch time: 124.3292694092s, input_fired_time: 1.4060547352, output_fired_time: 0.0000000000, propagate_forward_time: 120.5346832275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3885345459
validation[          , genome  6422] predictions:    6004/  10000 (60.04%), best:    6087/10000 (60.87%), error:     11887.03223, best error:     11631.89160 on epoch:    42, epoch:   43/100, mu: 0.9495411515, learning_rate: 0.0000108219, weight_decay: 0.0000558881

epoch time: 1439.8698730469s, input_fired_time: 16.8397216797, output_fired_time: 8.7638301849, propagate_forward_time: 582.5101318359, propagate_backward_time: 817.1827392578, weight_update_time: 0.5893321037, other_time: 14.5734863281
epoch time: 120.4881134033s, input_fired_time: 1.3919080496, output_fired_time: 0.0000000000, propagate_forward_time: 116.2840042114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8122024536
validation[          , genome  6422] predictions:    6040/  10000 (60.40%), best:    6087/10000 (60.87%), error:     11809.70898, best error:     11631.89160 on epoch:    42, epoch:   44/100, mu: 0.9517831802, learning_rate: 0.0000101385, weight_decay: 0.0000530422

epoch time: 1445.4687500000s, input_fired_time: 16.8006172180, output_fired_time: 8.7908248901, propagate_forward_time: 584.1247558594, propagate_backward_time: 821.6311035156, weight_update_time: 0.5904320478, other_time: 14.1214599609
epoch time: 120.9814758301s, input_fired_time: 1.4357533455, output_fired_time: 0.0000000000, propagate_forward_time: 117.2498168945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2959060669
validation[          , genome  6422] predictions:    6058/  10000 (60.58%), best:    6087/10000 (60.87%), error:     11664.90723, best error:     11631.89160 on epoch:    42, epoch:   45/100, mu: 0.9539009929, learning_rate: 0.0000094983, weight_decay: 0.0000503413

epoch time: 1452.3397216797s, input_fired_time: 16.8950138092, output_fired_time: 8.8183326721, propagate_forward_time: 589.0302734375, propagate_backward_time: 823.7460327148, weight_update_time: 0.5905569196, other_time: 13.8500366211
epoch time: 121.8871994019s, input_fired_time: 1.4103848934, output_fired_time: 0.0000000000, propagate_forward_time: 118.2949066162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1819076538
validation[          , genome  6422] predictions:    6035/  10000 (60.35%), best:    6087/10000 (60.87%), error:     11709.96777, best error:     11631.89160 on epoch:    42, epoch:   46/100, mu: 0.9559014440, learning_rate: 0.0000088984, weight_decay: 0.0000477779

epoch time: 1436.4951171875s, input_fired_time: 16.9306755066, output_fired_time: 8.7809896469, propagate_forward_time: 580.4238891602, propagate_backward_time: 815.5045166016, weight_update_time: 0.5956007242, other_time: 14.8550415039
epoch time: 121.5453567505s, input_fired_time: 1.4363274574, output_fired_time: 0.0000000000, propagate_forward_time: 117.8786239624, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2304077148
validation[          , genome  6422] predictions:    6083/  10000 (60.83%), best:    6083/10000 (60.83%), error:     11618.02051, best error:     11618.02051 on epoch:    47, epoch:   47/100, mu: 0.9577910304, learning_rate: 0.0000083365, weight_decay: 0.0000453450

epoch time: 1454.3155517578s, input_fired_time: 16.8611125946, output_fired_time: 8.8138370514, propagate_forward_time: 591.6789550781, propagate_backward_time: 823.8560791016, weight_update_time: 0.5843260884, other_time: 13.1055908203
epoch time: 117.9374618530s, input_fired_time: 1.3639855385, output_fired_time: 0.0000000000, propagate_forward_time: 113.2575836182, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3158950806
validation[          , genome  6422] predictions:    6096/  10000 (60.96%), best:    6083/10000 (60.83%), error:     11678.85156, best error:     11618.02051 on epoch:    47, epoch:   48/100, mu: 0.9595758915, learning_rate: 0.0000078101, weight_decay: 0.0000430360

epoch time: 1445.5910644531s, input_fired_time: 17.0283031464, output_fired_time: 8.8030319214, propagate_forward_time: 585.5010986328, propagate_backward_time: 819.5376586914, weight_update_time: 0.5973212719, other_time: 14.7210083008
epoch time: 121.4517745972s, input_fired_time: 1.4208998680, output_fired_time: 0.0000000000, propagate_forward_time: 116.9077453613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1231307983
validation[          , genome  6422] predictions:    5969/  10000 (59.69%), best:    6083/10000 (60.83%), error:     11857.80762, best error:     11618.02051 on epoch:    47, epoch:   49/100, mu: 0.9612618685, learning_rate: 0.0000073169, weight_decay: 0.0000408446

epoch time: 1447.6220703125s, input_fired_time: 16.8988876343, output_fired_time: 8.8049135208, propagate_forward_time: 586.2205200195, propagate_backward_time: 821.2631225586, weight_update_time: 0.5909901857, other_time: 14.4345703125
epoch time: 122.7347335815s, input_fired_time: 1.4216818810, output_fired_time: 0.0000000000, propagate_forward_time: 118.5555191040, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7575302124
validation[          , genome  6422] predictions:    5993/  10000 (59.93%), best:    6083/10000 (60.83%), error:     11823.58496, best error:     11618.02051 on epoch:    47, epoch:   50/100, mu: 0.9628543854, learning_rate: 0.0000068548, weight_decay: 0.0000387647

epoch time: 1444.6558837891s, input_fired_time: 16.9196910858, output_fired_time: 8.7955274582, propagate_forward_time: 585.0939331055, propagate_backward_time: 819.9333496094, weight_update_time: 0.5925729275, other_time: 13.9133911133
epoch time: 121.1159591675s, input_fired_time: 1.4246499538, output_fired_time: 0.0000000000, propagate_forward_time: 117.4675216675, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2237854004
validation[          , genome  6422] predictions:    6086/  10000 (60.86%), best:    6086/10000 (60.86%), error:     11543.07813, best error:     11543.07813 on epoch:    51, epoch:   51/100, mu: 0.9643586874, learning_rate: 0.0000064219, weight_decay: 0.0000367908

epoch time: 1445.4851074219s, input_fired_time: 17.0436515808, output_fired_time: 8.8306646347, propagate_forward_time: 584.1719970703, propagate_backward_time: 820.0609741211, weight_update_time: 0.6021697521, other_time: 15.3777465820
epoch time: 120.6047286987s, input_fired_time: 1.4225708246, output_fired_time: 0.0000000000, propagate_forward_time: 116.6099472046, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5722122192
validation[          , genome  6422] predictions:    6027/  10000 (60.27%), best:    6086/10000 (60.86%), error:     11705.04590, best error:     11543.07813 on epoch:    51, epoch:   52/100, mu: 0.9657796025, learning_rate: 0.0000060164, weight_decay: 0.0000349174

epoch time: 1446.0002441406s, input_fired_time: 17.0064220428, output_fired_time: 8.9658384323, propagate_forward_time: 584.3848266602, propagate_backward_time: 820.1953735352, weight_update_time: 0.5989527106, other_time: 15.4477539063
epoch time: 119.6172180176s, input_fired_time: 1.3853380680, output_fired_time: 0.0000000000, propagate_forward_time: 115.7957305908, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4361495972
validation[          , genome  6422] predictions:    6138/  10000 (61.38%), best:    6138/10000 (61.38%), error:     11488.02246, best error:     11488.02246 on epoch:    53, epoch:   53/100, mu: 0.9671217799, learning_rate: 0.0000056364, weight_decay: 0.0000331394

epoch time: 1451.3706054688s, input_fired_time: 16.8533897400, output_fired_time: 8.8049850464, propagate_forward_time: 589.2887573242, propagate_backward_time: 823.1786499023, weight_update_time: 0.5848409534, other_time: 13.2448730469
epoch time: 122.7160263062s, input_fired_time: 1.4553968906, output_fired_time: 0.0000000000, propagate_forward_time: 118.8455200195, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4151077271
validation[          , genome  6422] predictions:    5923/  10000 (59.23%), best:    6138/10000 (61.38%), error:     11778.05664, best error:     11488.02246 on epoch:    53, epoch:   54/100, mu: 0.9683895707, learning_rate: 0.0000052805, weight_decay: 0.0000314519

epoch time: 1462.5979003906s, input_fired_time: 16.9041137695, output_fired_time: 8.8565664291, propagate_forward_time: 593.4028320313, propagate_backward_time: 829.6230468750, weight_update_time: 0.5917733908, other_time: 13.8114013672
epoch time: 121.2677841187s, input_fired_time: 1.4071768522, output_fired_time: 0.0000000000, propagate_forward_time: 117.4534225464, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4071884155
validation[          , genome  6422] predictions:    6145/  10000 (61.45%), best:    6138/10000 (61.38%), error:     11507.16699, best error:     11488.02246 on epoch:    53, epoch:   55/100, mu: 0.9695870876, learning_rate: 0.0000049470, weight_decay: 0.0000298503

epoch time: 1465.2416992188s, input_fired_time: 17.0990009308, output_fired_time: 8.8867349625, propagate_forward_time: 595.7521972656, propagate_backward_time: 829.8795166016, weight_update_time: 0.6000463367, other_time: 13.6242675781
epoch time: 119.4207916260s, input_fired_time: 1.4081481695, output_fired_time: 0.0000000000, propagate_forward_time: 115.4036178589, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6090240479
validation[          , genome  6422] predictions:    6099/  10000 (60.99%), best:    6138/10000 (61.38%), error:     11595.86816, best error:     11488.02246 on epoch:    53, epoch:   56/100, mu: 0.9707182646, learning_rate: 0.0000046346, weight_decay: 0.0000283303

epoch time: 1453.1083984375s, input_fired_time: 16.8781833649, output_fired_time: 8.8282136917, propagate_forward_time: 588.9270019531, propagate_backward_time: 824.6181640625, weight_update_time: 0.5873881578, other_time: 13.8568115234
epoch time: 121.7213058472s, input_fired_time: 1.4711890221, output_fired_time: 0.0000000000, propagate_forward_time: 117.9738922119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2762222290
validation[          , genome  6422] predictions:    6111/  10000 (61.11%), best:    6138/10000 (61.38%), error:     11526.97656, best error:     11488.02246 on epoch:    53, epoch:   57/100, mu: 0.9717867374, learning_rate: 0.0000043420, weight_decay: 0.0000268877

epoch time: 1456.3780517578s, input_fired_time: 16.8814811707, output_fired_time: 8.8590021133, propagate_forward_time: 590.2818603516, propagate_backward_time: 826.3347167969, weight_update_time: 0.5950326920, other_time: 14.0209960938
epoch time: 120.6678237915s, input_fired_time: 1.3964204788, output_fired_time: 0.0000000000, propagate_forward_time: 116.5648345947, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7065658569
validation[          , genome  6422] predictions:    6138/  10000 (61.38%), best:    6138/10000 (61.38%), error:     11464.33984, best error:     11464.33984 on epoch:    58, epoch:   58/100, mu: 0.9727960229, learning_rate: 0.0000040678, weight_decay: 0.0000255186

epoch time: 1445.9454345703s, input_fired_time: 16.9985561371, output_fired_time: 8.8176689148, propagate_forward_time: 585.3662719727, propagate_backward_time: 820.4376220703, weight_update_time: 0.5972112417, other_time: 14.3253784180
epoch time: 119.7235488892s, input_fired_time: 1.3985584974, output_fired_time: 0.0000000000, propagate_forward_time: 115.8574142456, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4675750732
validation[          , genome  6422] predictions:    6078/  10000 (60.78%), best:    6138/10000 (61.38%), error:     11646.66406, best error:     11464.33984 on epoch:    58, epoch:   59/100, mu: 0.9737493992, learning_rate: 0.0000038109, weight_decay: 0.0000242192

epoch time: 1453.4473876953s, input_fired_time: 17.0909404755, output_fired_time: 8.8345575333, propagate_forward_time: 589.4248046875, propagate_backward_time: 824.2423706055, weight_update_time: 0.5964282751, other_time: 13.8546752930
epoch time: 120.0414276123s, input_fired_time: 1.3677381277, output_fired_time: 0.0000000000, propagate_forward_time: 116.0253295898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6483612061
validation[          , genome  6422] predictions:    6150/  10000 (61.50%), best:    6150/10000 (61.50%), error:     11459.59082, best error:     11459.59082 on epoch:    60, epoch:   60/100, mu: 0.9746499062, learning_rate: 0.0000035702, weight_decay: 0.0000229859

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422_1_r1261423154_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.520485
read mu: 0.975501
read mu_delta: 0.944585
read initial_learning_rate: 0.000178851
read learning_rate: 3.34476e-06
read learning_rate_delta: 0.93685
read initial_weight_decay: 0.000528825
read weight_decay: 2.18154e-05
read weight_decay_delta: 0.949079
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0783418
read input_dropout_probability: 0.00119955
read hidden_dropout_probability: 0.101751
read velocity_reset: 1904
read epoch: 61
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 60
read number_validation_images: 10000
read best_validation_predictions: 6150
read best_validation_error: 11459.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.357730239629745 0.418509364128113'
generator_str: '1266513867'
read generator: 1266513867
read generated_by_map:
1 merge_node 1
reading 80 nodes.
reading 297 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1477.14807128906s, input_fired_time: 17.0395641326904, output_fired_time: 8.97669887542725, propagate_forward_time: 602.223754882813, propagate_backward_time: 835.621215820313, weight_update_time: 0.596817553043365, other_time: 13.286865234375
epoch time: 123.762329101563s, input_fired_time: 1.43419873714447, output_fired_time: 0, propagate_forward_time: 119.749618530273, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.57851409912109
validation[          , genome  6422] predictions:    6144/  10000 (61.44%), best:    6150/10000 (61.50%), error:     11490.30078, best error:     11459.59082 on epoch:    60, epoch:   61/100, mu: 0.9755005240, learning_rate: 0.0000033448, weight_decay: 0.0000218154

epoch time: 1448.5472412109s, input_fired_time: 16.9833583832, output_fired_time: 8.8264913559, propagate_forward_time: 585.3051147461, propagate_backward_time: 821.9855957031, weight_update_time: 0.6022324562, other_time: 15.4465942383
epoch time: 121.9059524536s, input_fired_time: 1.4243224859, output_fired_time: 0.0000000000, propagate_forward_time: 118.3126983643, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1689300537
validation[          , genome  6422] predictions:    6122/  10000 (61.22%), best:    6122/10000 (61.22%), error:     11442.75879, best error:     11442.75879 on epoch:    62, epoch:   62/100, mu: 0.9763039947, learning_rate: 0.0000031335, weight_decay: 0.0000207046

epoch time: 1455.0064697266s, input_fired_time: 17.0697879791, output_fired_time: 8.8552570343, propagate_forward_time: 589.9797363281, propagate_backward_time: 824.8218994141, weight_update_time: 0.6015748382, other_time: 14.2797851563
epoch time: 123.2482833862s, input_fired_time: 1.4125562906, output_fired_time: 0.0000000000, propagate_forward_time: 119.6504211426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1853027344
validation[          , genome  6422] predictions:    6171/  10000 (61.71%), best:    6122/10000 (61.22%), error:     11451.67090, best error:     11442.75879 on epoch:    62, epoch:   63/100, mu: 0.9770629406, learning_rate: 0.0000029357, weight_decay: 0.0000196503

epoch time: 1462.9201660156s, input_fired_time: 17.1461219788, output_fired_time: 8.8955507278, propagate_forward_time: 592.6854858398, propagate_backward_time: 829.3342895508, weight_update_time: 0.6053132415, other_time: 14.8587646484
epoch time: 122.9148178101s, input_fired_time: 1.4076930285, output_fired_time: 0.0000000000, propagate_forward_time: 119.3239212036, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1832046509
validation[          , genome  6422] predictions:    6153/  10000 (61.53%), best:    6153/10000 (61.53%), error:     11433.47559, best error:     11433.47559 on epoch:    64, epoch:   64/100, mu: 0.9777798653, learning_rate: 0.0000027503, weight_decay: 0.0000186497

epoch time: 1458.8106689453s, input_fired_time: 16.9432792664, output_fired_time: 8.8462190628, propagate_forward_time: 592.1185302734, propagate_backward_time: 827.5318603516, weight_update_time: 0.5882827640, other_time: 13.3708496094
epoch time: 123.0068664551s, input_fired_time: 1.4497843981, output_fired_time: 0.0000000000, propagate_forward_time: 119.0469512939, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5101318359
validation[          , genome  6422] predictions:    6178/  10000 (61.78%), best:    6178/10000 (61.78%), error:     11370.27637, best error:     11370.27637 on epoch:    65, epoch:   65/100, mu: 0.9784570336, learning_rate: 0.0000025766, weight_decay: 0.0000177000

epoch time: 1459.6854248047s, input_fired_time: 17.0838241577, output_fired_time: 8.8721342087, propagate_forward_time: 591.1503295898, propagate_backward_time: 828.1708374023, weight_update_time: 0.6002665162, other_time: 14.4082031250
epoch time: 119.7215347290s, input_fired_time: 1.3988629580, output_fired_time: 0.0000000000, propagate_forward_time: 115.6154022217, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7072677612
validation[          , genome  6422] predictions:    6172/  10000 (61.72%), best:    6178/10000 (61.78%), error:     11379.59961, best error:     11370.27637 on epoch:    65, epoch:   66/100, mu: 0.9790967107, learning_rate: 0.0000024139, weight_decay: 0.0000167987

epoch time: 1468.4580078125s, input_fired_time: 16.9599742889, output_fired_time: 8.9230251312, propagate_forward_time: 595.8757934570, propagate_backward_time: 832.4856567383, weight_update_time: 0.5999493003, other_time: 14.2136230469
epoch time: 124.3330383301s, input_fired_time: 1.4181529284, output_fired_time: 0.0000000000, propagate_forward_time: 120.7090301514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2058563232
validation[          , genome  6422] predictions:    6205/  10000 (62.05%), best:    6178/10000 (61.78%), error:     11401.42871, best error:     11370.27637 on epoch:    65, epoch:   67/100, mu: 0.9797009230, learning_rate: 0.0000022614, weight_decay: 0.0000159433

epoch time: 1453.9111328125s, input_fired_time: 16.9357986450, output_fired_time: 8.8628168106, propagate_forward_time: 588.5200195313, propagate_backward_time: 825.2700195313, weight_update_time: 0.5941199660, other_time: 14.3225097656
epoch time: 124.0248184204s, input_fired_time: 1.4660081863, output_fired_time: 0.0000000000, propagate_forward_time: 120.1475982666, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4112091064
validation[          , genome  6422] predictions:    6124/  10000 (61.24%), best:    6178/10000 (61.78%), error:     11474.89551, best error:     11370.27637 on epoch:    65, epoch:   68/100, mu: 0.9802716374, learning_rate: 0.0000021186, weight_decay: 0.0000151315

epoch time: 1436.7415771484s, input_fired_time: 16.9619960785, output_fired_time: 8.7948026657, propagate_forward_time: 579.3242187500, propagate_backward_time: 816.2451171875, weight_update_time: 0.5983510017, other_time: 15.4154052734
epoch time: 121.4398880005s, input_fired_time: 1.4487984180, output_fired_time: 0.0000000000, propagate_forward_time: 117.4974365234, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4936523438
validation[          , genome  6422] predictions:    6209/  10000 (62.09%), best:    6209/10000 (62.09%), error:     11345.22656, best error:     11345.22656 on epoch:    69, epoch:   69/100, mu: 0.9808107615, learning_rate: 0.0000019848, weight_decay: 0.0000143610

epoch time: 1446.0604248047s, input_fired_time: 17.0624923706, output_fired_time: 8.8318691254, propagate_forward_time: 582.7839355469, propagate_backward_time: 821.6614379883, weight_update_time: 0.6044633985, other_time: 15.7206420898
epoch time: 119.2630996704s, input_fired_time: 1.3905845881, output_fired_time: 0.0000000000, propagate_forward_time: 115.1711502075, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7013626099
validation[          , genome  6422] predictions:    6158/  10000 (61.58%), best:    6209/10000 (62.09%), error:     11439.41016, best error:     11345.22656 on epoch:    69, epoch:   70/100, mu: 0.9813199639, learning_rate: 0.0000018595, weight_decay: 0.0000136297

epoch time: 1437.4813232422s, input_fired_time: 16.9032783508, output_fired_time: 8.7864227295, propagate_forward_time: 580.1569824219, propagate_backward_time: 816.4099731445, weight_update_time: 0.5964832902, other_time: 15.2246704102
epoch time: 119.2517852783s, input_fired_time: 1.4104237556, output_fired_time: 0.0000000000, propagate_forward_time: 114.5842361450, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2571258545
validation[          , genome  6422] predictions:    6181/  10000 (61.81%), best:    6209/10000 (62.09%), error:     11435.34082, best error:     11345.22656 on epoch:    69, epoch:   71/100, mu: 0.9818009734, learning_rate: 0.0000017421, weight_decay: 0.0000129357

epoch time: 1440.9870605469s, input_fired_time: 16.9830703735, output_fired_time: 8.7985534668, propagate_forward_time: 582.7448730469, propagate_backward_time: 817.9006347656, weight_update_time: 0.5933392644, other_time: 14.5599365234
epoch time: 118.2391662598s, input_fired_time: 1.3894668818, output_fired_time: 0.0000000000, propagate_forward_time: 113.3981170654, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4515838623
validation[          , genome  6422] predictions:    6182/  10000 (61.82%), best:    6209/10000 (62.09%), error:     11352.04102, best error:     11345.22656 on epoch:    69, epoch:   72/100, mu: 0.9822553396, learning_rate: 0.0000016321, weight_decay: 0.0000122770

epoch time: 1434.2207031250s, input_fired_time: 16.9699478149, output_fired_time: 8.7885694504, propagate_forward_time: 577.8390502930, propagate_backward_time: 814.6240234375, weight_update_time: 0.5997653008, other_time: 15.9990844727
epoch time: 119.5039825439s, input_fired_time: 1.4081008434, output_fired_time: 0.0000000000, propagate_forward_time: 114.6048202515, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4910583496
validation[          , genome  6422] predictions:    6197/  10000 (61.97%), best:    6209/10000 (62.09%), error:     11469.36426, best error:     11345.22656 on epoch:    69, epoch:   73/100, mu: 0.9826844931, learning_rate: 0.0000015290, weight_decay: 0.0000116518

epoch time: 1446.8576660156s, input_fired_time: 16.9967288971, output_fired_time: 8.8251237869, propagate_forward_time: 585.3220214844, propagate_backward_time: 821.1137084961, weight_update_time: 0.5968638062, other_time: 14.6001586914
epoch time: 124.1988449097s, input_fired_time: 1.4435304403, output_fired_time: 0.0000000000, propagate_forward_time: 120.3928756714, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3624420166
validation[          , genome  6422] predictions:    6170/  10000 (61.70%), best:    6209/10000 (62.09%), error:     11414.42480, best error:     11345.22656 on epoch:    69, epoch:   74/100, mu: 0.9830898643, learning_rate: 0.0000014324, weight_decay: 0.0000110585

epoch time: 1449.9868164063s, input_fired_time: 17.0090732574, output_fired_time: 8.8203592300, propagate_forward_time: 587.3529052734, propagate_backward_time: 822.3330688477, weight_update_time: 0.5930936337, other_time: 14.4714965820
epoch time: 124.6495056152s, input_fired_time: 1.4233032465, output_fired_time: 0.0000000000, propagate_forward_time: 120.9560546875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2701492310
validation[          , genome  6422] predictions:    6116/  10000 (61.16%), best:    6209/10000 (62.09%), error:     11437.89844, best error:     11345.22656 on epoch:    69, epoch:   75/100, mu: 0.9834727645, learning_rate: 0.0000013420, weight_decay: 0.0000104954

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422_1_r1261423154_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.520485
read mu: 0.983834
read mu_delta: 0.944585
read initial_learning_rate: 0.000178851
read learning_rate: 1.25723e-06
read learning_rate_delta: 0.93685
read initial_weight_decay: 0.000528825
read weight_decay: 9.96096e-06
read weight_decay_delta: 0.949079
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0783418
read input_dropout_probability: 0.00119955
read hidden_dropout_probability: 0.101751
read velocity_reset: 1904
read epoch: 76
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 69
read number_validation_images: 10000
read best_validation_predictions: 6209
read best_validation_error: 11345.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.357730239629745 0.418509364128113'
generator_str: '2033073102'
read generator: 2033073102
read generated_by_map:
1 merge_node 1
reading 80 nodes.
reading 297 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1477.11962890625s, input_fired_time: 17.317253112793, output_fired_time: 9.00146770477295, propagate_forward_time: 599.897338867188, propagate_backward_time: 836.073852539063, weight_update_time: 0.615759968757629, other_time: 14.8297119140625
epoch time: 123.988082885742s, input_fired_time: 1.43032383918762, output_fired_time: 0, propagate_forward_time: 120.320732116699, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.23703002929688
validation[          , genome  6422] predictions:    6164/  10000 (61.64%), best:    6209/10000 (62.09%), error:     11401.98730, best error:     11345.22656 on epoch:    69, epoch:   76/100, mu: 0.9838344455, learning_rate: 0.0000012572, weight_decay: 0.0000099610

epoch time: 1486.5507812500s, input_fired_time: 17.2388000488, output_fired_time: 9.0464439392, propagate_forward_time: 607.0255737305, propagate_backward_time: 839.6654052734, weight_update_time: 0.6076644659, other_time: 13.5746459961
epoch time: 125.1765670776s, input_fired_time: 1.4385795593, output_fired_time: 0.0000000000, propagate_forward_time: 121.4421844482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2958068848
validation[          , genome  6422] predictions:    6229/  10000 (62.29%), best:    6209/10000 (62.09%), error:     11365.17383, best error:     11345.22656 on epoch:    69, epoch:   77/100, mu: 0.9841760993, learning_rate: 0.0000011778, weight_decay: 0.0000094537

epoch time: 1473.0085449219s, input_fired_time: 17.2343025208, output_fired_time: 8.9672851563, propagate_forward_time: 599.6473999023, propagate_backward_time: 833.2823486328, weight_update_time: 0.6089758873, other_time: 13.8772583008
epoch time: 121.2249908447s, input_fired_time: 1.4018584490, output_fired_time: 0.0000000000, propagate_forward_time: 117.3230514526, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5000839233
validation[          , genome  6422] predictions:    6202/  10000 (62.02%), best:    6209/10000 (62.09%), error:     11385.95703, best error:     11345.22656 on epoch:    69, epoch:   78/100, mu: 0.9844988585, learning_rate: 0.0000011035, weight_decay: 0.0000089724

epoch time: 1458.2407226563s, input_fired_time: 16.9326400757, output_fired_time: 8.8550386429, propagate_forward_time: 592.5022583008, propagate_backward_time: 826.4512329102, weight_update_time: 0.5902548432, other_time: 13.4996337891
epoch time: 124.6071853638s, input_fired_time: 1.4437812567, output_fired_time: 0.0000000000, propagate_forward_time: 120.8541870117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3092193604
validation[          , genome  6422] predictions:    6148/  10000 (61.48%), best:    6209/10000 (62.09%), error:     11410.66992, best error:     11345.22656 on epoch:    69, epoch:   79/100, mu: 0.9848036766, learning_rate: 0.0000010338, weight_decay: 0.0000085155

epoch time: 1451.2675781250s, input_fired_time: 17.1004543304, output_fired_time: 8.8744773865, propagate_forward_time: 587.5555419922, propagate_backward_time: 822.6316528320, weight_update_time: 0.6067953110, other_time: 15.1054077148
epoch time: 125.2862854004s, input_fired_time: 1.4488778114, output_fired_time: 0.0000000000, propagate_forward_time: 121.5307617188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3066482544
validation[          , genome  6422] predictions:    6227/  10000 (62.27%), best:    6209/10000 (62.09%), error:     11360.20508, best error:     11345.22656 on epoch:    69, epoch:   80/100, mu: 0.9850916266, learning_rate: 0.0000009685, weight_decay: 0.0000080819

epoch time: 1456.5515136719s, input_fired_time: 17.1569366455, output_fired_time: 8.8586454391, propagate_forward_time: 590.7619628906, propagate_backward_time: 825.4755859375, weight_update_time: 0.6012570262, other_time: 14.2983398438
epoch time: 122.5778045654s, input_fired_time: 1.4302889109, output_fired_time: 0.0000000000, propagate_forward_time: 118.0995635986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0479507446
validation[          , genome  6422] predictions:    6211/  10000 (62.11%), best:    6209/10000 (62.09%), error:     11407.91016, best error:     11345.22656 on epoch:    69, epoch:   81/100, mu: 0.9853636026, learning_rate: 0.0000009073, weight_decay: 0.0000076703

epoch time: 1480.9368896484s, input_fired_time: 17.1863136292, output_fired_time: 8.9957103729, propagate_forward_time: 602.7506103516, propagate_backward_time: 838.1385498047, weight_update_time: 0.6089327335, other_time: 13.8657226563
epoch time: 124.7491989136s, input_fired_time: 1.4223775864, output_fired_time: 0.0000000000, propagate_forward_time: 121.1149139404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2119064331
validation[          , genome  6422] predictions:    6227/  10000 (62.27%), best:    6227/10000 (62.27%), error:     11327.13184, best error:     11327.13184 on epoch:    82, epoch:   82/100, mu: 0.9856205583, learning_rate: 0.0000008500, weight_decay: 0.0000072797

epoch time: 1456.7139892578s, input_fired_time: 17.0409049988, output_fired_time: 8.8845157623, propagate_forward_time: 590.6173706055, propagate_backward_time: 825.2729492188, weight_update_time: 0.6052030325, other_time: 14.8982543945
epoch time: 121.4164047241s, input_fired_time: 1.4217957258, output_fired_time: 0.0000000000, propagate_forward_time: 116.9601745605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0344314575
validation[          , genome  6422] predictions:    6251/  10000 (62.51%), best:    6227/10000 (62.27%), error:     11335.49512, best error:     11327.13184 on epoch:    82, epoch:   83/100, mu: 0.9858632684, learning_rate: 0.0000007964, weight_decay: 0.0000069091

epoch time: 1464.1071777344s, input_fired_time: 17.1447086334, output_fired_time: 8.9017982483, propagate_forward_time: 593.5783691406, propagate_backward_time: 829.6063232422, weight_update_time: 0.6043853760, other_time: 14.8759765625
epoch time: 122.1019973755s, input_fired_time: 1.4899241924, output_fired_time: 0.0000000000, propagate_forward_time: 118.3112945557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3007812500
validation[          , genome  6422] predictions:    6211/  10000 (62.11%), best:    6227/10000 (62.27%), error:     11381.00781, best error:     11327.13184 on epoch:    82, epoch:   84/100, mu: 0.9860925078, learning_rate: 0.0000007461, weight_decay: 0.0000065572

epoch time: 1475.2539062500s, input_fired_time: 17.1032848358, output_fired_time: 8.9378204346, propagate_forward_time: 601.1179199219, propagate_backward_time: 834.2150878906, weight_update_time: 0.5988855362, other_time: 13.8797607422
epoch time: 123.3948135376s, input_fired_time: 1.4554523230, output_fired_time: 0.0000000000, propagate_forward_time: 119.6739196777, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2654418945
validation[          , genome  6422] predictions:    6232/  10000 (62.32%), best:    6227/10000 (62.27%), error:     11327.91699, best error:     11327.13184 on epoch:    82, epoch:   85/100, mu: 0.9863090515, learning_rate: 0.0000006990, weight_decay: 0.0000062233

epoch time: 1474.5511474609s, input_fired_time: 17.0897235870, output_fired_time: 8.9516296387, propagate_forward_time: 598.9993286133, propagate_backward_time: 835.4125976563, weight_update_time: 0.6046943665, other_time: 14.0978393555
epoch time: 120.9197463989s, input_fired_time: 1.4347852468, output_fired_time: 0.0000000000, propagate_forward_time: 117.3190765381, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1658859253
validation[          , genome  6422] predictions:    6183/  10000 (61.83%), best:    6227/10000 (62.27%), error:     11397.38184, best error:     11327.13184 on epoch:    82, epoch:   86/100, mu: 0.9865136147, learning_rate: 0.0000006548, weight_decay: 0.0000059064

epoch time: 1470.2885742188s, input_fired_time: 17.1078796387, output_fired_time: 8.9424877167, propagate_forward_time: 597.6778564453, propagate_backward_time: 832.3677978516, weight_update_time: 0.6051694155, other_time: 14.1925048828
epoch time: 121.4945678711s, input_fired_time: 1.4382631779, output_fired_time: 0.0000000000, propagate_forward_time: 117.8876419067, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1686630249
validation[          , genome  6422] predictions:    6174/  10000 (61.74%), best:    6227/10000 (62.27%), error:     11412.76367, best error:     11327.13184 on epoch:    82, epoch:   87/100, mu: 0.9867067933, learning_rate: 0.0000006135, weight_decay: 0.0000056057

epoch time: 1459.3172607422s, input_fired_time: 17.0374317169, output_fired_time: 8.8845272064, propagate_forward_time: 592.1115722656, propagate_backward_time: 826.8031005859, weight_update_time: 0.6003433466, other_time: 14.4805908203
epoch time: 120.8865356445s, input_fired_time: 1.3931233883, output_fired_time: 0.0000000000, propagate_forward_time: 116.8279418945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6654739380
validation[          , genome  6422] predictions:    6194/  10000 (61.94%), best:    6227/10000 (62.27%), error:     11339.74609, best error:     11327.13184 on epoch:    82, epoch:   88/100, mu: 0.9868893027, learning_rate: 0.0000005747, weight_decay: 0.0000053202

epoch time: 1464.6453857422s, input_fired_time: 17.1138496399, output_fired_time: 8.9029140472, propagate_forward_time: 593.7536010742, propagate_backward_time: 830.2761840820, weight_update_time: 0.6026840210, other_time: 14.5987548828
epoch time: 121.9860458374s, input_fired_time: 1.4162378311, output_fired_time: 0.0000000000, propagate_forward_time: 117.5423507690, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0274581909
validation[          , genome  6422] predictions:    6223/  10000 (62.23%), best:    6227/10000 (62.27%), error:     11372.61426, best error:     11327.13184 on epoch:    82, epoch:   89/100, mu: 0.9870616794, learning_rate: 0.0000005384, weight_decay: 0.0000050493

epoch time: 1444.6839599609s, input_fired_time: 17.0099983215, output_fired_time: 8.8291254044, propagate_forward_time: 583.1013793945, propagate_backward_time: 819.9771118164, weight_update_time: 0.6023901105, other_time: 15.7663574219
epoch time: 117.4250564575s, input_fired_time: 1.3762966394, output_fired_time: 0.0000000000, propagate_forward_time: 113.2220001221, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8267593384
validation[          , genome  6422] predictions:    6207/  10000 (62.07%), best:    6207/10000 (62.07%), error:     11301.32422, best error:     11301.32422 on epoch:    90, epoch:   90/100, mu: 0.9872245193, learning_rate: 0.0000005044, weight_decay: 0.0000047922

epoch time: 1445.8438720703s, input_fired_time: 17.0447216034, output_fired_time: 8.8296718597, propagate_forward_time: 584.3624877930, propagate_backward_time: 820.4803466797, weight_update_time: 0.6007556319, other_time: 15.1266479492
epoch time: 123.3586578369s, input_fired_time: 1.4426540136, output_fired_time: 0.0000000000, propagate_forward_time: 119.5910797119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3249206543
validation[          , genome  6422] predictions:    6211/  10000 (62.11%), best:    6207/10000 (62.07%), error:     11358.07715, best error:     11301.32422 on epoch:    90, epoch:   91/100, mu: 0.9873782992, learning_rate: 0.0000004726, weight_decay: 0.0000045482

epoch time: 1466.2451171875s, input_fired_time: 17.1455001831, output_fired_time: 8.9362649918, propagate_forward_time: 594.3229980469, propagate_backward_time: 831.3083496094, weight_update_time: 0.6052561998, other_time: 14.5319824219
epoch time: 117.1055221558s, input_fired_time: 1.4039505720, output_fired_time: 0.0000000000, propagate_forward_time: 111.9533233643, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7482452393
validation[          , genome  6422] predictions:    6242/  10000 (62.42%), best:    6207/10000 (62.07%), error:     11345.07422, best error:     11301.32422 on epoch:    90, epoch:   92/100, mu: 0.9875235558, learning_rate: 0.0000004427, weight_decay: 0.0000043166

epoch time: 1447.5565185547s, input_fired_time: 17.0471725464, output_fired_time: 8.8561515808, propagate_forward_time: 584.9838867188, propagate_backward_time: 821.7274780273, weight_update_time: 0.6008683443, other_time: 14.9418334961
epoch time: 116.8860549927s, input_fired_time: 1.4007322788, output_fired_time: 0.0000000000, propagate_forward_time: 112.2391815186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2461395264
validation[          , genome  6422] predictions:    6201/  10000 (62.01%), best:    6207/10000 (62.07%), error:     11319.74023, best error:     11301.32422 on epoch:    90, epoch:   93/100, mu: 0.9876607656, learning_rate: 0.0000004148, weight_decay: 0.0000040968

epoch time: 1447.7569580078s, input_fired_time: 17.1556205750, output_fired_time: 8.8738222122, propagate_forward_time: 585.0596923828, propagate_backward_time: 820.2389526367, weight_update_time: 0.6129324436, other_time: 16.4288940430
epoch time: 117.6650009155s, input_fired_time: 1.3839524984, output_fired_time: 0.0000000000, propagate_forward_time: 113.3177490234, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9633026123
validation[          , genome  6422] predictions:    6211/  10000 (62.11%), best:    6207/10000 (62.07%), error:     11320.65625, best error:     11301.32422 on epoch:    90, epoch:   94/100, mu: 0.9877904058, learning_rate: 0.0000003886, weight_decay: 0.0000038882

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_186_6422_1_r1261423154_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.520485
read mu: 0.987913
read mu_delta: 0.944585
read initial_learning_rate: 0.000178851
read learning_rate: 3.64039e-07
read learning_rate_delta: 0.93685
read initial_weight_decay: 0.000528825
read weight_decay: 3.69019e-06
read weight_decay_delta: 0.949079
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0783418
read input_dropout_probability: 0.00119955
read hidden_dropout_probability: 0.101751
read velocity_reset: 1904
read epoch: 95
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 90
read number_validation_images: 10000
read best_validation_predictions: 6207
read best_validation_error: 11301.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6422
read normal distribution: '0 -0.357730239629745 0.418509364128113'
generator_str: '178489289'
read generator: 178489289
read generated_by_map:
1 merge_node 1
reading 80 nodes.
reading 297 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1468.69372558594s, input_fired_time: 17.2110443115234, output_fired_time: 8.97046375274658, propagate_forward_time: 595.976379394531, propagate_backward_time: 831.07080078125, weight_update_time: 0.611549019813538, other_time: 15.4650268554688
epoch time: 122.951713562012s, input_fired_time: 1.4261155128479, output_fired_time: 0, propagate_forward_time: 119.171043395996, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.35455322265625
validation[          , genome  6422] predictions:    6206/  10000 (62.06%), best:    6207/10000 (62.07%), error:     11349.10254, best error:     11301.32422 on epoch:    90, epoch:   95/100, mu: 0.9879128337, learning_rate: 0.0000003640, weight_decay: 0.0000036902

epoch time: 1485.0133056641s, input_fired_time: 17.3948230743, output_fired_time: 9.0802097321, propagate_forward_time: 605.5385742188, propagate_backward_time: 838.4747314453, weight_update_time: 0.6226902604, other_time: 14.5250244141
epoch time: 125.4302597046s, input_fired_time: 1.4594956636, output_fired_time: 0.0000000000, propagate_forward_time: 121.6393127441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3314514160
validation[          , genome  6422] predictions:    6248/  10000 (62.48%), best:    6248/10000 (62.48%), error:     11295.69727, best error:     11295.69727 on epoch:    96, epoch:   96/100, mu: 0.9880284667, learning_rate: 0.0000003410, weight_decay: 0.0000035023

epoch time: 1485.9167480469s, input_fired_time: 17.3370018005, output_fired_time: 9.0685214996, propagate_forward_time: 606.4165649414, propagate_backward_time: 838.9428710938, weight_update_time: 0.6302130818, other_time: 14.1517944336
epoch time: 127.7563095093s, input_fired_time: 1.4317984581, output_fired_time: 0.0000000000, propagate_forward_time: 124.0842590332, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2402496338
validation[          , genome  6422] predictions:    6216/  10000 (62.16%), best:    6248/10000 (62.48%), error:     11329.26660, best error:     11295.69727 on epoch:    96, epoch:   97/100, mu: 0.9881377220, learning_rate: 0.0000003195, weight_decay: 0.0000033239

epoch time: 1469.3402099609s, input_fired_time: 17.1308116913, output_fired_time: 8.9633665085, propagate_forward_time: 595.8332519531, propagate_backward_time: 832.5328369141, weight_update_time: 0.6049413681, other_time: 14.8798828125
epoch time: 124.9851074219s, input_fired_time: 1.4438561201, output_fired_time: 0.0000000000, propagate_forward_time: 121.1649627686, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3762893677
validation[          , genome  6422] predictions:    6199/  10000 (61.99%), best:    6248/10000 (62.48%), error:     11390.72168, best error:     11295.69727 on epoch:    96, epoch:   98/100, mu: 0.9882408977, learning_rate: 0.0000002993, weight_decay: 0.0000031547

epoch time: 1465.0307617188s, input_fired_time: 17.2294902802, output_fired_time: 8.9529418945, propagate_forward_time: 592.3892211914, propagate_backward_time: 831.1273193359, weight_update_time: 0.6123934984, other_time: 15.3318481445
epoch time: 123.2316284180s, input_fired_time: 1.4225649834, output_fired_time: 0.0000000000, propagate_forward_time: 119.2256393433, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5834274292
validation[          , genome  6422] predictions:    6241/  10000 (62.41%), best:    6248/10000 (62.48%), error:     11316.48047, best error:     11295.69727 on epoch:    96, epoch:   99/100, mu: 0.9883383512, learning_rate: 0.0000002804, weight_decay: 0.0000029940

epoch time: 1442.0081787109s, input_fired_time: 17.0191955566, output_fired_time: 8.8350210190, propagate_forward_time: 581.1580810547, propagate_backward_time: 819.4609375000, weight_update_time: 0.6042642593, other_time: 15.5350341797
epoch time: 122.4604644775s, input_fired_time: 1.4273899794, output_fired_time: 0.0000000000, propagate_forward_time: 118.6244430542, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4086303711
validation[          , genome  6422] predictions:    6210/  10000 (62.10%), best:    6248/10000 (62.48%), error:     11313.05566, best error:     11295.69727 on epoch:    96, epoch:  100/100, mu: 0.9884304404, learning_rate: 0.0000002627, weight_decay: 0.0000028416

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 602.1924438477s, input_fired_time: 7.0783953667, output_fired_time: 0.0000000000, propagate_forward_time: 582.4876708984, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.6264038086
best training[          , genome  6422] predictions:   32454/  50000 (64.91%), best:    6248/10000 (62.48%), error:     54066.58984, best error:     11295.69727 on epoch:    96, epoch:  101/100, mu: 0.9885174036, learning_rate: 0.0000002461, weight_decay: 0.0000026969
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 119.7163314819s, input_fired_time: 1.4316794872, output_fired_time: 0.0000000000, propagate_forward_time: 115.3009872437, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9836654663
   testing[          , genome  6422] predictions:    6248/  10000 (62.48%), best:    6248/10000 (62.48%), error:     11295.69727, best error:     11295.69727 on epoch:    96, epoch:  101/100, mu: 0.9885174036, learning_rate: 0.0000002461, weight_decay: 0.0000026969
backpropagation finished successfully!
06:09:17 (8832): called boinc_finish(0)

</stderr_txt>
]]>