Nameexact_genome_1542031203_188_5600_0
Workunit4081785
Created12 Nov 2018, 19:31:04 UTC
Sent12 Nov 2018, 19:33:57 UTC
Report deadline5 Feb 2019, 3:01:55 UTC
Received21 Nov 2018, 23:14:36 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID23692
Run time1 days 13 hours 15 min 22 sec
CPU time1 days 10 hours 25 min 25 sec
Validate stateValid
Credit21,077.77
Device peak FLOPS4.15 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size545.09 MB
Peak swap size564.77 MB
Peak disk usage11.00 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
86.0038452148, weight_update_time: 0.6619737148, other_time: 12.3516845703
epoch time: 91.6717453003s, input_fired_time: 1.1249704361, output_fired_time: 0.0000000000, propagate_forward_time: 88.4843139648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0624618530
validation[          , genome  5600] predictions:    4995/  10000 (49.95%), best:    6340/10000 (63.40%), error:     15624.64844, best error:     11116.94336 on epoch:     0, epoch:   42/100, mu: 0.9454082847, learning_rate: 0.0000006364, weight_decay: 0.0000575649

epoch time: 1146.7009277344s, input_fired_time: 14.4221105576, output_fired_time: 7.5856781006, propagate_forward_time: 436.0800170898, propagate_backward_time: 676.3741455078, weight_update_time: 0.6519242525, other_time: 12.2389526367
epoch time: 90.9479904175s, input_fired_time: 1.1179486513, output_fired_time: 0.0000000000, propagate_forward_time: 87.7395629883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0904769897
validation[          , genome  5600] predictions:    5028/  10000 (50.28%), best:    6340/10000 (63.40%), error:     15581.65625, best error:     11116.94336 on epoch:     0, epoch:   43/100, mu: 0.9478644133, learning_rate: 0.0000005960, weight_decay: 0.0000546588

epoch time: 1144.7546386719s, input_fired_time: 14.3851776123, output_fired_time: 7.6717572212, propagate_forward_time: 435.8975524902, propagate_backward_time: 674.5400390625, weight_update_time: 0.6506876945, other_time: 12.2601318359
epoch time: 90.6790924072s, input_fired_time: 1.0943709612, output_fired_time: 0.0000000000, propagate_forward_time: 87.5238418579, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0608825684
validation[          , genome  5600] predictions:    4893/  10000 (48.93%), best:    6340/10000 (63.40%), error:     15629.83691, best error:     11116.94336 on epoch:     0, epoch:   44/100, mu: 0.9501852989, learning_rate: 0.0000005582, weight_decay: 0.0000518994

epoch time: 1154.4099121094s, input_fired_time: 14.5196876526, output_fired_time: 7.6815648079, propagate_forward_time: 438.7839050293, propagate_backward_time: 681.1367797852, weight_update_time: 0.6660169363, other_time: 12.2880249023
epoch time: 92.2622985840s, input_fired_time: 1.1241174936, output_fired_time: 0.0000000000, propagate_forward_time: 89.0483779907, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0898056030
validation[          , genome  5600] predictions:    5007/  10000 (50.07%), best:    6340/10000 (63.40%), error:     15518.69336, best error:     11116.94336 on epoch:     0, epoch:   45/100, mu: 0.9523783326, learning_rate: 0.0000005227, weight_decay: 0.0000492794

epoch time: 1165.7723388672s, input_fired_time: 14.6472673416, output_fired_time: 7.7254920006, propagate_forward_time: 443.1820983887, propagate_backward_time: 687.8298339844, weight_update_time: 0.6568773389, other_time: 12.3876953125
epoch time: 90.7584228516s, input_fired_time: 1.1158084869, output_fired_time: 0.0000000000, propagate_forward_time: 87.5782699585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0643463135
validation[          , genome  5600] predictions:    4959/  10000 (49.59%), best:    6340/10000 (63.40%), error:     15593.17773, best error:     11116.94336 on epoch:     0, epoch:   46/100, mu: 0.9544505477, learning_rate: 0.0000004896, weight_decay: 0.0000467916

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600_0_r1785420667_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508406
read mu: 0.956409
read mu_delta: 0.944919
read initial_learning_rate: 1e-05
read learning_rate: 4.58477e-07
read learning_rate_delta: 0.936521
read initial_weight_decay: 0.000507052
read weight_decay: 4.44294e-05
read weight_decay_delta: 0.949516
read batch_size: 86
read epsilon: 1e-07
read alpha: 0.0882511
read input_dropout_probability: 0.00115475
read hidden_dropout_probability: 0.0934083
read velocity_reset: 1731
read epoch: 47
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6340
read best_validation_error: 11116.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5600
read normal distribution: '0 1.38065969944 -0.697693645954132'
generator_str: '1959278949'
read generator: 1959278949
read generated_by_map:
1 change_size_x 1
reading 77 nodes.
reading 311 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1593.69091796875s, input_fired_time: 21.5247039794922, output_fired_time: 10.9821443557739, propagate_forward_time: 610.883239746094, propagate_backward_time: 931.357666015625, weight_update_time: 0.924464464187622, other_time: 18.9431762695313
epoch time: 122.46800994873s, input_fired_time: 1.58721601963043, output_fired_time: 0, propagate_forward_time: 117.579315185547, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.30147552490234
validation[          , genome  5600] predictions:    5039/  10000 (50.39%), best:    6340/10000 (63.40%), error:     15542.73242, best error:     11116.94336 on epoch:     0, epoch:   47/100, mu: 0.9564086199, learning_rate: 0.0000004585, weight_decay: 0.0000444294

epoch time: 1752.8542480469s, input_fired_time: 23.6776485443, output_fired_time: 12.2347173691, propagate_forward_time: 670.8019409180, propagate_backward_time: 1025.7236328125, weight_update_time: 1.0120421648, other_time: 20.4162597656
epoch time: 132.2900848389s, input_fired_time: 1.6439650059, output_fired_time: 0.0000000000, propagate_forward_time: 127.4790344238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1670837402
validation[          , genome  5600] predictions:    4971/  10000 (49.71%), best:    6340/10000 (63.40%), error:     15524.58691, best error:     11116.94336 on epoch:     0, epoch:   48/100, mu: 0.9582588673, learning_rate: 0.0000004294, weight_decay: 0.0000421864

epoch time: 1714.1661376953s, input_fired_time: 22.4981479645, output_fired_time: 11.6229753494, propagate_forward_time: 655.5784301758, propagate_backward_time: 1004.0952758789, weight_update_time: 0.9911407828, other_time: 20.3712768555
epoch time: 130.5424346924s, input_fired_time: 1.5414477587, output_fired_time: 0.0000000000, propagate_forward_time: 125.5748367310, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4261550903
validation[          , genome  5600] predictions:    5018/  10000 (50.18%), best:    6340/10000 (63.40%), error:     15495.30078, best error:     11116.94336 on epoch:     0, epoch:   49/100, mu: 0.9600071907, learning_rate: 0.0000004021, weight_decay: 0.0000400567

epoch time: 1682.0021972656s, input_fired_time: 21.7981204987, output_fired_time: 11.5288200378, propagate_forward_time: 641.6422119141, propagate_backward_time: 986.7172851563, weight_update_time: 0.9483162165, other_time: 20.3157958984
epoch time: 114.3678741455s, input_fired_time: 1.5266804695, output_fired_time: 0.0000000000, propagate_forward_time: 109.8195419312, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0216522217
validation[          , genome  5600] predictions:    4999/  10000 (49.99%), best:    6340/10000 (63.40%), error:     15465.94824, best error:     11116.94336 on epoch:     0, epoch:   50/100, mu: 0.9616592526, learning_rate: 0.0000003766, weight_decay: 0.0000380345

epoch time: 1395.0118408203s, input_fired_time: 18.0692272186, output_fired_time: 9.4830331802, propagate_forward_time: 534.5579223633, propagate_backward_time: 816.4730834961, weight_update_time: 0.8228029609, other_time: 16.4285888672
epoch time: 112.3899688721s, input_fired_time: 1.3900309801, output_fired_time: 0.0000000000, propagate_forward_time: 108.3875656128, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6123733521
validation[          , genome  5600] predictions:    5006/  10000 (50.06%), best:    6340/10000 (63.40%), error:     15447.24121, best error:     11116.94336 on epoch:     0, epoch:   51/100, mu: 0.9632202983, learning_rate: 0.0000003527, weight_decay: 0.0000361144

epoch time: 1377.1781005859s, input_fired_time: 17.8701992035, output_fired_time: 9.5924034119, propagate_forward_time: 525.9932861328, propagate_backward_time: 807.7090454102, weight_update_time: 0.8480952978, other_time: 16.0131225586
epoch time: 109.7592926025s, input_fired_time: 1.3242076635, output_fired_time: 0.0000000000, propagate_forward_time: 105.6206817627, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8143997192
validation[          , genome  5600] predictions:    5039/  10000 (50.39%), best:    6340/10000 (63.40%), error:     15493.73535, best error:     11116.94336 on epoch:     0, epoch:   52/100, mu: 0.9646953344, learning_rate: 0.0000003303, weight_decay: 0.0000342912

epoch time: 1384.6173095703s, input_fired_time: 17.8706569672, output_fired_time: 9.6081342697, propagate_forward_time: 529.0155029297, propagate_backward_time: 812.1777343750, weight_update_time: 0.8203875422, other_time: 15.9453125000
epoch time: 113.4487533569s, input_fired_time: 1.3683099747, output_fired_time: 0.0000000000, propagate_forward_time: 109.3857727051, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6946716309
validation[          , genome  5600] predictions:    5022/  10000 (50.22%), best:    6340/10000 (63.40%), error:     15461.43945, best error:     11116.94336 on epoch:     0, epoch:   53/100, mu: 0.9660891294, learning_rate: 0.0000003093, weight_decay: 0.0000325600

epoch time: 28903.2031250000s, input_fired_time: 14.7783603668, output_fired_time: 7.7900171280, propagate_forward_time: 28209.9511718750, propagate_backward_time: 657.2906494141, weight_update_time: 0.6670863628, other_time: 13.3929443359
epoch time: 89.2718505859s, input_fired_time: 1.1139377356, output_fired_time: 0.0000000000, propagate_forward_time: 85.9135131836, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2444000244
validation[          , genome  5600] predictions:    4993/  10000 (49.93%), best:    6340/10000 (63.40%), error:     15444.54199, best error:     11116.94336 on epoch:     0, epoch:   54/100, mu: 0.9674061537, learning_rate: 0.0000002897, weight_decay: 0.0000309163

epoch time: 1121.5150146484s, input_fired_time: 14.8186254501, output_fired_time: 7.8308892250, propagate_forward_time: 429.5390319824, propagate_backward_time: 655.6640625000, weight_update_time: 0.6747875214, other_time: 13.6623535156
epoch time: 89.4980392456s, input_fired_time: 1.1216063499, output_fired_time: 0.0000000000, propagate_forward_time: 86.0374145508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3390197754
validation[          , genome  5600] predictions:    5018/  10000 (50.18%), best:    6340/10000 (63.40%), error:     15473.44336, best error:     11116.94336 on epoch:     0, epoch:   55/100, mu: 0.9686506391, learning_rate: 0.0000002713, weight_decay: 0.0000293555

epoch time: 1125.7789306641s, input_fired_time: 14.8586387634, output_fired_time: 7.9227385521, propagate_forward_time: 430.7178955078, propagate_backward_time: 658.6589355469, weight_update_time: 0.6783859730, other_time: 13.6207275391
epoch time: 93.8427581787s, input_fired_time: 1.1529575586, output_fired_time: 0.0000000000, propagate_forward_time: 90.4623870850, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2274169922
validation[          , genome  5600] predictions:    5077/  10000 (50.77%), best:    6340/10000 (63.40%), error:     15470.49316, best error:     11116.94336 on epoch:     0, epoch:   56/100, mu: 0.9698265791, learning_rate: 0.0000002541, weight_decay: 0.0000278736

epoch time: 1123.9418945313s, input_fired_time: 14.8440713882, output_fired_time: 7.8815388680, propagate_forward_time: 430.1293334961, propagate_backward_time: 657.5371704102, weight_update_time: 0.6804704666, other_time: 13.5496826172
epoch time: 88.8823776245s, input_fired_time: 1.1044539213, output_fired_time: 0.0000000000, propagate_forward_time: 85.4778137207, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3001098633
validation[          , genome  5600] predictions:    5027/  10000 (50.27%), best:    6340/10000 (63.40%), error:     15409.29492, best error:     11116.94336 on epoch:     0, epoch:   57/100, mu: 0.9709377289, learning_rate: 0.0000002380, weight_decay: 0.0000264664

epoch time: 1115.1546630859s, input_fired_time: 14.6713991165, output_fired_time: 7.7703876495, propagate_forward_time: 426.4198608398, propagate_backward_time: 652.9163208008, weight_update_time: 0.6684657931, other_time: 13.3767089844
epoch time: 88.7648239136s, input_fired_time: 1.0975437164, output_fired_time: 0.0000000000, propagate_forward_time: 85.5157699585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1515121460
validation[          , genome  5600] predictions:    5039/  10000 (50.39%), best:    6340/10000 (63.40%), error:     15418.08301, best error:     11116.94336 on epoch:     0, epoch:   58/100, mu: 0.9719877243, learning_rate: 0.0000002228, weight_decay: 0.0000251303

epoch time: 1113.9105224609s, input_fired_time: 14.6368074417, output_fired_time: 7.7266230583, propagate_forward_time: 426.8071594238, propagate_backward_time: 651.3985595703, weight_update_time: 0.6767406464, other_time: 13.3414306641
epoch time: 88.4548873901s, input_fired_time: 1.1116868258, output_fired_time: 0.0000000000, propagate_forward_time: 85.0905075073, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2526931763
validation[          , genome  5600] predictions:    5073/  10000 (50.73%), best:    6340/10000 (63.40%), error:     15428.53809, best error:     11116.94336 on epoch:     0, epoch:   59/100, mu: 0.9729798436, learning_rate: 0.0000002087, weight_decay: 0.0000238616

epoch time: 1114.6972656250s, input_fired_time: 14.6118383408, output_fired_time: 7.6920232773, propagate_forward_time: 426.3509826660, propagate_backward_time: 652.8895263672, weight_update_time: 0.6641889215, other_time: 13.1529541016
epoch time: 88.8110580444s, input_fired_time: 1.1208319664, output_fired_time: 0.0000000000, propagate_forward_time: 85.3388595581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3513641357
validation[          , genome  5600] predictions:    5090/  10000 (50.90%), best:    6340/10000 (63.40%), error:     15374.41895, best error:     11116.94336 on epoch:     0, epoch:   60/100, mu: 0.9739173055, learning_rate: 0.0000001955, weight_decay: 0.0000226570

epoch time: 1112.7424316406s, input_fired_time: 14.6411743164, output_fired_time: 7.7305216789, propagate_forward_time: 425.8970947266, propagate_backward_time: 651.2193603516, weight_update_time: 0.6650957465, other_time: 13.2543945313
epoch time: 88.2173004150s, input_fired_time: 1.0891826153, output_fired_time: 0.0000000000, propagate_forward_time: 84.8366546631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2914657593
validation[          , genome  5600] predictions:    5031/  10000 (50.31%), best:    6340/10000 (63.40%), error:     15450.94824, best error:     11116.94336 on epoch:     0, epoch:   61/100, mu: 0.9748031497, learning_rate: 0.0000001830, weight_decay: 0.0000215132

epoch time: 1121.5284423828s, input_fired_time: 14.6608381271, output_fired_time: 7.8299422264, propagate_forward_time: 427.9817810059, propagate_backward_time: 657.7567749023, weight_update_time: 0.6708619595, other_time: 13.2990112305
epoch time: 89.6359481812s, input_fired_time: 1.1142584085, output_fired_time: 0.0000000000, propagate_forward_time: 86.2953109741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2263793945
validation[          , genome  5600] predictions:    4993/  10000 (49.93%), best:    6340/10000 (63.40%), error:     15439.31836, best error:     11116.94336 on epoch:     0, epoch:   62/100, mu: 0.9756401777, learning_rate: 0.0000001714, weight_decay: 0.0000204271

epoch time: 1113.3128662109s, input_fired_time: 14.6082935333, output_fired_time: 7.7978501320, propagate_forward_time: 425.9352416992, propagate_backward_time: 651.6150512695, weight_update_time: 0.6658902168, other_time: 13.3564453125
epoch time: 88.8650360107s, input_fired_time: 1.0974596739, output_fired_time: 0.0000000000, propagate_forward_time: 85.4609451294, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3066329956
validation[          , genome  5600] predictions:    5066/  10000 (50.66%), best:    6340/10000 (63.40%), error:     15487.87988, best error:     11116.94336 on epoch:     0, epoch:   63/100, mu: 0.9764311314, learning_rate: 0.0000001605, weight_decay: 0.0000193959

epoch time: 1116.1018066406s, input_fired_time: 14.6486978531, output_fired_time: 7.7721390724, propagate_forward_time: 427.5276184082, propagate_backward_time: 653.0424194336, weight_update_time: 0.6654384732, other_time: 13.1110229492
epoch time: 88.9827957153s, input_fired_time: 1.1095196009, output_fired_time: 0.0000000000, propagate_forward_time: 85.5182952881, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3549804688
validation[          , genome  5600] predictions:    5034/  10000 (50.34%), best:    6340/10000 (63.40%), error:     15397.07813, best error:     11116.94336 on epoch:     0, epoch:   64/100, mu: 0.9771785140, learning_rate: 0.0000001504, weight_decay: 0.0000184167

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600_0_r1785420667_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508406
read mu: 0.977885
read mu_delta: 0.944919
read initial_learning_rate: 1e-05
read learning_rate: 1.40809e-07
read learning_rate_delta: 0.936521
read initial_weight_decay: 0.000507052
read weight_decay: 1.7487e-05
read weight_decay_delta: 0.949516
read batch_size: 86
read epsilon: 1e-07
read alpha: 0.0882511
read input_dropout_probability: 0.00115475
read hidden_dropout_probability: 0.0934083
read velocity_reset: 1731
read epoch: 65
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6340
read best_validation_error: 11116.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5600
read normal distribution: '0 1.38065969944 -0.697693645954132'
generator_str: '1476275847'
read generator: 1476275847
read generated_by_map:
1 change_size_x 1
reading 77 nodes.
reading 311 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1139.37902832031s, input_fired_time: 14.1117486953735, output_fired_time: 7.34913206100464, propagate_forward_time: 433.274932861328, propagate_backward_time: 673.138610839844, weight_update_time: 0.629276514053345, other_time: 11.5046997070313
epoch time: 90.221923828125s, input_fired_time: 1.0591596364975, output_fired_time: 0, propagate_forward_time: 87.2662124633789, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.89655303955078
validation[          , genome  5600] predictions:    5094/  10000 (50.94%), best:    6340/10000 (63.40%), error:     15398.48926, best error:     11116.94336 on epoch:     0, epoch:   65/100, mu: 0.9778847098, learning_rate: 0.0000001408, weight_decay: 0.0000174870

epoch time: 1139.6920166016s, input_fired_time: 14.1328430176, output_fired_time: 7.2998485565, propagate_forward_time: 432.9642333984, propagate_backward_time: 673.8446655273, weight_update_time: 0.6331409812, other_time: 11.4505004883
epoch time: 90.2146759033s, input_fired_time: 1.0548244715, output_fired_time: 0.0000000000, propagate_forward_time: 87.2684097290, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8914413452
validation[          , genome  5600] predictions:    5060/  10000 (50.60%), best:    6340/10000 (63.40%), error:     15388.89844, best error:     11116.94336 on epoch:     0, epoch:   66/100, mu: 0.9785520434, learning_rate: 0.0000001319, weight_decay: 0.0000166042

epoch time: 1140.1577148438s, input_fired_time: 14.1683416367, output_fired_time: 7.2856836319, propagate_forward_time: 432.9153442383, propagate_backward_time: 674.3264160156, weight_update_time: 0.6247872710, other_time: 11.4619750977
epoch time: 90.2790145874s, input_fired_time: 1.0618232489, output_fired_time: 0.0000000000, propagate_forward_time: 87.3129119873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9042816162
validation[          , genome  5600] predictions:    5061/  10000 (50.61%), best:    6340/10000 (63.40%), error:     15431.52930, best error:     11116.94336 on epoch:     0, epoch:   67/100, mu: 0.9791826010, learning_rate: 0.0000001235, weight_decay: 0.0000157660

epoch time: 37787.3007812500s, input_fired_time: 14.2388362885, output_fired_time: 7.4569201469, propagate_forward_time: 37081.8750000000, propagate_backward_time: 670.5057373047, weight_update_time: 0.6408155560, other_time: 13.2247314453
epoch time: 89.7788238525s, input_fired_time: 1.0918279886, output_fired_time: 0.0000000000, propagate_forward_time: 86.1301422119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5568542480
validation[          , genome  5600] predictions:    5094/  10000 (50.94%), best:    6340/10000 (63.40%), error:     15378.15625, best error:     11116.94336 on epoch:     0, epoch:   68/100, mu: 0.9797784090, learning_rate: 0.0000001157, weight_decay: 0.0000149700

epoch time: 1118.9877929688s, input_fired_time: 14.5133590698, output_fired_time: 7.7745137215, propagate_forward_time: 425.9556274414, propagate_backward_time: 657.4511718750, weight_update_time: 0.6578050852, other_time: 13.2931518555
epoch time: 90.4002151489s, input_fired_time: 1.1398214102, output_fired_time: 0.0000000000, propagate_forward_time: 86.9722137451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2881774902
validation[          , genome  5600] predictions:    5098/  10000 (50.98%), best:    6340/10000 (63.40%), error:     15413.98633, best error:     11116.94336 on epoch:     0, epoch:   69/100, mu: 0.9803414345, learning_rate: 0.0000001083, weight_decay: 0.0000142143

epoch time: 1135.4172363281s, input_fired_time: 14.6062755585, output_fired_time: 7.9539718628, propagate_forward_time: 432.6943969727, propagate_backward_time: 666.8915405273, weight_update_time: 0.6714417934, other_time: 13.2709960938
epoch time: 88.8739929199s, input_fired_time: 1.1059347391, output_fired_time: 0.0000000000, propagate_forward_time: 85.5176315308, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2504272461
validation[          , genome  5600] predictions:    5141/  10000 (51.41%), best:    6340/10000 (63.40%), error:     15435.04297, best error:     11116.94336 on epoch:     0, epoch:   70/100, mu: 0.9808734655, learning_rate: 0.0000001014, weight_decay: 0.0000134967

epoch time: 1130.4155273438s, input_fired_time: 14.5887165070, output_fired_time: 7.9051089287, propagate_forward_time: 430.6140747070, propagate_backward_time: 663.9285888672, weight_update_time: 0.6613360643, other_time: 13.3789672852
epoch time: 87.3527755737s, input_fired_time: 1.0970587730, output_fired_time: 0.0000000000, propagate_forward_time: 83.9738159180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2818984985
validation[          , genome  5600] predictions:    5043/  10000 (50.43%), best:    6340/10000 (63.40%), error:     15406.26563, best error:     11116.94336 on epoch:     0, epoch:   71/100, mu: 0.9813761711, learning_rate: 0.0000000950, weight_decay: 0.0000128153

epoch time: 1126.8431396484s, input_fired_time: 14.6080560684, output_fired_time: 7.8916244507, propagate_forward_time: 429.6396484375, propagate_backward_time: 661.5209350586, weight_update_time: 0.6619473696, other_time: 13.1829223633
epoch time: 89.6327743530s, input_fired_time: 1.1197509766, output_fired_time: 0.0000000000, propagate_forward_time: 86.3211364746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1918869019
validation[          , genome  5600] predictions:    5097/  10000 (50.97%), best:    6340/10000 (63.40%), error:     15443.75781, best error:     11116.94336 on epoch:     0, epoch:   72/100, mu: 0.9818511605, learning_rate: 0.0000000890, weight_decay: 0.0000121684

epoch time: 1124.4583740234s, input_fired_time: 14.5708560944, output_fired_time: 7.8148541451, propagate_forward_time: 428.8482055664, propagate_backward_time: 659.6961059570, weight_update_time: 0.6650143862, other_time: 13.5284423828
epoch time: 89.9740600586s, input_fired_time: 1.1262241602, output_fired_time: 0.0000000000, propagate_forward_time: 86.4952850342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3525543213
validation[          , genome  5600] predictions:    5111/  10000 (51.11%), best:    6340/10000 (63.40%), error:     15402.49121, best error:     11116.94336 on epoch:     0, epoch:   73/100, mu: 0.9822999835, learning_rate: 0.0000000833, weight_decay: 0.0000115541

epoch time: 1132.0200195313s, input_fired_time: 14.6446504593, output_fired_time: 7.8727684021, propagate_forward_time: 431.3401184082, propagate_backward_time: 664.8449707031, weight_update_time: 0.6635184884, other_time: 13.3175048828
epoch time: 92.4822769165s, input_fired_time: 1.1531269550, output_fired_time: 0.0000000000, propagate_forward_time: 88.9483337402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3808135986
validation[          , genome  5600] predictions:    5046/  10000 (50.46%), best:    6340/10000 (63.40%), error:     15385.38965, best error:     11116.94336 on epoch:     0, epoch:   74/100, mu: 0.9827241302, learning_rate: 0.0000000780, weight_decay: 0.0000109708

epoch time: 1139.8155517578s, input_fired_time: 14.5370349884, output_fired_time: 7.8633499146, propagate_forward_time: 433.8820495605, propagate_backward_time: 670.4578857422, weight_update_time: 0.6591444612, other_time: 13.0751953125
epoch time: 90.7290115356s, input_fired_time: 1.1010504961, output_fired_time: 0.0000000000, propagate_forward_time: 87.4173049927, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2106552124
validation[          , genome  5600] predictions:    5105/  10000 (51.05%), best:    6340/10000 (63.40%), error:     15395.77734, best error:     11116.94336 on epoch:     0, epoch:   75/100, mu: 0.9831249118, learning_rate: 0.0000000731, weight_decay: 0.0000104169

epoch time: 1150.8652343750s, input_fired_time: 14.6720819473, output_fired_time: 7.8806071281, propagate_forward_time: 438.3095703125, propagate_backward_time: 676.9340209961, weight_update_time: 0.6637937427, other_time: 13.0689086914
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600_0_r1785420667_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508406
read mu: 0.983504
read mu_delta: 0.944919
read initial_learning_rate: 1e-05
read learning_rate: 6.8442e-08
read learning_rate_delta: 0.936521
read initial_weight_decay: 0.000507052
read weight_decay: 9.89106e-06
read weight_decay_delta: 0.949516
read batch_size: 86
read epsilon: 1e-07
read alpha: 0.0882511
read input_dropout_probability: 0.00115475
read hidden_dropout_probability: 0.0934083
read velocity_reset: 1731
read epoch: 76
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6340
read best_validation_error: 11116.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5600
read normal distribution: '0 1.38065969944 -0.697693645954132'
generator_str: '584854677'
read generator: 584854677
read generated_by_map:
1 change_size_x 1
reading 77 nodes.
reading 311 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1196.9501953125s, input_fired_time: 14.8541946411133, output_fired_time: 7.91447973251343, propagate_forward_time: 471.117279052734, propagate_backward_time: 691.693481445313, weight_update_time: 0.65188193321228, other_time: 11.3707275390625
epoch time: 96.4843521118164s, input_fired_time: 1.09881663322449, output_fired_time: 0, propagate_forward_time: 93.4852447509766, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.90029144287109
validation[          , genome  5600] predictions:    5101/  10000 (51.01%), best:    6340/10000 (63.40%), error:     15416.34180, best error:     11116.94336 on epoch:     0, epoch:   76/100, mu: 0.9835035801, learning_rate: 0.0000000684, weight_decay: 0.0000098911

epoch time: 1185.5822753906s, input_fired_time: 14.6604890823, output_fired_time: 7.8603811264, propagate_forward_time: 465.8616027832, propagate_backward_time: 685.8850097656, weight_update_time: 0.6549745202, other_time: 11.3148193359
epoch time: 96.8783035278s, input_fired_time: 1.0891309977, output_fired_time: 0.0000000000, propagate_forward_time: 93.8969650269, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8922042847
validation[          , genome  5600] predictions:    5066/  10000 (50.66%), best:    6340/10000 (63.40%), error:     15439.63770, best error:     11116.94336 on epoch:     0, epoch:   77/100, mu: 0.9838613868, learning_rate: 0.0000000641, weight_decay: 0.0000093917

epoch time: 1206.0812988281s, input_fired_time: 14.8973169327, output_fired_time: 8.0024309158, propagate_forward_time: 474.1674499512, propagate_backward_time: 697.5617065430, weight_update_time: 0.6578093171, other_time: 11.4523315430
epoch time: 97.3227081299s, input_fired_time: 1.1038024426, output_fired_time: 0.0000000000, propagate_forward_time: 94.2696075439, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9492950439
validation[          , genome  5600] predictions:    5118/  10000 (51.18%), best:    6340/10000 (63.40%), error:     15384.65625, best error:     11116.94336 on epoch:     0, epoch:   78/100, mu: 0.9841995239, learning_rate: 0.0000000600, weight_decay: 0.0000089176

epoch time: 1186.9904785156s, input_fired_time: 14.7489995956, output_fired_time: 7.9194569588, propagate_forward_time: 466.2832031250, propagate_backward_time: 686.2352294922, weight_update_time: 0.6594523787, other_time: 11.8035888672
epoch time: 96.3053054810s, input_fired_time: 1.0967441797, output_fired_time: 0.0000000000, propagate_forward_time: 93.2350463867, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9735183716
validation[          , genome  5600] predictions:    5077/  10000 (50.77%), best:    6340/10000 (63.40%), error:     15435.81250, best error:     11116.94336 on epoch:     0, epoch:   79/100, mu: 0.9845190048, learning_rate: 0.0000000562, weight_decay: 0.0000084674

epoch time: 1186.9416503906s, input_fired_time: 14.8102922440, output_fired_time: 7.9174137115, propagate_forward_time: 465.7784729004, propagate_backward_time: 686.5713500977, weight_update_time: 0.6584120393, other_time: 11.8641967773
epoch time: 96.8343276978s, input_fired_time: 1.1077562571, output_fired_time: 0.0000000000, propagate_forward_time: 93.7581329346, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9684371948
validation[          , genome  5600] predictions:    5067/  10000 (50.67%), best:    6340/10000 (63.40%), error:     15419.44141, best error:     11116.94336 on epoch:     0, epoch:   80/100, mu: 0.9848209023, learning_rate: 0.0000000526, weight_decay: 0.0000080399

epoch time: 1184.7530517578s, input_fired_time: 14.7233505249, output_fired_time: 7.9034276009, propagate_forward_time: 464.1673278809, propagate_backward_time: 686.1204833984, weight_update_time: 0.6616194248, other_time: 11.8383789063
epoch time: 97.4176254272s, input_fired_time: 1.1051738262, output_fired_time: 0.0000000000, propagate_forward_time: 94.3918380737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9206161499
validation[          , genome  5600] predictions:    5056/  10000 (50.56%), best:    6340/10000 (63.40%), error:     15444.38770, best error:     11116.94336 on epoch:     0, epoch:   81/100, mu: 0.9851061702, learning_rate: 0.0000000493, weight_decay: 0.0000076341

epoch time: 1187.8608398438s, input_fired_time: 14.8307619095, output_fired_time: 7.9016633034, propagate_forward_time: 466.2113952637, propagate_backward_time: 687.1860961914, weight_update_time: 0.6961327791, other_time: 11.7308959961
epoch time: 95.9059448242s, input_fired_time: 1.0970983505, output_fired_time: 0.0000000000, propagate_forward_time: 92.8427734375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9660720825
validation[          , genome  5600] predictions:    5109/  10000 (51.09%), best:    6340/10000 (63.40%), error:     15453.78027, best error:     11116.94336 on epoch:     0, epoch:   82/100, mu: 0.9853757024, learning_rate: 0.0000000462, weight_decay: 0.0000072487

epoch time: 1182.3131103516s, input_fired_time: 14.7565736771, output_fired_time: 7.9043483734, propagate_forward_time: 463.2259216309, propagate_backward_time: 684.6113281250, weight_update_time: 0.6736282706, other_time: 11.8149414063
epoch time: 96.3821563721s, input_fired_time: 1.1022611856, output_fired_time: 0.0000000000, propagate_forward_time: 93.3034820557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9764099121
validation[          , genome  5600] predictions:    5107/  10000 (51.07%), best:    6340/10000 (63.40%), error:     15412.71484, best error:     11116.94336 on epoch:     0, epoch:   83/100, mu: 0.9856303930, learning_rate: 0.0000000432, weight_decay: 0.0000068827

epoch time: 1186.2556152344s, input_fired_time: 14.7212076187, output_fired_time: 7.9100842476, propagate_forward_time: 465.4191284180, propagate_backward_time: 686.5178833008, weight_update_time: 0.6625403762, other_time: 11.6873779297
epoch time: 97.1374816895s, input_fired_time: 1.1061762571, output_fired_time: 0.0000000000, propagate_forward_time: 94.0944747925, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9368286133
validation[          , genome  5600] predictions:    5085/  10000 (50.85%), best:    6340/10000 (63.40%), error:     15433.05078, best error:     11116.94336 on epoch:     0, epoch:   84/100, mu: 0.9858710766, learning_rate: 0.0000000405, weight_decay: 0.0000065353

epoch time: 1187.1668701172s, input_fired_time: 14.7554988861, output_fired_time: 7.9316844940, propagate_forward_time: 466.1416320801, propagate_backward_time: 686.6821899414, weight_update_time: 0.6617485285, other_time: 11.6559448242
epoch time: 95.9601440430s, input_fired_time: 1.1027419567, output_fired_time: 0.0000000000, propagate_forward_time: 92.8907928467, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9666061401
validation[          , genome  5600] predictions:    5074/  10000 (50.74%), best:    6340/10000 (63.40%), error:     15454.40137, best error:     11116.94336 on epoch:     0, epoch:   85/100, mu: 0.9860985279, learning_rate: 0.0000000379, weight_decay: 0.0000062053

epoch time: 1189.5102539063s, input_fired_time: 14.7759943008, output_fired_time: 7.9394001961, propagate_forward_time: 466.1386413574, propagate_backward_time: 689.0017700195, weight_update_time: 0.6581284404, other_time: 11.6543579102
epoch time: 96.8231124878s, input_fired_time: 1.0992298126, output_fired_time: 0.0000000000, propagate_forward_time: 93.8138885498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9099960327
validation[          , genome  5600] predictions:    5049/  10000 (50.49%), best:    6340/10000 (63.40%), error:     15456.15137, best error:     11116.94336 on epoch:     0, epoch:   86/100, mu: 0.9863134027, learning_rate: 0.0000000355, weight_decay: 0.0000058921

epoch time: 1184.7264404297s, input_fired_time: 14.7719697952, output_fired_time: 7.9497218132, propagate_forward_time: 464.6567993164, propagate_backward_time: 685.7465820313, weight_update_time: 0.6572480798, other_time: 11.6013793945
epoch time: 95.1362533569s, input_fired_time: 1.1087983847, output_fired_time: 0.0000000000, propagate_forward_time: 92.0648651123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9625930786
validation[          , genome  5600] predictions:    5061/  10000 (50.61%), best:    6340/10000 (63.40%), error:     15449.51367, best error:     11116.94336 on epoch:     0, epoch:   87/100, mu: 0.9865164757, learning_rate: 0.0000000333, weight_decay: 0.0000055946

epoch time: 1194.2882080078s, input_fired_time: 14.8806972504, output_fired_time: 7.9758849144, propagate_forward_time: 466.1490783691, propagate_backward_time: 693.3949584961, weight_update_time: 0.6724007726, other_time: 11.8876342773
epoch time: 96.8386459351s, input_fired_time: 1.1160408258, output_fired_time: 0.0000000000, propagate_forward_time: 93.7285308838, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9940719604
validation[          , genome  5600] predictions:    5072/  10000 (50.72%), best:    6340/10000 (63.40%), error:     15452.53613, best error:     11116.94336 on epoch:     0, epoch:   88/100, mu: 0.9867083430, learning_rate: 0.0000000312, weight_decay: 0.0000053122

epoch time: 1199.8903808594s, input_fired_time: 15.0373096466, output_fired_time: 8.0856981277, propagate_forward_time: 470.0391540527, propagate_backward_time: 694.8191528320, weight_update_time: 0.6714780331, other_time: 11.9089965820
epoch time: 96.7836151123s, input_fired_time: 1.1064661741, output_fired_time: 0.0000000000, propagate_forward_time: 93.7105178833, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9666290283
validation[          , genome  5600] predictions:    5081/  10000 (50.81%), best:    6340/10000 (63.40%), error:     15426.72266, best error:     11116.94336 on epoch:     0, epoch:   89/100, mu: 0.9868896604, learning_rate: 0.0000000292, weight_decay: 0.0000050440

epoch time: 26163.9921875000s, input_fired_time: 14.9550323486, output_fired_time: 8.1261625290, propagate_forward_time: 25437.2148437500, propagate_backward_time: 688.6057128906, weight_update_time: 0.6905186176, other_time: 15.0895996094
epoch time: 94.2293472290s, input_fired_time: 1.1804189682, output_fired_time: 0.0000000000, propagate_forward_time: 90.3073501587, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7415771484
validation[          , genome  5600] predictions:    5089/  10000 (50.89%), best:    6340/10000 (63.40%), error:     15459.11523, best error:     11116.94336 on epoch:     0, epoch:   90/100, mu: 0.9870609641, learning_rate: 0.0000000273, weight_decay: 0.0000047894

epoch time: 1168.3471679688s, input_fired_time: 14.8145990372, output_fired_time: 8.0243492126, propagate_forward_time: 446.9407958984, propagate_backward_time: 684.5280151367, weight_update_time: 0.6748609543, other_time: 14.0394897461
epoch time: 92.9470901489s, input_fired_time: 1.1334028244, output_fired_time: 0.0000000000, propagate_forward_time: 89.5170059204, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2966842651
validation[          , genome  5600] predictions:    5092/  10000 (50.92%), best:    6340/10000 (63.40%), error:     15420.53223, best error:     11116.94336 on epoch:     0, epoch:   91/100, mu: 0.9872228503, learning_rate: 0.0000000256, weight_decay: 0.0000045476

epoch time: 1169.4379882813s, input_fired_time: 14.8065385818, output_fired_time: 8.0111837387, propagate_forward_time: 448.6389160156, propagate_backward_time: 684.0353393555, weight_update_time: 0.6865177155, other_time: 13.9459838867
epoch time: 93.4707107544s, input_fired_time: 1.1504645348, output_fired_time: 0.0000000000, propagate_forward_time: 89.9510345459, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3692092896
validation[          , genome  5600] predictions:    5086/  10000 (50.86%), best:    6340/10000 (63.40%), error:     15442.03223, best error:     11116.94336 on epoch:     0, epoch:   92/100, mu: 0.9873757958, learning_rate: 0.0000000240, weight_decay: 0.0000043180

epoch time: 1162.5864257813s, input_fired_time: 14.7646884918, output_fired_time: 7.9938769341, propagate_forward_time: 444.1871948242, propagate_backward_time: 681.3922119141, weight_update_time: 0.6862977147, other_time: 14.2484741211
epoch time: 92.2160415649s, input_fired_time: 1.1609092951, output_fired_time: 0.0000000000, propagate_forward_time: 88.5679779053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4871520996
validation[          , genome  5600] predictions:    5058/  10000 (50.58%), best:    6340/10000 (63.40%), error:     15439.12402, best error:     11116.94336 on epoch:     0, epoch:   93/100, mu: 0.9875203371, learning_rate: 0.0000000224, weight_decay: 0.0000041000

epoch time: 1167.0924072266s, input_fired_time: 14.9321060181, output_fired_time: 8.1892518997, propagate_forward_time: 447.5556335449, propagate_backward_time: 682.0731811523, weight_update_time: 0.6881149411, other_time: 14.3422241211
epoch time: 93.5113830566s, input_fired_time: 1.1773854494, output_fired_time: 0.0000000000, propagate_forward_time: 89.8051071167, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5288925171
validation[          , genome  5600] predictions:    5082/  10000 (50.82%), best:    6340/10000 (63.40%), error:     15427.14941, best error:     11116.94336 on epoch:     0, epoch:   94/100, mu: 0.9876568913, learning_rate: 0.0000000210, weight_decay: 0.0000038930

epoch time: 1170.8994140625s, input_fired_time: 15.0903177261, output_fired_time: 8.3570623398, propagate_forward_time: 447.5734558105, propagate_backward_time: 685.2263793945, weight_update_time: 0.6936907768, other_time: 14.6521606445
epoch time: 93.9471740723s, input_fired_time: 1.2113841772, output_fired_time: 0.0000000000, propagate_forward_time: 90.1787033081, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5570831299
validation[          , genome  5600] predictions:    5103/  10000 (51.03%), best:    6340/10000 (63.40%), error:     15445.26367, best error:     11116.94336 on epoch:     0, epoch:   95/100, mu: 0.9877859354, learning_rate: 0.0000000197, weight_decay: 0.0000036965

epoch time: 1158.7297363281s, input_fired_time: 14.8959846497, output_fired_time: 8.2390308380, propagate_forward_time: 441.4246826172, propagate_backward_time: 679.5104370117, weight_update_time: 0.7070767879, other_time: 14.6596069336
epoch time: 92.6786270142s, input_fired_time: 1.1729842424, output_fired_time: 0.0000000000, propagate_forward_time: 89.0017166138, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5039291382
validation[          , genome  5600] predictions:    5074/  10000 (50.74%), best:    6340/10000 (63.40%), error:     15471.95996, best error:     11116.94336 on epoch:     0, epoch:   96/100, mu: 0.9879078865, learning_rate: 0.0000000184, weight_decay: 0.0000035099

epoch time: 1158.5583496094s, input_fired_time: 14.9553670883, output_fired_time: 8.0679464340, propagate_forward_time: 442.1899719238, propagate_backward_time: 679.0706787109, weight_update_time: 0.6930761933, other_time: 14.2744140625
epoch time: 92.5911788940s, input_fired_time: 1.2106654644, output_fired_time: 0.0000000000, propagate_forward_time: 88.9112701416, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4692459106
validation[          , genome  5600] predictions:    5070/  10000 (50.70%), best:    6340/10000 (63.40%), error:     15473.75195, best error:     11116.94336 on epoch:     0, epoch:   97/100, mu: 0.9880231023, learning_rate: 0.0000000173, weight_decay: 0.0000033327

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5600_0_r1785420667_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.508406
read mu: 0.988132
read mu_delta: 0.944919
read initial_learning_rate: 1e-05
read learning_rate: 1.61698e-08
read learning_rate_delta: 0.936521
read initial_weight_decay: 0.000507052
read weight_decay: 3.16445e-06
read weight_decay_delta: 0.949516
read batch_size: 86
read epsilon: 1e-07
read alpha: 0.0882511
read input_dropout_probability: 0.00115475
read hidden_dropout_probability: 0.0934083
read velocity_reset: 1731
read epoch: 98
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6340
read best_validation_error: 11116.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5600
read normal distribution: '0 1.38065969944 -0.697693645954132'
generator_str: '1933550808'
read generator: 1933550808
read generated_by_map:
1 change_size_x 1
reading 77 nodes.
reading 311 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1176.72766113281s, input_fired_time: 15.2832012176514, output_fired_time: 8.34232616424561, propagate_forward_time: 448.890808105469, propagate_backward_time: 689.234680175781, weight_update_time: 0.703029274940491, other_time: 14.9766845703125
epoch time: 93.3067245483398s, input_fired_time: 1.17262005805969, output_fired_time: 0, propagate_forward_time: 89.5685806274414, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.56552124023438
validation[          , genome  5600] predictions:    5098/  10000 (50.98%), best:    6340/10000 (63.40%), error:     15463.93750, best error:     11116.94336 on epoch:     0, epoch:   98/100, mu: 0.9881320000, learning_rate: 0.0000000162, weight_decay: 0.0000031644

epoch time: 1157.5032958984s, input_fired_time: 14.8462400436, output_fired_time: 8.1089448929, propagate_forward_time: 440.9263610840, propagate_backward_time: 678.7570800781, weight_update_time: 0.6877734065, other_time: 14.8647460938
epoch time: 94.7156295776s, input_fired_time: 1.1489760876, output_fired_time: 0.0000000000, propagate_forward_time: 91.2158660889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3507843018
validation[          , genome  5600] predictions:    5080/  10000 (50.80%), best:    6340/10000 (63.40%), error:     15440.04395, best error:     11116.94336 on epoch:     0, epoch:   99/100, mu: 0.9882348776, learning_rate: 0.0000000151, weight_decay: 0.0000030047

epoch time: 1178.5557861328s, input_fired_time: 14.6516275406, output_fired_time: 8.0053329468, propagate_forward_time: 461.0072326660, propagate_backward_time: 682.2270507813, weight_update_time: 0.6660566926, other_time: 12.6645507813
epoch time: 97.3559646606s, input_fired_time: 1.1117972136, output_fired_time: 0.0000000000, propagate_forward_time: 94.1776580811, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0665130615
validation[          , genome  5600] predictions:    5090/  10000 (50.90%), best:    6340/10000 (63.40%), error:     15441.26367, best error:     11116.94336 on epoch:     0, epoch:  100/100, mu: 0.9883320928, learning_rate: 0.0000000142, weight_decay: 0.0000028530

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 23310.5234375000s, input_fired_time: 6.0018863678, output_fired_time: 0.0000000000, propagate_forward_time: 23294.9355468750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.5859375000
best training[          , genome  5600] predictions:   34656/  50000 (69.31%), best:    6340/10000 (63.40%), error:     50234.91797, best error:     11116.94336 on epoch:     0, epoch:  101/100, mu: 0.9884239435, learning_rate: 0.0000000133, weight_decay: 0.0000027090
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 90.7187042236s, input_fired_time: 1.2093955278, output_fired_time: 0.0000000000, propagate_forward_time: 87.5922927856, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9170150757
   testing[          , genome  5600] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     11116.94336, best error:     11116.94336 on epoch:     0, epoch:  101/100, mu: 0.9884239435, learning_rate: 0.0000000133, weight_decay: 0.0000027090
backpropagation finished successfully!
18:13:10 (2928): called boinc_finish(0)

</stderr_txt>
]]>