Nameexact_genome_1542069603_186_5788_1
Workunit4086000
Created13 Nov 2018, 11:55:29 UTC
Sent13 Nov 2018, 12:27:47 UTC
Report deadline5 Feb 2019, 15:25:57 UTC
Received25 Nov 2018, 14:01:44 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID70175
Run time1 days 6 hours 12 min 14 sec
CPU time20 hours 55 min 18 sec
Validate stateValid
Credit21,030.93
Device peak FLOPS3.62 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34

Stderr output

<core_client_version>7.0.80</core_client_version>
<![CDATA[
<stderr_txt>
ropagation!
epoch time: 5001.50390625s, input_fired_time: 27.2496910095215, output_fired_time: 10.3258819580078, propagate_forward_time: 611.1240234375, propagate_backward_time: 1095.07214355469, weight_update_time: 3.27113914489746, other_time: 3257.732421875
epoch time: 150.899642944336s, input_fired_time: 2.65655040740967, output_fired_time: 0, propagate_forward_time: 118.319702148438, propagate_backward_time: 0, weight_update_time: 0, other_time: 29.9233856201172
validation[          , genome  5788] predictions:    6010/  10000 (60.10%), best:    6053/10000 (60.53%), error:     11572.78809, best error:     11463.34863 on epoch:    40, epoch:   42/100, mu: 0.9722827673, learning_rate: 0.0005088012, weight_decay: 0.0000609088

epoch time: 2770.8750000000s, input_fired_time: 40.3421669006, output_fired_time: 12.1357154846, propagate_forward_time: 570.7280883789, propagate_backward_time: 1069.9492187500, weight_update_time: 0.4634632766, other_time: 1077.7197265625
epoch time: 148.8800811768s, input_fired_time: 0.7922850251, output_fired_time: 0.0000000000, propagate_forward_time: 135.8685455322, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.2192535400
validation[          , genome  5788] predictions:    6013/  10000 (60.13%), best:    6013/10000 (60.13%), error:     11430.65332, best error:     11430.65332 on epoch:    43, epoch:   43/100, mu: 0.9735869169, learning_rate: 0.0004784127, weight_decay: 0.0000582861

epoch time: 8230.7343750000s, input_fired_time: 67.7533111572, output_fired_time: 12.6983518600, propagate_forward_time: 760.9049072266, propagate_backward_time: 1283.8646240234, weight_update_time: 6.7162375450, other_time: 6105.5131835938
epoch time: 165.8924713135s, input_fired_time: 0.7029394507, output_fired_time: 0.0000000000, propagate_forward_time: 130.7574157715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 34.4321136475
validation[          , genome  5788] predictions:    5919/  10000 (59.19%), best:    6013/10000 (60.13%), error:     11860.34766, best error:     11430.65332 on epoch:    43, epoch:   44/100, mu: 0.9747950435, learning_rate: 0.0004498392, weight_decay: 0.0000557763

epoch time: 2771.9709472656s, input_fired_time: 25.7395706177, output_fired_time: 16.6584930420, propagate_forward_time: 645.8550415039, propagate_backward_time: 1085.5815429688, weight_update_time: 0.5797325969, other_time: 998.1364746094
epoch time: 156.1960906982s, input_fired_time: 1.2121475935, output_fired_time: 0.0000000000, propagate_forward_time: 126.3788604736, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 28.6050872803
validation[          , genome  5788] predictions:    6044/  10000 (60.44%), best:    6013/10000 (60.13%), error:     11461.13477, best error:     11430.65332 on epoch:    43, epoch:   45/100, mu: 0.9759142399, learning_rate: 0.0004229723, weight_decay: 0.0000533746

epoch time: 2438.8395996094s, input_fired_time: 29.0868434906, output_fired_time: 15.8990907669, propagate_forward_time: 656.0364990234, propagate_backward_time: 1070.0323486328, weight_update_time: 0.5149618983, other_time: 667.7846679688
epoch time: 140.5658874512s, input_fired_time: 2.6282777786, output_fired_time: 0.0000000000, propagate_forward_time: 118.4976730347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.4399337769
validation[          , genome  5788] predictions:    5949/  10000 (59.49%), best:    6013/10000 (60.13%), error:     11655.10352, best error:     11430.65332 on epoch:    43, epoch:   46/100, mu: 0.9769510627, learning_rate: 0.0003977100, weight_decay: 0.0000510763

epoch time: 2440.5971679688s, input_fired_time: 20.5140037537, output_fired_time: 15.2020187378, propagate_forward_time: 621.6325683594, propagate_backward_time: 1047.5450439453, weight_update_time: 0.4595553279, other_time: 735.7037353516
epoch time: 141.0620269775s, input_fired_time: 2.6350460052, output_fired_time: 0.0000000000, propagate_forward_time: 127.8391494751, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.5878372192
validation[          , genome  5788] predictions:    5950/  10000 (59.50%), best:    6013/10000 (60.13%), error:     11589.46582, best error:     11430.65332 on epoch:    43, epoch:   47/100, mu: 0.9779115915, learning_rate: 0.0003739565, weight_decay: 0.0000488769

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788_1_r1609043831_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550452
read mu: 0.978801
read mu_delta: 0.926392
read initial_learning_rate: 0.00675858
read learning_rate: 0.000351622
read learning_rate_delta: 0.940274
read initial_weight_decay: 0.000386832
read weight_decay: 4.67723e-05
read weight_decay_delta: 0.95694
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0613628
read input_dropout_probability: 0.00100196
read hidden_dropout_probability: 0.110106
read velocity_reset: 2287
read epoch: 48
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 43
read number_validation_images: 10000
read best_validation_predictions: 6013
read best_validation_error: 11430.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5788
read normal distribution: '1 0 0'
generator_str: '388124648'
read generator: 388124648
read generated_by_map:
1 disable_node 1
reading 71 nodes.
reading 273 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 7856.0693359375s, input_fired_time: 49.8831787109375, output_fired_time: 23.0307426452637, propagate_forward_time: 888.246765136719, propagate_backward_time: 1297.39929199219, weight_update_time: 1.79165208339691, other_time: 5597.50927734375
epoch time: 168.138031005859s, input_fired_time: 2.18924331665039, output_fired_time: 0, propagate_forward_time: 138.647262573242, propagate_backward_time: 0, weight_update_time: 0, other_time: 27.3015289306641
validation[          , genome  5788] predictions:    6093/  10000 (60.93%), best:    6093/10000 (60.93%), error:     11280.21973, best error:     11280.21973 on epoch:    48, epoch:   48/100, mu: 0.9788013697, learning_rate: 0.0003516217, weight_decay: 0.0000467723

epoch time: 6052.5268554688s, input_fired_time: 33.4476356506, output_fired_time: 10.3226976395, propagate_forward_time: 674.9131469727, propagate_backward_time: 1145.7946777344, weight_update_time: 0.5288349986, other_time: 4188.0488281250
epoch time: 185.7827148438s, input_fired_time: 0.7026950717, output_fired_time: 0.0000000000, propagate_forward_time: 132.4644012451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 52.6156158447
validation[          , genome  5788] predictions:    6009/  10000 (60.09%), best:    6093/10000 (60.93%), error:     11510.39941, best error:     11280.21973 on epoch:    48, epoch:   49/100, mu: 0.9796257019, learning_rate: 0.0003306209, weight_decay: 0.0000447583

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788_1_r1609043831_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550452
read mu: 0.980389
read mu_delta: 0.926392
read initial_learning_rate: 0.00675858
read learning_rate: 0.000310874
read learning_rate_delta: 0.940274
read initial_weight_decay: 0.000386832
read weight_decay: 4.2831e-05
read weight_decay_delta: 0.95694
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0613628
read input_dropout_probability: 0.00100196
read hidden_dropout_probability: 0.110106
read velocity_reset: 2287
read epoch: 50
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 48
read number_validation_images: 10000
read best_validation_predictions: 6093
read best_validation_error: 11280.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5788
read normal distribution: '1 0 0'
generator_str: '2136862537'
read generator: 2136862537
read generated_by_map:
1 disable_node 1
reading 71 nodes.
reading 273 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 4545.037109375s, input_fired_time: 48.7432823181152, output_fired_time: 9.81468105316162, propagate_forward_time: 626.97802734375, propagate_backward_time: 1055.27026367188, weight_update_time: 2.90176343917847, other_time: 2804.23120117188
epoch time: 178.884735107422s, input_fired_time: 1.78525996208191, output_fired_time: 0, propagate_forward_time: 119.875427246094, propagate_backward_time: 0, weight_update_time: 0, other_time: 57.2240447998047
validation[          , genome  5788] predictions:    6101/  10000 (61.01%), best:    6093/10000 (60.93%), error:     11316.22852, best error:     11280.21973 on epoch:    48, epoch:   50/100, mu: 0.9803893566, learning_rate: 0.0003108743, weight_decay: 0.0000428310

epoch time: 2058.4826660156s, input_fired_time: 22.5637989044, output_fired_time: 12.9030694962, propagate_forward_time: 616.7368164063, propagate_backward_time: 1062.6195068359, weight_update_time: 0.5888602138, other_time: 343.6594238281
epoch time: 130.7813110352s, input_fired_time: 0.9985355735, output_fired_time: 0.0000000000, propagate_forward_time: 122.2327346802, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.5500411987
validation[          , genome  5788] predictions:    6136/  10000 (61.36%), best:    6136/10000 (61.36%), error:     11249.53027, best error:     11249.53027 on epoch:    51, epoch:   51/100, mu: 0.9810967445, learning_rate: 0.0002923071, weight_decay: 0.0000409867

epoch time: 1695.9034423828s, input_fired_time: 23.9295787811, output_fired_time: 9.4807939529, propagate_forward_time: 609.2980346680, propagate_backward_time: 1026.8502197266, weight_update_time: 0.5004940629, other_time: 26.3448486328
epoch time: 125.4886169434s, input_fired_time: 0.7026796937, output_fired_time: 0.0000000000, propagate_forward_time: 119.0254211426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7605133057
validation[          , genome  5788] predictions:    6003/  10000 (60.03%), best:    6136/10000 (61.36%), error:     11567.67383, best error:     11249.53027 on epoch:    51, epoch:   52/100, mu: 0.9817520976, learning_rate: 0.0002748489, weight_decay: 0.0000392218

epoch time: 1682.0432128906s, input_fired_time: 26.4415245056, output_fired_time: 9.1222896576, propagate_forward_time: 589.0393066406, propagate_backward_time: 1023.8190917969, weight_update_time: 0.4596980512, other_time: 33.6209716797
epoch time: 126.8646163940s, input_fired_time: 1.6846504211, output_fired_time: 0.0000000000, propagate_forward_time: 121.6349868774, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5449752808
validation[          , genome  5788] predictions:    6182/  10000 (61.82%), best:    6182/10000 (61.82%), error:     11127.53906, best error:     11127.53906 on epoch:    53, epoch:   53/100, mu: 0.9823592305, learning_rate: 0.0002584334, weight_decay: 0.0000375329

epoch time: 1688.5860595703s, input_fired_time: 33.5063400269, output_fired_time: 8.1120023727, propagate_forward_time: 588.4537353516, propagate_backward_time: 1037.6497802734, weight_update_time: 0.4588562250, other_time: 20.8641357422
epoch time: 123.4950256348s, input_fired_time: 0.7014088035, output_fired_time: 0.0000000000, propagate_forward_time: 120.2130813599, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5805358887
validation[          , genome  5788] predictions:    6155/  10000 (61.55%), best:    6182/10000 (61.82%), error:     11134.99902, best error:     11127.53906 on epoch:    53, epoch:   54/100, mu: 0.9829216599, learning_rate: 0.0002429983, weight_decay: 0.0000359167

epoch time: 1677.4705810547s, input_fired_time: 25.4524288177, output_fired_time: 11.0928850174, propagate_forward_time: 597.8179321289, propagate_backward_time: 1024.3811035156, weight_update_time: 2.4308471680, other_time: 18.7263183594
epoch time: 125.5133132935s, input_fired_time: 0.6973821521, output_fired_time: 0.0000000000, propagate_forward_time: 120.2056121826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6103210449
validation[          , genome  5788] predictions:    6160/  10000 (61.60%), best:    6160/10000 (61.60%), error:     11122.80762, best error:     11122.80762 on epoch:    55, epoch:   55/100, mu: 0.9834426641, learning_rate: 0.0002284851, weight_decay: 0.0000343702

epoch time: 1664.9383544922s, input_fired_time: 35.1183052063, output_fired_time: 15.9560613632, propagate_forward_time: 583.4788208008, propagate_backward_time: 1004.6422119141, weight_update_time: 0.4486120045, other_time: 25.7430419922
epoch time: 122.8348159790s, input_fired_time: 1.6806023121, output_fired_time: 0.0000000000, propagate_forward_time: 115.6338424683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5203704834
validation[          , genome  5788] predictions:    6197/  10000 (61.97%), best:    6197/10000 (61.97%), error:     11027.56250, best error:     11027.56250 on epoch:    56, epoch:   56/100, mu: 0.9839253426, learning_rate: 0.0002148386, weight_decay: 0.0000328902

epoch time: 1645.4327392578s, input_fired_time: 20.3275966644, output_fired_time: 10.9038724899, propagate_forward_time: 597.7295532227, propagate_backward_time: 983.0167236328, weight_update_time: 0.4514113069, other_time: 33.4548950195
epoch time: 125.0231704712s, input_fired_time: 0.7035560608, output_fired_time: 0.0000000000, propagate_forward_time: 119.6356124878, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.6839981079
validation[          , genome  5788] predictions:    6152/  10000 (61.52%), best:    6152/10000 (61.52%), error:     11009.39746, best error:     11009.39746 on epoch:    57, epoch:   57/100, mu: 0.9843724966, learning_rate: 0.0002020073, weight_decay: 0.0000314739

epoch time: 1668.3244628906s, input_fired_time: 27.6350307465, output_fired_time: 7.3988204002, propagate_forward_time: 589.3140258789, propagate_backward_time: 1019.3930053711, weight_update_time: 0.4759401083, other_time: 24.5835571289
epoch time: 130.6622619629s, input_fired_time: 4.9157791138, output_fired_time: 0.0000000000, propagate_forward_time: 120.2571029663, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4893798828
validation[          , genome  5788] predictions:    6157/  10000 (61.57%), best:    6152/10000 (61.52%), error:     11078.50977, best error:     11009.39746 on epoch:    57, epoch:   58/100, mu: 0.9847867489, learning_rate: 0.0001899423, weight_decay: 0.0000301187

epoch time: 1705.7354736328s, input_fired_time: 27.2707767487, output_fired_time: 21.1531639099, propagate_forward_time: 596.3690185547, propagate_backward_time: 1029.3659667969, weight_update_time: 0.5751894116, other_time: 31.5765380859
epoch time: 130.2721557617s, input_fired_time: 2.9030067921, output_fired_time: 0.0000000000, propagate_forward_time: 123.9392318726, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4299163818
validation[          , genome  5788] predictions:    6262/  10000 (62.62%), best:    6262/10000 (62.62%), error:     10856.67285, best error:     10856.67285 on epoch:    59, epoch:   59/100, mu: 0.9851704836, learning_rate: 0.0001785978, weight_decay: 0.0000288217

epoch time: 12407.5859375000s, input_fired_time: 54.1993179321, output_fired_time: 7.6829996109, propagate_forward_time: 9309.3535156250, propagate_backward_time: 1058.2635498047, weight_update_time: 0.6461352706, other_time: 1978.0870361328
epoch time: 140.1522979736s, input_fired_time: 1.7770912647, output_fired_time: 0.0000000000, propagate_forward_time: 122.0227508545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 16.3524627686
validation[          , genome  5788] predictions:    6199/  10000 (61.99%), best:    6262/10000 (62.62%), error:     11038.81836, best error:     10856.67285 on epoch:    59, epoch:   60/100, mu: 0.9855259657, learning_rate: 0.0001679310, weight_decay: 0.0000275807

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788_1_r1609043831_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550452
read mu: 0.985855
read mu_delta: 0.926392
read initial_learning_rate: 0.00675858
read learning_rate: 0.000157901
read learning_rate_delta: 0.940274
read initial_weight_decay: 0.000386832
read weight_decay: 2.63931e-05
read weight_decay_delta: 0.95694
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0613628
read input_dropout_probability: 0.00100196
read hidden_dropout_probability: 0.110106
read velocity_reset: 2287
read epoch: 61
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 59
read number_validation_images: 10000
read best_validation_predictions: 6262
read best_validation_error: 10856.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5788
read normal distribution: '1 0 0'
generator_str: '1849084797'
read generator: 1849084797
read generated_by_map:
1 disable_node 1
reading 71 nodes.
reading 273 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_186_5788_1_r1609043831_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550452
read mu: 0.985855
read mu_delta: 0.926392
read initial_learning_rate: 0.00675858
read learning_rate: 0.000157901
read learning_rate_delta: 0.940274
read initial_weight_decay: 0.000386832
read weight_decay: 2.63931e-05
read weight_decay_delta: 0.95694
read batch_size: 91
read epsilon: 1e-07
read alpha: 0.0613628
read input_dropout_probability: 0.00100196
read hidden_dropout_probability: 0.110106
read velocity_reset: 2287
read epoch: 61
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 59
read number_validation_images: 10000
read best_validation_predictions: 6262
read best_validation_error: 10856.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5788
read normal distribution: '1 0 0'
generator_str: '1849084797'
read generator: 1849084797
read generated_by_map:
1 disable_node 1
reading 71 nodes.
reading 273 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1709.26135253906s, input_fired_time: 23.5048980712891, output_fired_time: 8.31904983520508, propagate_forward_time: 621.862182617188, propagate_backward_time: 1033.3134765625, weight_update_time: 0.464852839708328, other_time: 22.26171875
epoch time: 132.643188476563s, input_fired_time: 0.761003196239471, output_fired_time: 0, propagate_forward_time: 128.176284790039, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.70590209960938
validation[          , genome  5788] predictions:    6274/  10000 (62.74%), best:    6274/10000 (62.74%), error:     10797.90723, best error:     10797.90723 on epoch:    61, epoch:   61/100, mu: 0.9858552814, learning_rate: 0.0001579012, weight_decay: 0.0000263931

epoch time: 3485.6884765625s, input_fired_time: 37.7867012024, output_fired_time: 5.3328304291, propagate_forward_time: 616.0874633789, propagate_backward_time: 1070.8959960938, weight_update_time: 0.4727916121, other_time: 1755.5856933594
epoch time: 142.9027099609s, input_fired_time: 1.6940238476, output_fired_time: 0.0000000000, propagate_forward_time: 121.9608535767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.2478256226
validation[          , genome  5788] predictions:    6239/  10000 (62.39%), best:    6239/10000 (62.39%), error:     10782.89063, best error:     10782.89063 on epoch:    62, epoch:   62/100, mu: 0.9861603379, learning_rate: 0.0001484704, weight_decay: 0.0000252566

epoch time: 6514.5351562500s, input_fired_time: 104.8208160400, output_fired_time: 12.7477254868, propagate_forward_time: 696.9764404297, propagate_backward_time: 1104.5523681641, weight_update_time: 4.7986655235, other_time: 4595.4379882813
epoch time: 188.5719604492s, input_fired_time: 3.3802931309, output_fired_time: 0.0000000000, propagate_forward_time: 154.1264648438, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 31.0652008057
validation[          , genome  5788] predictions:    6211/  10000 (62.11%), best:    6239/10000 (62.39%), error:     10895.73047, best error:     10782.89063 on epoch:    62, epoch:   63/100, mu: 0.9864429832, learning_rate: 0.0001396029, weight_decay: 0.0000241690

epoch time: 4854.3842773438s, input_fired_time: 57.5377693176, output_fired_time: 15.7548484802, propagate_forward_time: 652.1832275391, propagate_backward_time: 1063.1662597656, weight_update_time: 0.9440399408, other_time: 3065.7424316406
epoch time: 143.1823272705s, input_fired_time: 5.0785470009, output_fired_time: 0.0000000000, propagate_forward_time: 118.6451644897, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.4586105347
validation[          , genome  5788] predictions:    6156/  10000 (61.56%), best:    6239/10000 (62.39%), error:     11153.97363, best error:     10782.89063 on epoch:    62, epoch:   64/100, mu: 0.9867048264, learning_rate: 0.0001312651, weight_decay: 0.0000231283

epoch time: 1797.2755126953s, input_fired_time: 33.1697235107, output_fired_time: 16.0639629364, propagate_forward_time: 628.7873535156, propagate_backward_time: 1076.3393554688, weight_update_time: 1.6104575396, other_time: 42.9151611328
epoch time: 131.3265075684s, input_fired_time: 0.9808341861, output_fired_time: 0.0000000000, propagate_forward_time: 126.0791015625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2665710449
validation[          , genome  5788] predictions:    6192/  10000 (61.92%), best:    6239/10000 (62.39%), error:     11107.33594, best error:     10782.89063 on epoch:    62, epoch:   65/100, mu: 0.9869473577, learning_rate: 0.0001234252, weight_decay: 0.0000221324

epoch time: 1690.8585205078s, input_fired_time: 31.7222423553, output_fired_time: 13.9806537628, propagate_forward_time: 600.2182006836, propagate_backward_time: 1030.3548583984, weight_update_time: 0.4642127156, other_time: 14.5823974609
epoch time: 126.5485000610s, input_fired_time: 2.6897573471, output_fired_time: 0.0000000000, propagate_forward_time: 120.3098983765, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5488433838
validation[          , genome  5788] predictions:    6242/  10000 (62.42%), best:    6239/10000 (62.39%), error:     10795.10645, best error:     10782.89063 on epoch:    62, epoch:   66/100, mu: 0.9871720672, learning_rate: 0.0001160535, weight_decay: 0.0000211794

epoch time: 1689.8173828125s, input_fired_time: 26.4485759735, output_fired_time: 10.0253963470, propagate_forward_time: 612.5401000977, propagate_backward_time: 1016.9735717773, weight_update_time: 0.4543509185, other_time: 23.8296508789
epoch time: 124.9898147583s, input_fired_time: 1.8151050806, output_fired_time: 0.0000000000, propagate_forward_time: 115.5250930786, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.6496200562
validation[          , genome  5788] predictions:    6275/  10000 (62.75%), best:    6275/10000 (62.75%), error:     10743.59570, best error:     10743.59570 on epoch:    67, epoch:   67/100, mu: 0.9873802066, learning_rate: 0.0001091222, weight_decay: 0.0000202674

epoch time: 1687.5690917969s, input_fired_time: 31.5103988647, output_fired_time: 10.1080675125, propagate_forward_time: 618.2036132813, propagate_backward_time: 1008.1833496094, weight_update_time: 1.4358615875, other_time: 19.5637207031
epoch time: 122.4710464478s, input_fired_time: 0.7167341709, output_fired_time: 0.0000000000, propagate_forward_time: 116.2147140503, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5395965576
validation[          , genome  5788] predictions:    6207/  10000 (62.07%), best:    6275/10000 (62.75%), error:     10901.07910, best error:     10743.59570 on epoch:    67, epoch:   68/100, mu: 0.9875730276, learning_rate: 0.0001026048, weight_decay: 0.0000193947

epoch time: 1694.4460449219s, input_fired_time: 26.5729045868, output_fired_time: 13.0765247345, propagate_forward_time: 621.2312622070, propagate_backward_time: 1010.1188354492, weight_update_time: 0.4531427622, other_time: 23.4465942383
epoch time: 127.2592086792s, input_fired_time: 1.7030719519, output_fired_time: 0.0000000000, propagate_forward_time: 116.9366836548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.6194534302
validation[          , genome  5788] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10712.26465, best error:     10712.26465 on epoch:    69, epoch:   69/100, mu: 0.9877516627, learning_rate: 0.0000964766, weight_decay: 0.0000185595

epoch time: 1685.1835937500s, input_fired_time: 32.3697128296, output_fired_time: 7.1482024193, propagate_forward_time: 605.5814208984, propagate_backward_time: 1000.1541748047, weight_update_time: 1.4524700642, other_time: 39.9300537109
epoch time: 126.8485336304s, input_fired_time: 0.7240084410, output_fired_time: 0.0000000000, propagate_forward_time: 122.5330505371, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5914764404
validation[          , genome  5788] predictions:    6254/  10000 (62.54%), best:    6317/10000 (63.17%), error:     10757.19336, best error:     10712.26465 on epoch:    69, epoch:   70/100, mu: 0.9879171848, learning_rate: 0.0000907145, weight_decay: 0.0000177604

epoch time: 1698.3137207031s, input_fired_time: 30.9765377045, output_fired_time: 10.9833269119, propagate_forward_time: 604.6233520508, propagate_backward_time: 1020.9151000977, weight_update_time: 0.4558738470, other_time: 30.8153686523
epoch time: 127.5296783447s, input_fired_time: 0.7159802318, output_fired_time: 0.0000000000, propagate_forward_time: 119.1463394165, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.6673583984
validation[          , genome  5788] predictions:    6227/  10000 (62.27%), best:    6317/10000 (63.17%), error:     10953.62109, best error:     10712.26465 on epoch:    69, epoch:   71/100, mu: 0.9880704880, learning_rate: 0.0000852965, weight_decay: 0.0000169956

epoch time: 1682.8281250000s, input_fired_time: 20.4591312408, output_fired_time: 17.3267402649, propagate_forward_time: 595.3883666992, propagate_backward_time: 1018.7312011719, weight_update_time: 0.4514964223, other_time: 30.9226074219
epoch time: 122.7932815552s, input_fired_time: 1.6947765350, output_fired_time: 0.0000000000, propagate_forward_time: 117.5685043335, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5299987793
validation[          , genome  5788] predictions:    6266/  10000 (62.66%), best:    6317/10000 (63.17%), error:     10758.10449, best error:     10712.26465 on epoch:    69, epoch:   72/100, mu: 0.9882125258, learning_rate: 0.0000802021, weight_decay: 0.0000162638

epoch time: 1677.9270019531s, input_fired_time: 18.3292503357, output_fired_time: 11.0521717072, propagate_forward_time: 605.8038330078, propagate_backward_time: 1019.0504150391, weight_update_time: 0.4528850615, other_time: 23.6914062500
epoch time: 124.5128097534s, input_fired_time: 0.7198703289, output_fired_time: 0.0000000000, propagate_forward_time: 118.1738891602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6190490723
validation[          , genome  5788] predictions:    6299/  10000 (62.99%), best:    6299/10000 (62.99%), error:     10687.45605, best error:     10687.45605 on epoch:    73, epoch:   73/100, mu: 0.9883440733, learning_rate: 0.0000754120, weight_decay: 0.0000155635

epoch time: 1668.5145263672s, input_fired_time: 29.4549217224, output_fired_time: 11.1961574554, propagate_forward_time: 579.5711669922, propagate_backward_time: 1025.5161132813, weight_update_time: 0.4610295296, other_time: 22.7761230469
epoch time: 125.6552581787s, input_fired_time: 0.7097030878, output_fired_time: 0.0000000000, propagate_forward_time: 121.3578414917, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5877151489
validation[          , genome  5788] predictions:    6301/  10000 (63.01%), best:    6299/10000 (62.99%), error:     10702.05273, best error:     10687.45605 on epoch:    73, epoch:   74/100, mu: 0.9884659648, learning_rate: 0.0000709080, weight_decay: 0.0000148933

epoch time: 1627.8471679688s, input_fired_time: 32.0826492310, output_fired_time: 9.8822984695, propagate_forward_time: 587.6145019531, propagate_backward_time: 979.9989013672, weight_update_time: 0.4320826232, other_time: 18.2687988281
epoch time: 120.1235198975s, input_fired_time: 2.6557569504, output_fired_time: 0.0000000000, propagate_forward_time: 112.9674530029, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5003128052
validation[          , genome  5788] predictions:    6252/  10000 (62.52%), best:    6299/10000 (62.99%), error:     10811.90234, best error:     10687.45605 on epoch:    73, epoch:   75/100, mu: 0.9885788560, learning_rate: 0.0000666730, weight_decay: 0.0000142520

epoch time: 1608.2900390625s, input_fired_time: 33.9247436523, output_fired_time: 10.8447628021, propagate_forward_time: 572.9272460938, propagate_backward_time: 962.3435668945, weight_update_time: 1.4145869017, other_time: 28.2496948242
epoch time: 122.4221191406s, input_fired_time: 2.6445734501, output_fired_time: 0.0000000000, propagate_forward_time: 116.1743316650, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6032104492
validation[          , genome  5788] predictions:    6267/  10000 (62.67%), best:    6299/10000 (62.99%), error:     10756.09473, best error:     10687.45605 on epoch:    73, epoch:   76/100, mu: 0.9886834621, learning_rate: 0.0000626909, weight_decay: 0.0000136383

epoch time: 1618.1978759766s, input_fired_time: 27.1825523376, output_fired_time: 9.8603038788, propagate_forward_time: 574.4291992188, propagate_backward_time: 984.3698120117, weight_update_time: 0.4327089190, other_time: 22.3560180664
epoch time: 122.3955612183s, input_fired_time: 2.6505732536, output_fired_time: 0.0000000000, propagate_forward_time: 116.2499008179, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4950866699
validation[          , genome  5788] predictions:    6290/  10000 (62.90%), best:    6290/10000 (62.90%), error:     10658.23535, best error:     10658.23535 on epoch:    77, epoch:   77/100, mu: 0.9887803793, learning_rate: 0.0000589466, weight_decay: 0.0000130510

epoch time: 1625.8703613281s, input_fired_time: 25.0908279419, output_fired_time: 7.7539911270, propagate_forward_time: 577.1310424805, propagate_backward_time: 983.4214477539, weight_update_time: 0.4343971014, other_time: 32.4730224609
epoch time: 122.9048919678s, input_fired_time: 0.6784228683, output_fired_time: 0.0000000000, propagate_forward_time: 115.6531219482, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5733489990
validation[          , genome  5788] predictions:    6391/  10000 (63.91%), best:    6391/10000 (63.91%), error:     10519.24316, best error:     10519.24316 on epoch:    78, epoch:   78/100, mu: 0.9888701439, learning_rate: 0.0000554260, weight_decay: 0.0000124891

epoch time: 1637.4720458984s, input_fired_time: 36.2515296936, output_fired_time: 13.9113521576, propagate_forward_time: 585.3319091797, propagate_backward_time: 977.7573852539, weight_update_time: 0.4310460091, other_time: 24.2197875977
epoch time: 122.7073669434s, input_fired_time: 1.6707046032, output_fired_time: 0.0000000000, propagate_forward_time: 114.4512176514, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5854415894
validation[          , genome  5788] predictions:    6321/  10000 (63.21%), best:    6391/10000 (63.91%), error:     10671.08887, best error:     10519.24316 on epoch:    78, epoch:   79/100, mu: 0.9889532924, learning_rate: 0.0000521156, weight_decay: 0.0000119513

epoch time: 1634.9013671875s, input_fired_time: 22.1266994476, output_fired_time: 10.8662843704, propagate_forward_time: 586.5000610352, propagate_backward_time: 993.0924682617, weight_update_time: 0.4512338936, other_time: 22.3157958984
epoch time: 119.9901123047s, input_fired_time: 1.6746025085, output_fired_time: 0.0000000000, propagate_forward_time: 113.8086700439, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5068359375
validation[          , genome  5788] predictions:    6341/  10000 (63.41%), best:    6391/10000 (63.91%), error:     10644.50781, best error:     10519.24316 on epoch:    78, epoch:   80/100, mu: 0.9890303612, learning_rate: 0.0000490030, weight_decay: 0.0000114367

epoch time: 1640.0777587891s, input_fired_time: 32.2798728943, output_fired_time: 9.9829587936, propagate_forward_time: 579.7908325195, propagate_backward_time: 998.5291748047, weight_update_time: 0.4474980533, other_time: 19.4949340820
epoch time: 120.4233932495s, input_fired_time: 2.6574361324, output_fired_time: 0.0000000000, propagate_forward_time: 114.2378387451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5281219482
validation[          , genome  5788] predictions:    6296/  10000 (62.96%), best:    6391/10000 (63.91%), error:     10680.58789, best error:     10519.24316 on epoch:    78, epoch:   81/100, mu: 0.9891017079, learning_rate: 0.0000460763, weight_decay: 0.0000109442

epoch time: 1632.8609619141s, input_fired_time: 28.1117305756, output_fired_time: 10.7305250168, propagate_forward_time: 579.1173706055, propagate_backward_time: 985.8201904297, weight_update_time: 0.4336642325, other_time: 29.0812377930
epoch time: 122.0956420898s, input_fired_time: 0.6777899265, output_fired_time: 0.0000000000, propagate_forward_time: 117.9308319092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4870223999
validation[          , genome  5788] predictions:    6268/  10000 (62.68%), best:    6391/10000 (63.91%), error:     10672.19922, best error:     10519.24316 on epoch:    78, epoch:   82/100, mu: 0.9891678095, learning_rate: 0.0000433243, weight_decay: 0.0000104729

epoch time: 1620.9069824219s, input_fired_time: 23.0660800934, output_fired_time: 8.7836618423, propagate_forward_time: 579.0263061523, propagate_backward_time: 986.6687011719, weight_update_time: 0.4398304224, other_time: 23.3622436523
epoch time: 121.3366699219s, input_fired_time: 0.6913477778, output_fired_time: 0.0000000000, propagate_forward_time: 118.0465927124, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5987319946
validation[          , genome  5788] predictions:    6312/  10000 (63.12%), best:    6391/10000 (63.91%), error:     10579.01563, best error:     10519.24316 on epoch:    78, epoch:   83/100, mu: 0.9892290831, learning_rate: 0.0000407368, weight_decay: 0.0000100220

epoch time: 1628.4796142578s, input_fired_time: 25.8962688446, output_fired_time: 9.7159824371, propagate_forward_time: 583.7468261719, propagate_backward_time: 989.8314819336, weight_update_time: 0.4371471405, other_time: 19.2891235352
epoch time: 121.1370620728s, input_fired_time: 5.6037940979, output_fired_time: 0.0000000000, propagate_forward_time: 110.9780502319, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5552139282
validation[          , genome  5788] predictions:    6337/  10000 (63.37%), best:    6391/10000 (63.91%), error:     10562.66309, best error:     10519.24316 on epoch:    78, epoch:   84/100, mu: 0.9892858267, learning_rate: 0.0000383037, weight_decay: 0.0000095904

epoch time: 1625.3271484375s, input_fired_time: 31.1284084320, output_fired_time: 14.7626571655, propagate_forward_time: 564.1069335938, propagate_backward_time: 986.7908935547, weight_update_time: 3.5100390911, other_time: 28.5382080078
epoch time: 122.1884994507s, input_fired_time: 2.6545975208, output_fired_time: 0.0000000000, propagate_forward_time: 116.0236129761, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5102920532
validation[          , genome  5788] predictions:    6369/  10000 (63.69%), best:    6391/10000 (63.91%), error:     10546.62891, best error:     10519.24316 on epoch:    78, epoch:   85/100, mu: 0.9893383980, learning_rate: 0.0000360160, weight_decay: 0.0000091775

epoch time: 1628.5412597656s, input_fired_time: 21.0064716339, output_fired_time: 8.8984270096, propagate_forward_time: 574.5084228516, propagate_backward_time: 994.5258178711, weight_update_time: 4.5947785378, other_time: 29.6021118164
epoch time: 119.9761123657s, input_fired_time: 1.6787312031, output_fired_time: 0.0000000000, propagate_forward_time: 116.7615737915, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5358047485
validation[          , genome  5788] predictions:    6361/  10000 (63.61%), best:    6391/10000 (63.91%), error:     10536.93750, best error:     10519.24316 on epoch:    78, epoch:   86/100, mu: 0.9893870950, learning_rate: 0.0000338649, weight_decay: 0.0000087823

epoch time: 1627.3752441406s, input_fired_time: 25.0307312012, output_fired_time: 9.8902120590, propagate_forward_time: 590.1513671875, propagate_backward_time: 981.7611694336, weight_update_time: 1.5334885120, other_time: 20.5416870117
epoch time: 124.6131744385s, input_fired_time: 0.6804072261, output_fired_time: 0.0000000000, propagate_forward_time: 119.3465881348, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5861816406
validation[          , genome  5788] predictions:    6370/  10000 (63.70%), best:    6370/10000 (63.70%), error:     10463.42969, best error:     10463.42969 on epoch:    87, epoch:   87/100, mu: 0.9894322157, learning_rate: 0.0000318423, weight_decay: 0.0000084041

epoch time: 1624.9323730469s, input_fired_time: 26.1211833954, output_fired_time: 8.9117851257, propagate_forward_time: 576.9271240234, propagate_backward_time: 978.6609497070, weight_update_time: 2.3998918533, other_time: 34.3113403320
epoch time: 120.3286056519s, input_fired_time: 2.6597788334, output_fired_time: 0.0000000000, propagate_forward_time: 115.0942840576, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5745391846
validation[          , genome  5788] predictions:    6363/  10000 (63.63%), best:    6370/10000 (63.70%), error:     10497.43848, best error:     10463.42969 on epoch:    87, epoch:   88/100, mu: 0.9894739985, learning_rate: 0.0000299405, weight_decay: 0.0000080422

epoch time: 1609.9307861328s, input_fired_time: 31.9919357300, output_fired_time: 10.6867761612, propagate_forward_time: 560.2539672852, propagate_backward_time: 985.6297607422, weight_update_time: 1.4184327126, other_time: 21.3683471680
epoch time: 121.1841583252s, input_fired_time: 4.8393964767, output_fired_time: 0.0000000000, propagate_forward_time: 109.8728332520, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4719314575
validation[          , genome  5788] predictions:    6352/  10000 (63.52%), best:    6370/10000 (63.70%), error:     10536.86426, best error:     10463.42969 on epoch:    87, epoch:   89/100, mu: 0.9895127416, learning_rate: 0.0000281523, weight_decay: 0.0000076959

epoch time: 1607.5072021484s, input_fired_time: 25.0464439392, output_fired_time: 8.9541702271, propagate_forward_time: 569.1811523438, propagate_backward_time: 980.1326293945, weight_update_time: 0.4364013970, other_time: 24.1928100586
epoch time: 122.2993927002s, input_fired_time: 3.6370263100, output_fired_time: 0.0000000000, propagate_forward_time: 113.1373825073, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5249862671
validation[          , genome  5788] predictions:    6390/  10000 (63.90%), best:    6370/10000 (63.70%), error:     10479.15039, best error:     10463.42969 on epoch:    87, epoch:   90/100, mu: 0.9895486236, learning_rate: 0.0000264709, weight_decay: 0.0000073646

epoch time: 1613.3645019531s, input_fired_time: 26.0361480713, output_fired_time: 7.8385505676, propagate_forward_time: 563.4454345703, propagate_backward_time: 994.6591796875, weight_update_time: 2.5173273087, other_time: 21.3852539063
epoch time: 119.7341384888s, input_fired_time: 0.6792964339, output_fired_time: 0.0000000000, propagate_forward_time: 112.3794784546, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6753616333
validation[          , genome  5788] predictions:    6325/  10000 (63.25%), best:    6370/10000 (63.70%), error:     10579.75000, best error:     10463.42969 on epoch:    87, epoch:   91/100, mu: 0.9895818233, learning_rate: 0.0000248899, weight_decay: 0.0000070474

epoch time: 1604.2923583984s, input_fired_time: 24.9728107452, output_fired_time: 12.7960948944, propagate_forward_time: 572.2562866211, propagate_backward_time: 970.9752807617, weight_update_time: 0.4370759726, other_time: 23.2918701172
epoch time: 119.9378738403s, input_fired_time: 4.6203045845, output_fired_time: 0.0000000000, propagate_forward_time: 111.7443389893, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5732269287
validation[          , genome  5788] predictions:    6391/  10000 (63.91%), best:    6370/10000 (63.70%), error:     10464.64941, best error:     10463.42969 on epoch:    87, epoch:   92/100, mu: 0.9896125793, learning_rate: 0.0000234033, weight_decay: 0.0000067440

epoch time: 1622.0123291016s, input_fired_time: 18.1441898346, output_fired_time: 7.8220696449, propagate_forward_time: 596.9260864258, propagate_backward_time: 978.8390502930, weight_update_time: 0.4419609606, other_time: 20.2810058594
epoch time: 122.0020675659s, input_fired_time: 0.6784027219, output_fired_time: 0.0000000000, propagate_forward_time: 116.8535003662, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4701614380
validation[          , genome  5788] predictions:    6344/  10000 (63.44%), best:    6370/10000 (63.70%), error:     10517.57813, best error:     10463.42969 on epoch:    87, epoch:   93/100, mu: 0.9896410704, learning_rate: 0.0000220056, weight_decay: 0.0000064536

epoch time: 1607.0255126953s, input_fired_time: 32.8625907898, output_fired_time: 12.8725662231, propagate_forward_time: 578.1688232422, propagate_backward_time: 960.8657836914, weight_update_time: 0.4331404269, other_time: 22.2557983398
epoch time: 117.3943099976s, input_fired_time: 1.6696140766, output_fired_time: 0.0000000000, propagate_forward_time: 114.2076034546, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5170898438
validation[          , genome  5788] predictions:    6386/  10000 (63.86%), best:    6386/10000 (63.86%), error:     10442.68457, best error:     10442.68457 on epoch:    94, epoch:   94/100, mu: 0.9896674752, learning_rate: 0.0000206913, weight_decay: 0.0000061757

epoch time: 1606.8647460938s, input_fired_time: 26.8621921539, output_fired_time: 12.7930994034, propagate_forward_time: 575.4159545898, propagate_backward_time: 964.6326904297, weight_update_time: 0.4357972741, other_time: 27.1608276367
epoch time: 120.3582992554s, input_fired_time: 0.6869839430, output_fired_time: 0.0000000000, propagate_forward_time: 118.1531600952, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5181579590
validation[          , genome  5788] predictions:    6367/  10000 (63.67%), best:    6367/10000 (63.67%), error:     10438.50879, best error:     10438.50879 on epoch:    95, epoch:   95/100, mu: 0.9896919727, learning_rate: 0.0000194555, weight_decay: 0.0000059098

epoch time: 1614.6918945313s, input_fired_time: 24.0807189941, output_fired_time: 11.9330539703, propagate_forward_time: 576.9871215820, propagate_backward_time: 985.5675048828, weight_update_time: 0.4339983761, other_time: 16.1234741211
epoch time: 118.5203704834s, input_fired_time: 4.6291217804, output_fired_time: 0.0000000000, propagate_forward_time: 110.3849105835, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5063400269
validation[          , genome  5788] predictions:    6381/  10000 (63.81%), best:    6367/10000 (63.67%), error:     10451.48828, best error:     10438.50879 on epoch:    95, epoch:   96/100, mu: 0.9897146225, learning_rate: 0.0000182935, weight_decay: 0.0000056553

epoch time: 1637.8872070313s, input_fired_time: 25.0663452148, output_fired_time: 8.8395881653, propagate_forward_time: 598.6937866211, propagate_backward_time: 978.9055175781, weight_update_time: 3.5029804707, other_time: 26.3818969727
epoch time: 122.9192352295s, input_fired_time: 4.6357998848, output_fired_time: 0.0000000000, propagate_forward_time: 111.8216857910, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4617462158
validation[          , genome  5788] predictions:    6377/  10000 (63.77%), best:    6367/10000 (63.67%), error:     10466.69922, best error:     10438.50879 on epoch:    95, epoch:   97/100, mu: 0.9897356033, learning_rate: 0.0000172009, weight_decay: 0.0000054118

epoch time: 1640.1295166016s, input_fired_time: 31.9329299927, output_fired_time: 10.7759151459, propagate_forward_time: 572.5537719727, propagate_backward_time: 997.3693847656, weight_update_time: 2.5201373100, other_time: 27.4975585938
epoch time: 124.9372787476s, input_fired_time: 3.7616589069, output_fired_time: 0.0000000000, propagate_forward_time: 112.3838500977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.7917709351
validation[          , genome  5788] predictions:    6355/  10000 (63.55%), best:    6367/10000 (63.67%), error:     10456.37500, best error:     10438.50879 on epoch:    95, epoch:   98/100, mu: 0.9897550941, learning_rate: 0.0000161735, weight_decay: 0.0000051787

epoch time: 1639.8088378906s, input_fired_time: 23.3066616058, output_fired_time: 10.7342824936, propagate_forward_time: 587.1185913086, propagate_backward_time: 992.3540039063, weight_update_time: 1.4305557013, other_time: 26.2953491211
epoch time: 119.9442291260s, input_fired_time: 3.6490988731, output_fired_time: 0.0000000000, propagate_forward_time: 109.8121719360, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4829559326
validation[          , genome  5788] predictions:    6398/  10000 (63.98%), best:    6367/10000 (63.67%), error:     10474.22266, best error:     10438.50879 on epoch:    95, epoch:   99/100, mu: 0.9897730947, learning_rate: 0.0000152076, weight_decay: 0.0000049557

epoch time: 1632.7355957031s, input_fired_time: 28.0212726593, output_fired_time: 12.7437801361, propagate_forward_time: 577.7416992188, propagate_backward_time: 989.6709594727, weight_update_time: 0.4466499984, other_time: 24.5579223633
epoch time: 125.9496231079s, input_fired_time: 1.6824214458, output_fired_time: 0.0000000000, propagate_forward_time: 121.7556915283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5115127563
validation[          , genome  5788] predictions:    6350/  10000 (63.50%), best:    6367/10000 (63.67%), error:     10500.27246, best error:     10438.50879 on epoch:    95, epoch:  100/100, mu: 0.9897897840, learning_rate: 0.0000142993, weight_decay: 0.0000047423

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 826.6431884766s, input_fired_time: 15.6865987778, output_fired_time: 0.0000000000, propagate_forward_time: 624.1049194336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 186.8516845703
best training[          , genome  5788] predictions:   36896/  50000 (73.79%), best:    6367/10000 (63.67%), error:     39363.84375, best error:     10438.50879 on epoch:    95, epoch:  101/100, mu: 0.9898052812, learning_rate: 0.0000134453, weight_decay: 0.0000045381
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 145.0372772217s, input_fired_time: 0.7167004943, output_fired_time: 0.0000000000, propagate_forward_time: 121.7035751343, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 22.6169967651
   testing[          , genome  5788] predictions:    6367/  10000 (63.67%), best:    6367/10000 (63.67%), error:     10438.50879, best error:     10438.50879 on epoch:    95, epoch:  101/100, mu: 0.9898052812, learning_rate: 0.0000134453, weight_decay: 0.0000045381
backpropagation finished successfully!
21:43:29 (3664): called boinc_finish(0)

</stderr_txt>
]]>