Nameexact_genome_1534504202_164_10949_2
Workunit3916582
Created25 Aug 2018, 14:39:05 UTC
Sent25 Aug 2018, 16:10:09 UTC
Report deadline23 Jan 2019, 16:28:12 UTC
Received30 Aug 2018, 20:12:14 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60269
Run time2 days 13 hours 36 min 18 sec
CPU time2 days 12 hours 29 min 14 sec
Validate stateInitial
Credit0.00
Device peak FLOPS4.38 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size562.16 MB
Peak swap size582.06 MB
Peak disk usage14.10 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
 with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1901.76940917969s, input_fired_time: 23.2164821624756, output_fired_time: 9.51291084289551, propagate_forward_time: 742.740600585938, propagate_backward_time: 1112.20007324219, weight_update_time: 0.517242431640625, other_time: 14.099365234375
epoch time: 152.736755371094s, input_fired_time: 1.7263697385788, output_fired_time: 0, propagate_forward_time: 148.550445556641, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.45994567871094
validation[          , genome 10949] predictions:    6546/  10000 (65.46%), best:    6618/10000 (66.18%), error:     10076.04492, best error:      9799.39063 on epoch:    35, epoch:   37/100, mu: 0.9282016754, learning_rate: 0.0000935584, weight_decay: 0.0000598230

epoch time: 1905.9509277344s, input_fired_time: 23.2579879761, output_fired_time: 9.5755977631, propagate_forward_time: 744.3729858398, propagate_backward_time: 1114.8334960938, weight_update_time: 0.5086828470, other_time: 13.9108886719
epoch time: 154.5955352783s, input_fired_time: 1.7555598021, output_fired_time: 0.0000000000, propagate_forward_time: 150.4518432617, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3881378174
validation[          , genome 10949] predictions:    6606/  10000 (66.06%), best:    6606/10000 (66.06%), error:      9759.41895, best error:      9759.41895 on epoch:    38, epoch:   38/100, mu: 0.9316853285, learning_rate: 0.0000865001, weight_decay: 0.0000563523

epoch time: 1905.1325683594s, input_fired_time: 23.3523674011, output_fired_time: 9.6297731400, propagate_forward_time: 746.6187744141, propagate_backward_time: 1111.5629882813, weight_update_time: 0.5170994997, other_time: 13.9686279297
epoch time: 156.4217681885s, input_fired_time: 1.7588722706, output_fired_time: 0.0000000000, propagate_forward_time: 152.1584320068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5044708252
validation[          , genome 10949] predictions:    6560/  10000 (65.60%), best:    6606/10000 (66.06%), error:      9837.37988, best error:      9759.41895 on epoch:    38, epoch:   39/100, mu: 0.9349726439, learning_rate: 0.0000799744, weight_decay: 0.0000530828

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949_2_r317080869_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.461141
read mu: 0.938075
read mu_delta: 0.943628
read initial_learning_rate: 0.00170423
read learning_rate: 7.39409e-05
read learning_rate_delta: 0.924557
read initial_weight_decay: 0.000546113
read weight_decay: 5.00031e-05
read weight_decay_delta: 0.941983
read batch_size: 85
read epsilon: 1e-07
read alpha: 0.0775803
read input_dropout_probability: 0.00131918
read hidden_dropout_probability: 0.0822247
read velocity_reset: 2235
read epoch: 40
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 38
read number_validation_images: 10000
read best_validation_predictions: 6606
read best_validation_error: 9759.42
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 10949
read normal distribution: '1 0 0'
generator_str: '821621940'
read generator: 821621940
read generated_by_map:
1 disable_edge 1
reading 124 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1878.61303710938s, input_fired_time: 23.3053741455078, output_fired_time: 9.45805931091309, propagate_forward_time: 732.613525390625, propagate_backward_time: 1098.27795410156, weight_update_time: 0.507908940315247, other_time: 14.9581298828125
epoch time: 152.174514770508s, input_fired_time: 1.79063665866852, output_fired_time: 0, propagate_forward_time: 147.866836547852, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.51704406738281
validation[          , genome 10949] predictions:    6653/  10000 (66.53%), best:    6653/10000 (66.53%), error:      9676.43066, best error:      9676.43066 on epoch:    40, epoch:   40/100, mu: 0.9380746484, learning_rate: 0.0000739409, weight_decay: 0.0000500031

epoch time: 1850.2111816406s, input_fired_time: 23.2278747559, output_fired_time: 9.4597358704, propagate_forward_time: 720.5963745117, propagate_backward_time: 1082.3303222656, weight_update_time: 0.5054739118, other_time: 14.5969238281
epoch time: 146.9776153564s, input_fired_time: 1.7205069065, output_fired_time: 0.0000000000, propagate_forward_time: 142.7520599365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5050506592
validation[          , genome 10949] predictions:    6639/  10000 (66.39%), best:    6653/10000 (66.53%), error:      9759.45410, best error:      9676.43066 on epoch:    40, epoch:   41/100, mu: 0.9410017729, learning_rate: 0.0000683626, weight_decay: 0.0000471021

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949_2_r317080869_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.461141
read mu: 0.943764
read mu_delta: 0.943628
read initial_learning_rate: 0.00170423
read learning_rate: 6.32051e-05
read learning_rate_delta: 0.924557
read initial_weight_decay: 0.000546113
read weight_decay: 4.43693e-05
read weight_decay_delta: 0.941983
read batch_size: 85
read epsilon: 1e-07
read alpha: 0.0775803
read input_dropout_probability: 0.00131918
read hidden_dropout_probability: 0.0822247
read velocity_reset: 2235
read epoch: 42
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 40
read number_validation_images: 10000
read best_validation_predictions: 6653
read best_validation_error: 9676.43
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 10949
read normal distribution: '1 0 0'
generator_str: '1661023335'
read generator: 1661023335
read generated_by_map:
1 disable_edge 1
reading 124 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1793.34033203125s, input_fired_time: 22.9519958496094, output_fired_time: 9.15652656555176, propagate_forward_time: 696.835876464844, propagate_backward_time: 1049.0595703125, weight_update_time: 0.49726465344429, other_time: 15.33642578125
epoch time: 144.814590454102s, input_fired_time: 1.70967650413513, output_fired_time: 0, propagate_forward_time: 140.244552612305, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.86036682128906
validation[          , genome 10949] predictions:    6560/  10000 (65.60%), best:    6653/10000 (66.53%), error:      9846.91309, best error:      9676.43066 on epoch:    40, epoch:   42/100, mu: 0.9437639117, learning_rate: 0.0000632051, weight_decay: 0.0000443693

epoch time: 1793.0506591797s, input_fired_time: 22.8615531921, output_fired_time: 9.2028398514, propagate_forward_time: 696.3481445313, propagate_backward_time: 1049.0236816406, weight_update_time: 0.4987607896, other_time: 15.6143798828
epoch time: 145.5303802490s, input_fired_time: 1.7045454979, output_fired_time: 0.0000000000, propagate_forward_time: 141.0552673340, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7705688477
validation[          , genome 10949] predictions:    6697/  10000 (66.97%), best:    6697/10000 (66.97%), error:      9583.45605, best error:      9583.45605 on epoch:    43, epoch:   43/100, mu: 0.9463703036, learning_rate: 0.0000584368, weight_decay: 0.0000417951

epoch time: 1846.8178710938s, input_fired_time: 22.9436664581, output_fired_time: 9.3995246887, propagate_forward_time: 718.6508789063, propagate_backward_time: 1080.0520019531, weight_update_time: 0.5022036433, other_time: 15.7717285156
epoch time: 150.3898925781s, input_fired_time: 1.7163270712, output_fired_time: 0.0000000000, propagate_forward_time: 145.8018646240, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8717041016
validation[          , genome 10949] predictions:    6682/  10000 (66.82%), best:    6697/10000 (66.97%), error:      9642.44922, best error:      9583.45605 on epoch:    43, epoch:   44/100, mu: 0.9488297701, learning_rate: 0.0000540282, weight_decay: 0.0000393703

epoch time: 1905.2478027344s, input_fired_time: 23.0627079010, output_fired_time: 9.6425895691, propagate_forward_time: 739.8504638672, propagate_backward_time: 1116.3339843750, weight_update_time: 0.5028945208, other_time: 16.3580322266
epoch time: 151.7110137939s, input_fired_time: 1.7335423231, output_fired_time: 0.0000000000, propagate_forward_time: 147.0955963135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8818817139
validation[          , genome 10949] predictions:    6676/  10000 (66.76%), best:    6697/10000 (66.97%), error:      9693.33984, best error:      9583.45605 on epoch:    43, epoch:   45/100, mu: 0.9511505961, learning_rate: 0.0000499521, weight_decay: 0.0000370861

epoch time: 1860.8026123047s, input_fired_time: 23.0597820282, output_fired_time: 9.4241800308, propagate_forward_time: 722.2770385742, propagate_backward_time: 1090.0910644531, weight_update_time: 0.5028675199, other_time: 15.9504394531
epoch time: 149.4677581787s, input_fired_time: 1.7277662754, output_fired_time: 0.0000000000, propagate_forward_time: 144.8359375000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9040527344
validation[          , genome 10949] predictions:    6631/  10000 (66.31%), best:    6697/10000 (66.97%), error:      9730.49805, best error:      9583.45605 on epoch:    43, epoch:   46/100, mu: 0.9533405900, learning_rate: 0.0000461836, weight_decay: 0.0000349345

epoch time: 1877.2487792969s, input_fired_time: 23.0070056915, output_fired_time: 9.5186691284, propagate_forward_time: 729.2002563477, propagate_backward_time: 1099.2857666016, weight_update_time: 0.5023189783, other_time: 16.2371826172
epoch time: 149.7155151367s, input_fired_time: 1.7267360687, output_fired_time: 0.0000000000, propagate_forward_time: 145.2953796387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6934051514
validation[          , genome 10949] predictions:    6701/  10000 (67.01%), best:    6697/10000 (66.97%), error:      9673.82617, best error:      9583.45605 on epoch:    43, epoch:   47/100, mu: 0.9554071426, learning_rate: 0.0000426994, weight_decay: 0.0000329077

epoch time: 1883.3714599609s, input_fired_time: 23.0458354950, output_fired_time: 9.5023088455, propagate_forward_time: 731.4497070313, propagate_backward_time: 1102.9831542969, weight_update_time: 0.4975507557, other_time: 16.3905029297
epoch time: 154.6080322266s, input_fired_time: 1.7571564913, output_fired_time: 0.0000000000, propagate_forward_time: 149.8725585938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9783172607
validation[          , genome 10949] predictions:    6689/  10000 (66.89%), best:    6689/10000 (66.89%), error:      9543.99609, best error:      9543.99609 on epoch:    48, epoch:   48/100, mu: 0.9573572278, learning_rate: 0.0000394781, weight_decay: 0.0000309984

epoch time: 1969.2000732422s, input_fired_time: 23.0430068970, output_fired_time: 9.8112726212, propagate_forward_time: 761.4846801758, propagate_backward_time: 1157.7757568359, weight_update_time: 0.5003762841, other_time: 17.0853271484
epoch time: 160.9816284180s, input_fired_time: 1.7565714121, output_fired_time: 0.0000000000, propagate_forward_time: 156.1404724121, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0845794678
validation[          , genome 10949] predictions:    6729/  10000 (67.29%), best:    6689/10000 (66.89%), error:      9639.48047, best error:      9543.99609 on epoch:    48, epoch:   49/100, mu: 0.9591973424, learning_rate: 0.0000364997, weight_decay: 0.0000292000

epoch time: 1997.3319091797s, input_fired_time: 23.1175231934, output_fired_time: 9.9209804535, propagate_forward_time: 771.3604736328, propagate_backward_time: 1175.6262207031, weight_update_time: 0.5011482835, other_time: 17.3066406250
epoch time: 159.4360809326s, input_fired_time: 1.7785468102, output_fired_time: 0.0000000000, propagate_forward_time: 154.6286010742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0289306641
validation[          , genome 10949] predictions:    6729/  10000 (67.29%), best:    6689/10000 (66.89%), error:      9603.02734, best error:      9543.99609 on epoch:    48, epoch:   50/100, mu: 0.9609337449, learning_rate: 0.0000337461, weight_decay: 0.0000275059

epoch time: 2007.7221679688s, input_fired_time: 23.1021881104, output_fired_time: 9.9282083511, propagate_forward_time: 778.0516357422, propagate_backward_time: 1179.5098876953, weight_update_time: 0.5040100217, other_time: 17.1302490234
epoch time: 158.1890258789s, input_fired_time: 1.7482478619, output_fired_time: 0.0000000000, propagate_forward_time: 153.3544769287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0863037109
validation[          , genome 10949] predictions:    6744/  10000 (67.44%), best:    6744/10000 (67.44%), error:      9346.76953, best error:      9346.76953 on epoch:    51, epoch:   51/100, mu: 0.9625722766, learning_rate: 0.0000312002, weight_decay: 0.0000259101

epoch time: 2015.0233154297s, input_fired_time: 23.1304473877, output_fired_time: 9.9424686432, propagate_forward_time: 779.1659545898, propagate_backward_time: 1185.1630859375, weight_update_time: 0.4978514910, other_time: 17.6213378906
epoch time: 162.4619293213s, input_fired_time: 1.7773925066, output_fired_time: 0.0000000000, propagate_forward_time: 157.6538848877, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0306549072
validation[          , genome 10949] predictions:    6716/  10000 (67.16%), best:    6744/10000 (67.44%), error:      9522.53809, best error:      9346.76953 on epoch:    51, epoch:   52/100, mu: 0.9641184211, learning_rate: 0.0000288464, weight_decay: 0.0000244068

epoch time: 2051.3908691406s, input_fired_time: 23.7078380585, output_fired_time: 10.1617574692, propagate_forward_time: 795.2253417969, propagate_backward_time: 1204.6663818359, weight_update_time: 0.5218475461, other_time: 17.6295166016
epoch time: 161.0757904053s, input_fired_time: 1.7894010544, output_fired_time: 0.0000000000, propagate_forward_time: 156.2135314941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0728607178
validation[          , genome 10949] predictions:    6761/  10000 (67.61%), best:    6744/10000 (67.44%), error:      9428.04297, best error:      9346.76953 on epoch:    51, epoch:   53/100, mu: 0.9655774236, learning_rate: 0.0000266701, weight_decay: 0.0000229908

epoch time: 2024.4089355469s, input_fired_time: 23.3154048920, output_fired_time: 10.0428361893, propagate_forward_time: 785.6193847656, propagate_backward_time: 1188.0469970703, weight_update_time: 0.5070459843, other_time: 17.3842773438
epoch time: 165.0252838135s, input_fired_time: 1.7991125584, output_fired_time: 0.0000000000, propagate_forward_time: 160.1531677246, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0729980469
validation[          , genome 10949] predictions:    6820/  10000 (68.20%), best:    6820/10000 (68.20%), error:      9309.78320, best error:      9309.78320 on epoch:    54, epoch:   54/100, mu: 0.9669541717, learning_rate: 0.0000246581, weight_decay: 0.0000216569

epoch time: 2000.5988769531s, input_fired_time: 23.1538867950, output_fired_time: 9.8606185913, propagate_forward_time: 769.6657714844, propagate_backward_time: 1180.0889892578, weight_update_time: 0.4989865124, other_time: 17.8295898438
epoch time: 160.0919189453s, input_fired_time: 1.7807358503, output_fired_time: 0.0000000000, propagate_forward_time: 155.0695648193, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2416229248
validation[          , genome 10949] predictions:    6802/  10000 (68.02%), best:    6820/10000 (68.20%), error:      9349.83008, best error:      9309.78320 on epoch:    54, epoch:   55/100, mu: 0.9682533145, learning_rate: 0.0000227978, weight_decay: 0.0000204005

epoch time: 2024.6538085938s, input_fired_time: 23.2603130341, output_fired_time: 9.9559555054, propagate_forward_time: 776.0779418945, propagate_backward_time: 1197.1008300781, weight_update_time: 0.5064333677, other_time: 18.2587890625
epoch time: 160.0578765869s, input_fired_time: 1.7995717525, output_fired_time: 0.0000000000, propagate_forward_time: 154.9073486328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3509521484
validation[          , genome 10949] predictions:    6824/  10000 (68.24%), best:    6824/10000 (68.24%), error:      9253.18359, best error:      9253.18359 on epoch:    56, epoch:   56/100, mu: 0.9694792032, learning_rate: 0.0000210779, weight_decay: 0.0000192169

epoch time: 2115.6860351563s, input_fired_time: 24.4526405334, output_fired_time: 10.4401597977, propagate_forward_time: 836.7201538086, propagate_backward_time: 1226.4458007813, weight_update_time: 0.5303533673, other_time: 17.6271972656
epoch time: 187.8478088379s, input_fired_time: 1.9672260284, output_fired_time: 0.0000000000, propagate_forward_time: 182.7011718750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1794128418
validation[          , genome 10949] predictions:    6721/  10000 (67.21%), best:    6824/10000 (68.24%), error:      9559.37012, best error:      9253.18359 on epoch:    56, epoch:   57/100, mu: 0.9706360102, learning_rate: 0.0000194877, weight_decay: 0.0000181020

epoch time: 2253.0283203125s, input_fired_time: 25.8442344666, output_fired_time: 10.9853124619, propagate_forward_time: 917.3387451172, propagate_backward_time: 1281.6337890625, weight_update_time: 0.5506719351, other_time: 17.2261962891
epoch time: 187.7063903809s, input_fired_time: 2.0054881573, output_fired_time: 0.0000000000, propagate_forward_time: 182.5721282959, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1287689209
validation[          , genome 10949] predictions:    6792/  10000 (67.92%), best:    6824/10000 (68.24%), error:      9347.50586, best error:      9253.18359 on epoch:    56, epoch:   58/100, mu: 0.9717276096, learning_rate: 0.0000180175, weight_decay: 0.0000170517

epoch time: 2208.9204101563s, input_fired_time: 25.8464336395, output_fired_time: 10.7572965622, propagate_forward_time: 897.0427856445, propagate_backward_time: 1258.4020996094, weight_update_time: 0.5684766173, other_time: 16.8718261719
epoch time: 181.9613342285s, input_fired_time: 1.9330304861, output_fired_time: 0.0000000000, propagate_forward_time: 177.1924438477, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8358612061
validation[          , genome 10949] predictions:    6783/  10000 (67.83%), best:    6824/10000 (68.24%), error:      9344.54004, best error:      9253.18359 on epoch:    56, epoch:   59/100, mu: 0.9727576375, learning_rate: 0.0000166582, weight_decay: 0.0000160624

epoch time: 2154.5703125000s, input_fired_time: 25.2359390259, output_fired_time: 10.5293054581, propagate_forward_time: 874.8937377930, propagate_backward_time: 1227.1815185547, weight_update_time: 0.5463872552, other_time: 16.7298583984
epoch time: 190.4405364990s, input_fired_time: 2.0421774387, output_fired_time: 0.0000000000, propagate_forward_time: 185.4998168945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8985443115
validation[          , genome 10949] predictions:    6811/  10000 (68.11%), best:    6824/10000 (68.24%), error:      9253.89551, best error:      9253.18359 on epoch:    56, epoch:   60/100, mu: 0.9737296104, learning_rate: 0.0000154015, weight_decay: 0.0000151305

epoch time: 2044.9030761719s, input_fired_time: 23.6913414001, output_fired_time: 10.0474281311, propagate_forward_time: 798.9989013672, propagate_backward_time: 1195.0335693359, weight_update_time: 0.5396590829, other_time: 17.1318359375
epoch time: 155.9233703613s, input_fired_time: 1.7619467974, output_fired_time: 0.0000000000, propagate_forward_time: 150.9660797119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1953430176
validation[          , genome 10949] predictions:    6746/  10000 (67.46%), best:    6824/10000 (68.24%), error:      9392.41309, best error:      9253.18359 on epoch:    56, epoch:   61/100, mu: 0.9746468067, learning_rate: 0.0000142396, weight_decay: 0.0000142527

epoch time: 1949.0147705078s, input_fired_time: 23.0629158020, output_fired_time: 9.7009563446, propagate_forward_time: 747.8243408203, propagate_backward_time: 1150.7589111328, weight_update_time: 0.5010809302, other_time: 17.6677246094
epoch time: 156.1779785156s, input_fired_time: 1.7739286423, output_fired_time: 0.0000000000, propagate_forward_time: 151.1622467041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2418060303
validation[          , genome 10949] predictions:    6831/  10000 (68.31%), best:    6831/10000 (68.31%), error:      9217.09375, best error:      9217.09375 on epoch:    62, epoch:   62/100, mu: 0.9755122662, learning_rate: 0.0000131653, weight_decay: 0.0000134258

epoch time: 1939.1690673828s, input_fired_time: 23.0344219208, output_fired_time: 9.6563405991, propagate_forward_time: 743.1665649414, propagate_backward_time: 1145.8182373047, weight_update_time: 0.4979121983, other_time: 17.4935302734
epoch time: 153.1177062988s, input_fired_time: 1.7709947824, output_fired_time: 0.0000000000, propagate_forward_time: 148.2560577393, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0906524658
validation[          , genome 10949] predictions:    6834/  10000 (68.34%), best:    6831/10000 (68.31%), error:      9232.06543, best error:      9217.09375 on epoch:    62, epoch:   63/100, mu: 0.9763289690, learning_rate: 0.0000121721, weight_decay: 0.0000126469

epoch time: 1919.8220214844s, input_fired_time: 23.0267696381, output_fired_time: 9.5515174866, propagate_forward_time: 738.7393798828, propagate_backward_time: 1131.3232421875, weight_update_time: 0.5072009563, other_time: 17.1811523438
epoch time: 152.5204925537s, input_fired_time: 1.7591203451, output_fired_time: 0.0000000000, propagate_forward_time: 147.6342620850, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1271057129
validation[          , genome 10949] predictions:    6835/  10000 (68.35%), best:    6835/10000 (68.35%), error:      9191.04883, best error:      9191.04883 on epoch:    64, epoch:   64/100, mu: 0.9770996571, learning_rate: 0.0000112538, weight_decay: 0.0000119131

epoch time: 1908.6480712891s, input_fired_time: 22.9740886688, output_fired_time: 9.5640516281, propagate_forward_time: 732.1394653320, propagate_backward_time: 1126.5296630859, weight_update_time: 0.4938667715, other_time: 17.4407958984
epoch time: 150.6555480957s, input_fired_time: 1.7395424843, output_fired_time: 0.0000000000, propagate_forward_time: 145.8431396484, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0728607178
validation[          , genome 10949] predictions:    6791/  10000 (67.91%), best:    6835/10000 (68.35%), error:      9270.55762, best error:      9191.04883 on epoch:    64, epoch:   65/100, mu: 0.9778268933, learning_rate: 0.0000104048, weight_decay: 0.0000112220

epoch time: 1925.6708984375s, input_fired_time: 23.0112304688, output_fired_time: 9.5509910583, propagate_forward_time: 739.1912231445, propagate_backward_time: 1136.1981201172, weight_update_time: 0.5058070421, other_time: 17.7193603516
epoch time: 154.5104217529s, input_fired_time: 1.7590857744, output_fired_time: 0.0000000000, propagate_forward_time: 149.4619445801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2893981934
validation[          , genome 10949] predictions:    6802/  10000 (68.02%), best:    6835/10000 (68.35%), error:      9201.06934, best error:      9191.04883 on epoch:    64, epoch:   66/100, mu: 0.9785131216, learning_rate: 0.0000096198, weight_decay: 0.0000105709

epoch time: 1952.0158691406s, input_fired_time: 23.0413417816, output_fired_time: 9.7007074356, propagate_forward_time: 749.1792602539, propagate_backward_time: 1152.0106201172, weight_update_time: 0.4946582317, other_time: 18.0838623047
epoch time: 154.7883911133s, input_fired_time: 1.7688945532, output_fired_time: 0.0000000000, propagate_forward_time: 149.8677673340, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1517333984
validation[          , genome 10949] predictions:    6836/  10000 (68.36%), best:    6836/10000 (68.36%), error:      9183.57324, best error:      9183.57324 on epoch:    67, epoch:   67/100, mu: 0.9791606665, learning_rate: 0.0000088941, weight_decay: 0.0000099576

epoch time: 1928.3569335938s, input_fired_time: 22.9861431122, output_fired_time: 9.5324401855, propagate_forward_time: 735.5434570313, propagate_backward_time: 1142.6853027344, weight_update_time: 0.4968914092, other_time: 17.6096191406
epoch time: 153.7779388428s, input_fired_time: 1.7516244650, output_fired_time: 0.0000000000, propagate_forward_time: 148.7601165771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2662048340
validation[          , genome 10949] predictions:    6844/  10000 (68.44%), best:    6836/10000 (68.36%), error:      9197.26074, best error:      9183.57324 on epoch:    67, epoch:   68/100, mu: 0.9797716737, learning_rate: 0.0000082231, weight_decay: 0.0000093799

epoch time: 1989.2847900391s, input_fired_time: 23.0564594269, output_fired_time: 9.7696275711, propagate_forward_time: 757.0417480469, propagate_backward_time: 1181.1925048828, weight_update_time: 0.4969311655, other_time: 18.2243652344
epoch time: 156.6508026123s, input_fired_time: 1.7710987329, output_fired_time: 0.0000000000, propagate_forward_time: 151.4418334961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4378662109
validation[          , genome 10949] predictions:    6848/  10000 (68.48%), best:    6848/10000 (68.48%), error:      9143.32031, best error:      9143.32031 on epoch:    69, epoch:   69/100, mu: 0.9803482890, learning_rate: 0.0000076027, weight_decay: 0.0000088357

epoch time: 1981.8807373047s, input_fired_time: 23.0943717957, output_fired_time: 9.7508726120, propagate_forward_time: 754.8576660156, propagate_backward_time: 1175.5211181641, weight_update_time: 0.4968402386, other_time: 18.6567382813
epoch time: 154.3929138184s, input_fired_time: 1.7627871037, output_fired_time: 0.0000000000, propagate_forward_time: 149.4273376465, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2027893066
validation[          , genome 10949] predictions:    6834/  10000 (68.34%), best:    6834/10000 (68.34%), error:      9136.75781, best error:      9136.75781 on epoch:    70, epoch:   70/100, mu: 0.9808923602, learning_rate: 0.0000070291, weight_decay: 0.0000083231

epoch time: 1938.6148681641s, input_fired_time: 23.0574817657, output_fired_time: 9.5929546356, propagate_forward_time: 738.6748046875, propagate_backward_time: 1149.6130371094, weight_update_time: 0.4968876839, other_time: 17.6766357422
epoch time: 151.3818817139s, input_fired_time: 1.7568411827, output_fired_time: 0.0000000000, propagate_forward_time: 146.3993682861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2256774902
validation[          , genome 10949] predictions:    6894/  10000 (68.94%), best:    6834/10000 (68.34%), error:      9217.06738, best error:      9136.75781 on epoch:    70, epoch:   71/100, mu: 0.9814057946, learning_rate: 0.0000064988, weight_decay: 0.0000078402

epoch time: 1931.5576171875s, input_fired_time: 23.0943298340, output_fired_time: 9.5372438431, propagate_forward_time: 735.9341430664, propagate_backward_time: 1145.5767822266, weight_update_time: 0.4977240264, other_time: 17.4151611328
epoch time: 153.6390228271s, input_fired_time: 1.7546920776, output_fired_time: 0.0000000000, propagate_forward_time: 148.5748748779, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3094635010
validation[          , genome 10949] predictions:    6868/  10000 (68.68%), best:    6834/10000 (68.34%), error:      9146.87109, best error:      9136.75781 on epoch:    70, epoch:   72/100, mu: 0.9818902612, learning_rate: 0.0000060085, weight_decay: 0.0000073853

epoch time: 1971.0976562500s, input_fired_time: 23.2235279083, output_fired_time: 9.7873191833, propagate_forward_time: 750.9766235352, propagate_backward_time: 1169.1878662109, weight_update_time: 0.5035014749, other_time: 17.9222412109
epoch time: 155.1994781494s, input_fired_time: 1.7928493023, output_fired_time: 0.0000000000, propagate_forward_time: 150.2160644531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1905670166
validation[          , genome 10949] predictions:    6883/  10000 (68.83%), best:    6883/10000 (68.83%), error:      9096.19824, best error:      9096.19824 on epoch:    73, epoch:   73/100, mu: 0.9823474288, learning_rate: 0.0000055552, weight_decay: 0.0000069568

epoch time: 1966.7764892578s, input_fired_time: 23.0101833344, output_fired_time: 9.6939096451, propagate_forward_time: 750.9567871094, propagate_backward_time: 1165.2805175781, weight_update_time: 0.5034644008, other_time: 17.8350830078
epoch time: 155.7551116943s, input_fired_time: 1.7960441113, output_fired_time: 0.0000000000, propagate_forward_time: 150.7485504150, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2105102539
validation[          , genome 10949] predictions:    6884/  10000 (68.84%), best:    6883/10000 (68.83%), error:      9105.96777, best error:      9096.19824 on epoch:    73, epoch:   74/100, mu: 0.9827787876, learning_rate: 0.0000051361, weight_decay: 0.0000065532

epoch time: 1991.7414550781s, input_fired_time: 23.1072959900, output_fired_time: 9.8556194305, propagate_forward_time: 760.8325195313, propagate_backward_time: 1180.0303955078, weight_update_time: 0.5135488510, other_time: 17.9156494141
epoch time: 159.2335968018s, input_fired_time: 1.7884725332, output_fired_time: 0.0000000000, propagate_forward_time: 154.2803192139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1648101807
validation[          , genome 10949] predictions:    6874/  10000 (68.74%), best:    6883/10000 (68.83%), error:      9098.28418, best error:      9096.19824 on epoch:    73, epoch:   75/100, mu: 0.9831858873, learning_rate: 0.0000047487, weight_decay: 0.0000061730

epoch time: 2002.9545898438s, input_fired_time: 23.1999473572, output_fired_time: 9.8684501648, propagate_forward_time: 769.3137207031, propagate_backward_time: 1182.5733642578, weight_update_time: 0.4989944696, other_time: 17.9991455078
epoch time: 157.2210845947s, input_fired_time: 1.7832120657, output_fired_time: 0.0000000000, propagate_forward_time: 152.2781829834, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1596832275
validation[          , genome 10949] predictions:    6877/  10000 (68.77%), best:    6883/10000 (68.83%), error:      9107.90918, best error:      9096.19824 on epoch:    73, epoch:   76/100, mu: 0.9835700393, learning_rate: 0.0000043904, weight_decay: 0.0000058149

epoch time: 1922.2644042969s, input_fired_time: 23.1429462433, output_fired_time: 9.5677099228, propagate_forward_time: 739.9179687500, propagate_backward_time: 1132.6278076172, weight_update_time: 0.4979316592, other_time: 17.0079345703
epoch time: 151.3262939453s, input_fired_time: 1.7546945810, output_fired_time: 0.0000000000, propagate_forward_time: 146.3430786133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2285156250
validation[          , genome 10949] predictions:    6854/  10000 (68.54%), best:    6883/10000 (68.83%), error:      9100.83594, best error:      9096.19824 on epoch:    73, epoch:   77/100, mu: 0.9839324951, learning_rate: 0.0000040592, weight_decay: 0.0000054775

epoch time: 1934.1821289063s, input_fired_time: 23.0931377411, output_fired_time: 9.6001110077, propagate_forward_time: 742.6081542969, propagate_backward_time: 1141.9317626953, weight_update_time: 0.4982565641, other_time: 16.9489746094
epoch time: 158.0226287842s, input_fired_time: 1.7789720297, output_fired_time: 0.0000000000, propagate_forward_time: 152.9281005859, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3155517578
validation[          , genome 10949] predictions:    6898/  10000 (68.98%), best:    6898/10000 (68.98%), error:      9065.71387, best error:      9065.71387 on epoch:    78, epoch:   78/100, mu: 0.9842745066, learning_rate: 0.0000037530, weight_decay: 0.0000051597

epoch time: 2010.2506103516s, input_fired_time: 23.1648368835, output_fired_time: 9.8586378098, propagate_forward_time: 767.7283325195, propagate_backward_time: 1191.7915039063, weight_update_time: 0.5001444221, other_time: 17.7072753906
epoch time: 159.4919281006s, input_fired_time: 1.7985941172, output_fired_time: 0.0000000000, propagate_forward_time: 154.5266571045, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1666717529
validation[          , genome 10949] predictions:    6890/  10000 (68.90%), best:    6898/10000 (68.98%), error:      9155.50293, best error:      9065.71387 on epoch:    78, epoch:   79/100, mu: 0.9845972657, learning_rate: 0.0000034698, weight_decay: 0.0000048604

epoch time: 2001.4516601563s, input_fired_time: 23.0873336792, output_fired_time: 9.8158445358, propagate_forward_time: 760.5440063477, propagate_backward_time: 1189.9471435547, weight_update_time: 0.5017991662, other_time: 18.0574951172
epoch time: 158.7667999268s, input_fired_time: 1.7879755497, output_fired_time: 0.0000000000, propagate_forward_time: 153.7247314453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2540893555
validation[          , genome 10949] predictions:    6869/  10000 (68.69%), best:    6898/10000 (68.98%), error:      9149.94824, best error:      9065.71387 on epoch:    78, epoch:   80/100, mu: 0.9849018455, learning_rate: 0.0000032080, weight_decay: 0.0000045784

epoch time: 1947.7239990234s, input_fired_time: 23.1644916534, output_fired_time: 9.6294660568, propagate_forward_time: 749.5140991211, propagate_backward_time: 1147.8846435547, weight_update_time: 0.4985437691, other_time: 17.5311279297
epoch time: 155.7167358398s, input_fired_time: 1.7883495092, output_fired_time: 0.0000000000, propagate_forward_time: 150.7440490723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1843414307
validation[          , genome 10949] predictions:    6876/  10000 (68.76%), best:    6898/10000 (68.98%), error:      9124.04980, best error:      9065.71387 on epoch:    78, epoch:   81/100, mu: 0.9851892591, learning_rate: 0.0000029660, weight_decay: 0.0000043128

epoch time: 1884.4210205078s, input_fired_time: 22.9129486084, output_fired_time: 9.4901580811, propagate_forward_time: 726.8045043945, propagate_backward_time: 1107.9855957031, weight_update_time: 0.4850628674, other_time: 17.2277832031
epoch time: 151.4715423584s, input_fired_time: 1.7395628691, output_fired_time: 0.0000000000, propagate_forward_time: 146.7461242676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9858551025
validation[          , genome 10949] predictions:    6887/  10000 (68.87%), best:    6898/10000 (68.98%), error:      9074.17480, best error:      9065.71387 on epoch:    78, epoch:   82/100, mu: 0.9854604602, learning_rate: 0.0000027423, weight_decay: 0.0000040625

epoch time: 1973.4913330078s, input_fired_time: 23.1485252380, output_fired_time: 9.7826347351, propagate_forward_time: 756.7112426758, propagate_backward_time: 1166.4429931641, weight_update_time: 0.5010939240, other_time: 17.4058837891
epoch time: 155.9527893066s, input_fired_time: 1.7739430666, output_fired_time: 0.0000000000, propagate_forward_time: 150.9416656494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2371826172
validation[          , genome 10949] predictions:    6853/  10000 (68.53%), best:    6898/10000 (68.98%), error:      9087.12402, best error:      9065.71387 on epoch:    78, epoch:   83/100, mu: 0.9857163429, learning_rate: 0.0000025354, weight_decay: 0.0000038268

epoch time: 2024.5600585938s, input_fired_time: 23.1746559143, output_fired_time: 9.8733482361, propagate_forward_time: 774.0024414063, propagate_backward_time: 1198.9752197266, weight_update_time: 0.5063399673, other_time: 18.5344238281
epoch time: 162.7111663818s, input_fired_time: 1.8147845268, output_fired_time: 0.0000000000, propagate_forward_time: 157.5375823975, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3587951660
validation[          , genome 10949] predictions:    6881/  10000 (68.81%), best:    6898/10000 (68.98%), error:      9074.03320, best error:      9065.71387 on epoch:    78, epoch:   84/100, mu: 0.9859578013, learning_rate: 0.0000023441, weight_decay: 0.0000036048

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1534504202_164_10949_2_r317080869_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.461141
read mu: 0.986186
read mu_delta: 0.943628
read initial_learning_rate: 0.00170423
read learning_rate: 2.16726e-06
read learning_rate_delta: 0.924557
read initial_weight_decay: 0.000546113
read weight_decay: 3.39567e-06
read weight_decay_delta: 0.941983
read batch_size: 85
read epsilon: 1e-07
read alpha: 0.0775803
read input_dropout_probability: 0.00131918
read hidden_dropout_probability: 0.0822247
read velocity_reset: 2235
read epoch: 85
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 78
read number_validation_images: 10000
read best_validation_predictions: 6898
read best_validation_error: 9065.71
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 10949
read normal distribution: '1 0 0'
generator_str: '1950334037'
read generator: 1950334037
read generated_by_map:
1 disable_edge 1
reading 124 nodes.
reading 516 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2238.93286132813s, input_fired_time: 25.6616611480713, output_fired_time: 11.1593780517578, propagate_forward_time: 877.860534667969, propagate_backward_time: 1307.77795410156, weight_update_time: 0.550795376300812, other_time: 16.4732666015625
epoch time: 178.393768310547s, input_fired_time: 1.98638677597046, output_fired_time: 0, propagate_forward_time: 173.341003417969, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.06637573242188
validation[          , genome 10949] predictions:    6868/  10000 (68.68%), best:    6898/10000 (68.98%), error:      9095.55273, best error:      9065.71387 on epoch:    78, epoch:   85/100, mu: 0.9861856699, learning_rate: 0.0000021673, weight_decay: 0.0000033957

epoch time: 2193.6379394531s, input_fired_time: 25.0474243164, output_fired_time: 10.8793516159, propagate_forward_time: 858.2840576172, propagate_backward_time: 1282.3331298828, weight_update_time: 0.5504510999, other_time: 17.0939941406
epoch time: 165.2140960693s, input_fired_time: 1.8145915270, output_fired_time: 0.0000000000, propagate_forward_time: 160.3921203613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0073852539
validation[          , genome 10949] predictions:    6892/  10000 (68.92%), best:    6892/10000 (68.92%), error:      9038.22461, best error:      9038.22461 on epoch:    86, epoch:   86/100, mu: 0.9864006639, learning_rate: 0.0000020038, weight_decay: 0.0000031987

epoch time: 2087.9479980469s, input_fired_time: 23.7898635864, output_fired_time: 10.2872009277, propagate_forward_time: 814.1791381836, propagate_backward_time: 1221.6210937500, weight_update_time: 0.5085745454, other_time: 18.0708007813
epoch time: 164.7725067139s, input_fired_time: 1.8078621626, output_fired_time: 0.0000000000, propagate_forward_time: 159.5124359131, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4522094727
validation[          , genome 10949] predictions:    6851/  10000 (68.51%), best:    6892/10000 (68.92%), error:      9067.12305, best error:      9038.22461 on epoch:    86, epoch:   87/100, mu: 0.9866035581, learning_rate: 0.0000018526, weight_decay: 0.0000030131

epoch time: 2068.2045898438s, input_fired_time: 23.5373287201, output_fired_time: 10.2492389679, propagate_forward_time: 808.7409057617, propagate_backward_time: 1207.1566162109, weight_update_time: 0.5014149547, other_time: 18.5203857422
epoch time: 166.6610870361s, input_fired_time: 1.8300261497, output_fired_time: 0.0000000000, propagate_forward_time: 161.4718627930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3591918945
validation[          , genome 10949] predictions:    6848/  10000 (68.48%), best:    6892/10000 (68.92%), error:      9101.87305, best error:      9038.22461 on epoch:    86, epoch:   88/100, mu: 0.9867950082, learning_rate: 0.0000017128, weight_decay: 0.0000028383

epoch time: 2070.3103027344s, input_fired_time: 23.6266040802, output_fired_time: 10.1925392151, propagate_forward_time: 809.0572509766, propagate_backward_time: 1208.6016845703, weight_update_time: 0.5010723472, other_time: 18.8322753906
epoch time: 168.5291137695s, input_fired_time: 1.8277405500, output_fired_time: 0.0000000000, propagate_forward_time: 163.1957855225, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5055847168
validation[          , genome 10949] predictions:    6903/  10000 (69.03%), best:    6903/10000 (69.03%), error:      9037.52441, best error:      9037.52441 on epoch:    89, epoch:   89/100, mu: 0.9869756699, learning_rate: 0.0000015836, weight_decay: 0.0000026736

epoch time: 2205.7585449219s, input_fired_time: 24.8970508575, output_fired_time: 10.7911405563, propagate_forward_time: 859.3380737305, propagate_backward_time: 1293.1077880859, weight_update_time: 0.5258202553, other_time: 17.6243896484
epoch time: 177.6079711914s, input_fired_time: 1.9857176542, output_fired_time: 0.0000000000, propagate_forward_time: 172.6232604980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9989929199
validation[          , genome 10949] predictions:    6906/  10000 (69.06%), best:    6903/10000 (69.03%), error:      9061.58984, best error:      9037.52441 on epoch:    89, epoch:   90/100, mu: 0.9871461391, learning_rate: 0.0000014641, weight_decay: 0.0000025185

epoch time: 2285.8967285156s, input_fired_time: 26.3552894592, output_fired_time: 11.4036445618, propagate_forward_time: 886.0949707031, propagate_backward_time: 1344.9243164063, weight_update_time: 0.5810570121, other_time: 17.1186523438
epoch time: 178.7587738037s, input_fired_time: 1.9985694885, output_fired_time: 0.0000000000, propagate_forward_time: 173.8100891113, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9501190186
validation[          , genome 10949] predictions:    6893/  10000 (68.93%), best:    6903/10000 (69.03%), error:      9043.10254, best error:      9037.52441 on epoch:    89, epoch:   91/100, mu: 0.9873070121, learning_rate: 0.0000013537, weight_decay: 0.0000023724

epoch time: 2247.7875976563s, input_fired_time: 26.1826915741, output_fired_time: 11.1889429092, propagate_forward_time: 872.3480834961, propagate_backward_time: 1321.5123291016, weight_update_time: 0.5715708733, other_time: 16.5555419922
epoch time: 169.9933929443s, input_fired_time: 1.9337462187, output_fired_time: 0.0000000000, propagate_forward_time: 165.2491607666, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8104858398
validation[          , genome 10949] predictions:    6894/  10000 (68.94%), best:    6903/10000 (69.03%), error:      9056.10156, best error:      9037.52441 on epoch:    89, epoch:   92/100, mu: 0.9874588251, learning_rate: 0.0000012515, weight_decay: 0.0000022347

epoch time: 2194.1828613281s, input_fired_time: 25.4465923309, output_fired_time: 10.8527412415, propagate_forward_time: 848.0170898438, propagate_backward_time: 1293.5076904297, weight_update_time: 0.5472841263, other_time: 16.3587646484
epoch time: 175.3654937744s, input_fired_time: 1.9730566740, output_fired_time: 0.0000000000, propagate_forward_time: 170.5904998779, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8019409180
validation[          , genome 10949] predictions:    6874/  10000 (68.74%), best:    6903/10000 (69.03%), error:      9043.39063, best error:      9037.52441 on epoch:    89, epoch:   93/100, mu: 0.9876020551, learning_rate: 0.0000011571, weight_decay: 0.0000021051

epoch time: 2228.6247558594s, input_fired_time: 25.8703975677, output_fired_time: 11.0228281021, propagate_forward_time: 864.2197875977, propagate_backward_time: 1311.0302734375, weight_update_time: 0.5920257568, other_time: 16.4814453125
epoch time: 180.6857757568s, input_fired_time: 2.0653851032, output_fired_time: 0.0000000000, propagate_forward_time: 175.6437835693, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9766082764
validation[          , genome 10949] predictions:    6895/  10000 (68.95%), best:    6903/10000 (69.03%), error:      9043.54102, best error:      9037.52441 on epoch:    89, epoch:   94/100, mu: 0.9877372384, learning_rate: 0.0000010698, weight_decay: 0.0000019830

epoch time: 2170.5683593750s, input_fired_time: 25.1285667419, output_fired_time: 10.7881002426, propagate_forward_time: 842.7127075195, propagate_backward_time: 1274.8762207031, weight_update_time: 0.5443612933, other_time: 17.0627441406
epoch time: 164.1618347168s, input_fired_time: 1.8289272785, output_fired_time: 0.0000000000, propagate_forward_time: 159.1726226807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1602783203
validation[          , genome 10949] predictions:    6907/  10000 (69.07%), best:    6907/10000 (69.07%), error:      9019.32422, best error:      9019.32422 on epoch:    95, epoch:   95/100, mu: 0.9878647923, learning_rate: 0.0000009891, weight_decay: 0.0000018679

epoch time: 2052.7131347656s, input_fired_time: 23.9665222168, output_fired_time: 10.4773893356, propagate_forward_time: 802.7731933594, propagate_backward_time: 1197.7419433594, weight_update_time: 0.5082181096, other_time: 17.7540283203
epoch time: 164.2449798584s, input_fired_time: 1.8365793228, output_fired_time: 0.0000000000, propagate_forward_time: 159.2329864502, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1754150391
validation[          , genome 10949] predictions:    6893/  10000 (68.93%), best:    6907/10000 (69.07%), error:      9044.19922, best error:      9019.32422 on epoch:    95, epoch:   96/100, mu: 0.9879851341, learning_rate: 0.0000009145, weight_decay: 0.0000017595

epoch time: 2097.4462890625s, input_fired_time: 24.2018203735, output_fired_time: 10.5101261139, propagate_forward_time: 820.5496826172, propagate_backward_time: 1224.7154541016, weight_update_time: 0.5221168399, other_time: 17.4692382813
epoch time: 164.1403350830s, input_fired_time: 1.7941229343, output_fired_time: 0.0000000000, propagate_forward_time: 158.8183135986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5278930664
validation[          , genome 10949] predictions:    6918/  10000 (69.18%), best:    6907/10000 (69.07%), error:      9045.42578, best error:      9019.32422 on epoch:    95, epoch:   97/100, mu: 0.9880987406, learning_rate: 0.0000008455, weight_decay: 0.0000016574

epoch time: 2055.3601074219s, input_fired_time: 23.5335483551, output_fired_time: 10.1301441193, propagate_forward_time: 803.3433837891, propagate_backward_time: 1199.6202392578, weight_update_time: 0.5091300011, other_time: 18.7327880859
epoch time: 164.6761779785s, input_fired_time: 1.7993290424, output_fired_time: 0.0000000000, propagate_forward_time: 159.4526214600, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4242248535
validation[          , genome 10949] predictions:    6902/  10000 (69.02%), best:    6907/10000 (69.07%), error:      9034.35547, best error:      9019.32422 on epoch:    95, epoch:   98/100, mu: 0.9882059097, learning_rate: 0.0000007817, weight_decay: 0.0000015613

epoch time: 2095.8298339844s, input_fired_time: 23.4690227509, output_fired_time: 10.3118352890, propagate_forward_time: 818.7545776367, propagate_backward_time: 1224.5063476563, weight_update_time: 0.5066006184, other_time: 18.7880859375
epoch time: 167.9004364014s, input_fired_time: 1.8008886576, output_fired_time: 0.0000000000, propagate_forward_time: 162.6094207764, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4901275635
validation[          , genome 10949] predictions:    6902/  10000 (69.02%), best:    6907/10000 (69.07%), error:      9029.93652, best error:      9019.32422 on epoch:    95, epoch:   99/100, mu: 0.9883070588, learning_rate: 0.0000007227, weight_decay: 0.0000014707

epoch time: 2083.9892578125s, input_fired_time: 23.5225372314, output_fired_time: 10.1853923798, propagate_forward_time: 813.5343627930, propagate_backward_time: 1217.5606689453, weight_update_time: 0.5155916810, other_time: 19.1864013672
epoch time: 165.2998199463s, input_fired_time: 1.8049948215, output_fired_time: 0.0000000000, propagate_forward_time: 160.1620635986, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3327636719
validation[          , genome 10949] predictions:    6899/  10000 (68.99%), best:    6907/10000 (69.07%), error:      9035.42871, best error:      9019.32422 on epoch:    95, epoch:  100/100, mu: 0.9884024858, learning_rate: 0.0000006682, weight_decay: 0.0000013854

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 826.8786621094s, input_fired_time: 9.0063247681, output_fired_time: 0.0000000000, propagate_forward_time: 800.7349853516, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 17.1373291016
best training[          , genome 10949] predictions:   38770/  50000 (77.54%), best:    6907/10000 (69.07%), error:     35017.22656, best error:      9019.32422 on epoch:    95, epoch:  101/100, mu: 0.9884925485, learning_rate: 0.0000006178, weight_decay: 0.0000013050
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 165.0949249268s, input_fired_time: 1.7969934940, output_fired_time: 0.0000000000, propagate_forward_time: 159.8780364990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4198913574
   testing[          , genome 10949] predictions:    6907/  10000 (69.07%), best:    6907/10000 (69.07%), error:      9019.32422, best error:      9019.32422 on epoch:    95, epoch:  101/100, mu: 0.9884925485, learning_rate: 0.0000006178, weight_decay: 0.0000013050
backpropagation finished successfully!
03:10:48 (10860): called boinc_finish(0)

</stderr_txt>
]]>