Nameexact_genome_1542495604_171_6196_1
Workunit4095963
Created18 Nov 2018, 3:04:32 UTC
Sent18 Nov 2018, 3:17:03 UTC
Report deadline22 Dec 2018, 16:08:05 UTC
Received19 Nov 2018, 7:40:17 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60920
Run time15 hours 19 min 40 sec
CPU time15 hours 14 min 37 sec
Validate stateValid
Credit8,633.86
Device peak FLOPS3.88 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size588.11 MB
Peak swap size586.46 MB
Peak disk usage14.40 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196_1_r92895482_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.453795
read mu: 0.453795
read mu_delta: 0.966086
read initial_learning_rate: 0.00118955
read learning_rate: 0.00118955
read learning_rate_delta: 0.927159
read initial_weight_decay: 0.000262441
read weight_decay: 0.000262441
read weight_decay_delta: 0.920805
read batch_size: 114
read epsilon: 1e-07
read alpha: 0.0933746
read input_dropout_probability: 0.00146728
read hidden_dropout_probability: 0.120654
read velocity_reset: 1739
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6196
read normal distribution: '0 -0.542166650295258 -0.569507360458374'
generator_str: '1350581348'
read generator: 1350581348
read generated_by_map:
1 split_node 1
reading 128 nodes.
reading 505 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 318567046
post shuffle 1: 1158463115
epoch time: 1929.39086914063s, input_fired_time: 26.4796504974365, output_fired_time: 12.8394765853882, propagate_forward_time: 743.671875, propagate_backward_time: 1131.07312011719, weight_update_time: 0.485271483659744, other_time: 15.3267822265625
epoch time: 155.228286743164s, input_fired_time: 1.97500205039978, output_fired_time: 0, propagate_forward_time: 150.588302612305, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.66497802734375
validation[          , genome  6196] predictions:    5803/  10000 (58.03%), best:    5803/10000 (58.03%), error:     12078.57422, best error:     12078.57422 on epoch:     0, epoch:    0/25, mu: 0.4537951946, learning_rate: 0.0011895527, weight_decay: 0.0002624414

epoch time: 1938.0068359375s, input_fired_time: 26.5592117310, output_fired_time: 12.8675584793, propagate_forward_time: 748.2167358398, propagate_backward_time: 1135.0963134766, weight_update_time: 0.4850803912, other_time: 15.2669677734
epoch time: 154.6114196777s, input_fired_time: 1.9792815447, output_fired_time: 0.0000000000, propagate_forward_time: 149.9742889404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6578521729
validation[          , genome  6196] predictions:    5946/  10000 (59.46%), best:    5946/10000 (59.46%), error:     11798.34961, best error:     11798.34961 on epoch:     1, epoch:    1/25, mu: 0.4719800949, learning_rate: 0.0011029040, weight_decay: 0.0002416574

epoch time: 1938.0529785156s, input_fired_time: 26.6106033325, output_fired_time: 12.8765392303, propagate_forward_time: 747.7102661133, propagate_backward_time: 1135.6816406250, weight_update_time: 0.4871706069, other_time: 15.1738281250
epoch time: 154.2501678467s, input_fired_time: 1.9640346766, output_fired_time: 0.0000000000, propagate_forward_time: 149.6225280762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6636047363
validation[          , genome  6196] predictions:    5947/  10000 (59.47%), best:    5946/10000 (59.46%), error:     11798.74609, best error:     11798.34961 on epoch:     1, epoch:    2/25, mu: 0.4895482659, learning_rate: 0.0010225668, weight_decay: 0.0002225193

epoch time: 1937.1884765625s, input_fired_time: 26.5731639862, output_fired_time: 12.8265686035, propagate_forward_time: 747.7973022461, propagate_backward_time: 1134.7796630859, weight_update_time: 0.4840973318, other_time: 15.2117919922
epoch time: 154.5030975342s, input_fired_time: 1.9785234928, output_fired_time: 0.0000000000, propagate_forward_time: 149.8683319092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6562347412
validation[          , genome  6196] predictions:    6207/  10000 (62.07%), best:    6207/10000 (62.07%), error:     11072.20801, best error:     11072.20801 on epoch:     3, epoch:    3/25, mu: 0.5065206289, learning_rate: 0.0009480815, weight_decay: 0.0002048970

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196_1_r92895482_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.453795
read mu: 0.522917
read mu_delta: 0.966086
read initial_learning_rate: 0.00118955
read learning_rate: 0.000879022
read learning_rate_delta: 0.927159
read initial_weight_decay: 0.000262441
read weight_decay: 0.00018867
read weight_decay_delta: 0.920805
read batch_size: 114
read epsilon: 1e-07
read alpha: 0.0933746
read input_dropout_probability: 0.00146728
read hidden_dropout_probability: 0.120654
read velocity_reset: 1739
read epoch: 4
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 3
read number_validation_images: 10000
read best_validation_predictions: 6207
read best_validation_error: 11072.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6196
read normal distribution: '0 -0.542166650295258 -0.569507360458374'
generator_str: '523227953'
read generator: 523227953
read generated_by_map:
1 split_node 1
reading 128 nodes.
reading 505 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1942.97033691406s, input_fired_time: 26.6889419555664, output_fired_time: 13.0011253356934, propagate_forward_time: 752.644409179688, propagate_backward_time: 1135.52880859375, weight_update_time: 0.485433757305145, other_time: 15.1070556640625
epoch time: 155.437850952148s, input_fired_time: 1.98261868953705, output_fired_time: 0, propagate_forward_time: 150.836242675781, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.61898803710938
validation[          , genome  6196] predictions:    6131/  10000 (61.31%), best:    6207/10000 (62.07%), error:     11155.20801, best error:     11072.20801 on epoch:     3, epoch:    4/25, mu: 0.5229173899, learning_rate: 0.0008790219, weight_decay: 0.0001886702

epoch time: 1948.0861816406s, input_fired_time: 26.8114414215, output_fired_time: 13.0584278107, propagate_forward_time: 759.0465087891, propagate_backward_time: 1134.1215820313, weight_update_time: 0.4993486702, other_time: 15.0482177734
epoch time: 156.4615936279s, input_fired_time: 1.9855328798, output_fired_time: 0.0000000000, propagate_forward_time: 151.8594665527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6165924072
validation[          , genome  6196] predictions:    6174/  10000 (61.74%), best:    6207/10000 (62.07%), error:     11131.13477, best error:     11072.20801 on epoch:     3, epoch:    5/25, mu: 0.5387580991, learning_rate: 0.0008149926, weight_decay: 0.0001737285

epoch time: 1947.1055908203s, input_fired_time: 26.8337459564, output_fired_time: 13.1119709015, propagate_forward_time: 758.8880615234, propagate_backward_time: 1133.1767578125, weight_update_time: 0.4947106838, other_time: 15.0950927734
epoch time: 156.6154174805s, input_fired_time: 1.9951376915, output_fired_time: 0.0000000000, propagate_forward_time: 152.0022735596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6180114746
validation[          , genome  6196] predictions:    6251/  10000 (62.51%), best:    6251/10000 (62.51%), error:     10835.04199, best error:     10835.04199 on epoch:     6, epoch:    6/25, mu: 0.5540615320, learning_rate: 0.0007556274, weight_decay: 0.0001599701

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6196_1_r92895482_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.453795
read mu: 0.568846
read mu_delta: 0.966086
read initial_learning_rate: 0.00118955
read learning_rate: 0.000700586
read learning_rate_delta: 0.927159
read initial_weight_decay: 0.000262441
read weight_decay: 0.000147301
read weight_decay_delta: 0.920805
read batch_size: 114
read epsilon: 1e-07
read alpha: 0.0933746
read input_dropout_probability: 0.00146728
read hidden_dropout_probability: 0.120654
read velocity_reset: 1739
read epoch: 7
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 6
read number_validation_images: 10000
read best_validation_predictions: 6251
read best_validation_error: 10835
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6196
read normal distribution: '0 -0.542166650295258 -0.569507360458374'
generator_str: '1079271633'
read generator: 1079271633
read generated_by_map:
1 split_node 1
reading 128 nodes.
reading 505 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1939.37573242188s, input_fired_time: 26.6273498535156, output_fired_time: 12.8850612640381, propagate_forward_time: 749.94482421875, propagate_backward_time: 1134.85107421875, weight_update_time: 0.480286300182343, other_time: 15.0675048828125
epoch time: 154.452392578125s, input_fired_time: 1.9759019613266, output_fired_time: 0, propagate_forward_time: 149.858825683594, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.61766052246094
validation[          , genome  6196] predictions:    6119/  10000 (61.19%), best:    6251/10000 (62.51%), error:     11132.40820, best error:     10835.04199 on epoch:     6, epoch:    7/25, mu: 0.5688459873, learning_rate: 0.0007005864, weight_decay: 0.0001473013

epoch time: 1940.2855224609s, input_fired_time: 26.6897811890, output_fired_time: 12.8512592316, propagate_forward_time: 750.1416015625, propagate_backward_time: 1135.5247802734, weight_update_time: 0.4762625992, other_time: 15.0780029297
epoch time: 154.9988708496s, input_fired_time: 1.9942973852, output_fired_time: 0.0000000000, propagate_forward_time: 150.3715057373, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6330718994
validation[          , genome  6196] predictions:    6196/  10000 (61.96%), best:    6251/10000 (62.51%), error:     10981.73828, best error:     10835.04199 on epoch:     6, epoch:    8/25, mu: 0.5831290483, learning_rate: 0.0006495547, weight_decay: 0.0001356358

epoch time: 1944.7736816406s, input_fired_time: 26.7398929596, output_fired_time: 12.9199132919, propagate_forward_time: 751.8824462891, propagate_backward_time: 1138.1901855469, weight_update_time: 0.4821387529, other_time: 15.0412597656
epoch time: 155.4745483398s, input_fired_time: 1.9842126369, output_fired_time: 0.0000000000, propagate_forward_time: 150.8436584473, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6466827393
validation[          , genome  6196] predictions:    6261/  10000 (62.61%), best:    6261/10000 (62.61%), error:     10818.04492, best error:     10818.04492 on epoch:     9, epoch:    9/25, mu: 0.5969277024, learning_rate: 0.0006022402, weight_decay: 0.0001248941

epoch time: 1939.5820312500s, input_fired_time: 26.6775283813, output_fired_time: 12.8500585556, propagate_forward_time: 748.7455444336, propagate_backward_time: 1136.2410888672, weight_update_time: 0.4823863506, other_time: 15.0677490234
epoch time: 154.7136688232s, input_fired_time: 1.9955979586, output_fired_time: 0.0000000000, propagate_forward_time: 150.0694122314, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6486511230
validation[          , genome  6196] predictions:    6310/  10000 (63.10%), best:    6310/10000 (63.10%), error:     10695.74316, best error:     10695.74316 on epoch:    10, epoch:   10/25, mu: 0.6102584004, learning_rate: 0.0005583721, weight_decay: 0.0001150031

epoch time: 1929.4757080078s, input_fired_time: 26.6180667877, output_fired_time: 12.7395381927, propagate_forward_time: 742.1617431641, propagate_backward_time: 1132.8060302734, weight_update_time: 0.4738255739, other_time: 15.1503906250
epoch time: 152.9172821045s, input_fired_time: 1.9915019274, output_fired_time: 0.0000000000, propagate_forward_time: 148.2486724854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6771087646
validation[          , genome  6196] predictions:    6341/  10000 (63.41%), best:    6341/10000 (63.41%), error:     10635.72949, best error:     10635.72949 on epoch:    11, epoch:   11/25, mu: 0.6231369972, learning_rate: 0.0005176995, weight_decay: 0.0001058955

epoch time: 1923.8781738281s, input_fired_time: 26.5736503601, output_fired_time: 12.6925106049, propagate_forward_time: 738.5312500000, propagate_backward_time: 1130.9277343750, weight_update_time: 0.4724060297, other_time: 15.1530761719
epoch time: 152.3542785645s, input_fired_time: 1.9764339924, output_fired_time: 0.0000000000, propagate_forward_time: 147.6939697266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6838684082
validation[          , genome  6196] predictions:    6065/  10000 (60.65%), best:    6341/10000 (63.41%), error:     11238.79492, best error:     10635.72949 on epoch:    11, epoch:   12/25, mu: 0.6355788112, learning_rate: 0.0004799895, weight_decay: 0.0000975091

epoch time: 1921.7614746094s, input_fired_time: 26.6154766083, output_fired_time: 12.6763601303, propagate_forward_time: 738.8716430664, propagate_backward_time: 1128.3277587891, weight_update_time: 0.4761726558, other_time: 15.2701416016
epoch time: 153.0768432617s, input_fired_time: 1.9935460091, output_fired_time: 0.0000000000, propagate_forward_time: 148.4110870361, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6722106934
validation[          , genome  6196] predictions:    6413/  10000 (64.13%), best:    6413/10000 (64.13%), error:     10350.80371, best error:     10350.80371 on epoch:    13, epoch:   13/25, mu: 0.6475986838, learning_rate: 0.0004450264, weight_decay: 0.0000897869

epoch time: 1928.3217773438s, input_fired_time: 26.6999015808, output_fired_time: 12.7162399292, propagate_forward_time: 740.9216918945, propagate_backward_time: 1132.8596191406, weight_update_time: 0.4691773951, other_time: 15.1242675781
epoch time: 153.4459991455s, input_fired_time: 1.9960727692, output_fired_time: 0.0000000000, propagate_forward_time: 148.7871856689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6627349854
validation[          , genome  6196] predictions:    6404/  10000 (64.04%), best:    6413/10000 (64.13%), error:     10462.58301, best error:     10350.80371 on epoch:    13, epoch:   14/25, mu: 0.6592109203, learning_rate: 0.0004126100, weight_decay: 0.0000826762

epoch time: 1923.8308105469s, input_fired_time: 26.6192989349, output_fired_time: 12.6929559708, propagate_forward_time: 739.8328857422, propagate_backward_time: 1129.4808349609, weight_update_time: 0.4749079049, other_time: 15.2048339844
epoch time: 152.9399871826s, input_fired_time: 1.9941743612, output_fired_time: 0.0000000000, propagate_forward_time: 148.2528228760, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6929931641
validation[          , genome  6196] predictions:    6442/  10000 (64.42%), best:    6413/10000 (64.13%), error:     10376.84961, best error:     10350.80371 on epoch:    13, epoch:   15/25, mu: 0.6704293489, learning_rate: 0.0003825549, weight_decay: 0.0000761287

epoch time: 1923.9901123047s, input_fired_time: 26.6541748047, output_fired_time: 12.6949243546, propagate_forward_time: 740.4117431641, propagate_backward_time: 1128.9309082031, weight_update_time: 0.4749037325, other_time: 15.2983398438
epoch time: 152.3821716309s, input_fired_time: 1.9948079586, output_fired_time: 0.0000000000, propagate_forward_time: 147.7100830078, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6772766113
validation[          , genome  6196] predictions:    6451/  10000 (64.51%), best:    6451/10000 (64.51%), error:     10249.59570, best error:     10249.59570 on epoch:    16, epoch:   16/25, mu: 0.6812673211, learning_rate: 0.0003546890, weight_decay: 0.0000700997

epoch time: 1929.6497802734s, input_fired_time: 26.6942558289, output_fired_time: 12.7659969330, propagate_forward_time: 743.3574829102, propagate_backward_time: 1131.6818847656, weight_update_time: 0.4785668552, other_time: 15.1501464844
epoch time: 154.1270294189s, input_fired_time: 2.0119645596, output_fired_time: 0.0000000000, propagate_forward_time: 149.4812927246, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6337738037
validation[          , genome  6196] predictions:    6342/  10000 (63.42%), best:    6451/10000 (64.51%), error:     10390.45313, best error:     10249.59570 on epoch:    16, epoch:   17/25, mu: 0.6917377114, learning_rate: 0.0003288529, weight_decay: 0.0000645482

epoch time: 1931.3453369141s, input_fired_time: 26.7426204681, output_fired_time: 12.7463989258, propagate_forward_time: 743.7812500000, propagate_backward_time: 1133.0140380859, weight_update_time: 0.4730351865, other_time: 15.0609130859
epoch time: 153.2061004639s, input_fired_time: 1.9948227406, output_fired_time: 0.0000000000, propagate_forward_time: 148.5314941406, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6797790527
validation[          , genome  6196] predictions:    6447/  10000 (64.47%), best:    6451/10000 (64.51%), error:     10304.54688, best error:     10249.59570 on epoch:    16, epoch:   18/25, mu: 0.7018529773, learning_rate: 0.0003048988, weight_decay: 0.0000594363

epoch time: 1927.2810058594s, input_fired_time: 26.7026920319, output_fired_time: 12.7415075302, propagate_forward_time: 741.7235107422, propagate_backward_time: 1130.9891357422, weight_update_time: 0.4741569161, other_time: 15.1242675781
epoch time: 152.8863220215s, input_fired_time: 1.9953403473, output_fired_time: 0.0000000000, propagate_forward_time: 148.2500915527, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6408843994
validation[          , genome  6196] predictions:    6480/  10000 (64.80%), best:    6451/10000 (64.51%), error:     10253.82715, best error:     10249.59570 on epoch:    16, epoch:   19/25, mu: 0.7116252184, learning_rate: 0.0002826895, weight_decay: 0.0000547292

epoch time: 1925.7683105469s, input_fired_time: 26.7131404877, output_fired_time: 12.7464179993, propagate_forward_time: 741.4627685547, propagate_backward_time: 1129.7260742188, weight_update_time: 0.4746519625, other_time: 15.1198730469
epoch time: 153.1502838135s, input_fired_time: 2.0009701252, output_fired_time: 0.0000000000, propagate_forward_time: 148.4867706299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6625366211
validation[          , genome  6196] predictions:    6415/  10000 (64.15%), best:    6451/10000 (64.51%), error:     10361.43750, best error:     10249.59570 on epoch:    16, epoch:   20/25, mu: 0.7210660577, learning_rate: 0.0002620980, weight_decay: 0.0000503950

epoch time: 1921.9375000000s, input_fired_time: 26.7176189423, output_fired_time: 12.7303133011, propagate_forward_time: 740.1904907227, propagate_backward_time: 1127.0711669922, weight_update_time: 0.4752624631, other_time: 15.2279052734
epoch time: 152.4836273193s, input_fired_time: 1.9979057312, output_fired_time: 0.0000000000, propagate_forward_time: 147.7914733887, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6942443848
validation[          , genome  6196] predictions:    6502/  10000 (65.02%), best:    6502/10000 (65.02%), error:     10150.74023, best error:     10150.74023 on epoch:    21, epoch:   21/25, mu: 0.7301867008, learning_rate: 0.0002430064, weight_decay: 0.0000464039

epoch time: 1917.6915283203s, input_fired_time: 26.7204914093, output_fired_time: 12.7257728577, propagate_forward_time: 738.3897094727, propagate_backward_time: 1124.5495605469, weight_update_time: 0.4739480019, other_time: 15.3059082031
epoch time: 152.8181610107s, input_fired_time: 2.0022764206, output_fired_time: 0.0000000000, propagate_forward_time: 148.1396484375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6762390137
validation[          , genome  6196] predictions:    6405/  10000 (64.05%), best:    6502/10000 (65.02%), error:     10411.75879, best error:     10150.74023 on epoch:    21, epoch:   22/25, mu: 0.7389980555, learning_rate: 0.0002253055, weight_decay: 0.0000427290

epoch time: 1918.0354003906s, input_fired_time: 26.7448787689, output_fired_time: 12.7625064850, propagate_forward_time: 738.4741821289, propagate_backward_time: 1124.8159179688, weight_update_time: 0.4762097299, other_time: 15.2380371094
epoch time: 152.4911956787s, input_fired_time: 2.0097038746, output_fired_time: 0.0000000000, propagate_forward_time: 147.8149719238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6665191650
validation[          , genome  6196] predictions:    6485/  10000 (64.85%), best:    6502/10000 (65.02%), error:     10232.26660, best error:     10150.74023 on epoch:    21, epoch:   23/25, mu: 0.7475105524, learning_rate: 0.0002088939, weight_decay: 0.0000393451

epoch time: 1919.5128173828s, input_fired_time: 26.7697544098, output_fired_time: 12.7861690521, propagate_forward_time: 738.6741333008, propagate_backward_time: 1126.0061035156, weight_update_time: 0.4765060544, other_time: 15.2766113281
epoch time: 154.0092468262s, input_fired_time: 2.0257384777, output_fired_time: 0.0000000000, propagate_forward_time: 149.3118286133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6716766357
validation[          , genome  6196] predictions:    6521/  10000 (65.21%), best:    6521/10000 (65.21%), error:     10115.83789, best error:     10115.83789 on epoch:    24, epoch:   24/25, mu: 0.7557343841, learning_rate: 0.0001936777, weight_decay: 0.0000362291

epoch time: 1926.9310302734s, input_fired_time: 26.7901382446, output_fired_time: 12.7419700623, propagate_forward_time: 741.3505859375, propagate_backward_time: 1130.8720703125, weight_update_time: 0.4766542614, other_time: 15.1762695313
epoch time: 153.8941192627s, input_fired_time: 2.0261864662, output_fired_time: 0.0000000000, propagate_forward_time: 149.2415161133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6264190674
validation[          , genome  6196] predictions:    6563/  10000 (65.63%), best:    6563/10000 (65.63%), error:     10003.97559, best error:     10003.97559 on epoch:    25, epoch:   25/25, mu: 0.7636792660, learning_rate: 0.0001795700, weight_decay: 0.0000333600

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 766.6466674805s, input_fired_time: 10.1102743149, output_fired_time: 0.0000000000, propagate_forward_time: 743.3393554688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.1970214844
best training[          , genome  6196] predictions:   36134/  50000 (72.27%), best:    6563/10000 (65.63%), error:     41589.37500, best error:     10003.97559 on epoch:    25, epoch:   26/25, mu: 0.7713547349, learning_rate: 0.0001664898, weight_decay: 0.0000307180
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 153.2625274658s, input_fired_time: 2.0212156773, output_fired_time: 0.0000000000, propagate_forward_time: 148.6185302734, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6227874756
   testing[          , genome  6196] predictions:    6563/  10000 (65.63%), best:    6563/10000 (65.63%), error:     10003.97559, best error:     10003.97559 on epoch:    25, epoch:   26/25, mu: 0.7713547349, learning_rate: 0.0001664898, weight_decay: 0.0000307180
backpropagation finished successfully!
00:39:29 (21876): called boinc_finish(0)

</stderr_txt>
]]>