Nameexact_genome_1542649503_173_6366_0
Workunit4099970
Created20 Nov 2018, 4:50:41 UTC
Sent20 Nov 2018, 5:03:26 UTC
Report deadline30 Dec 2018, 23:01:01 UTC
Received26 Nov 2018, 17:48:36 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88000
Run time20 hours 36 min 29 sec
CPU time20 hours 35 min 51 sec
Validate stateValid
Credit10,187.08
Device peak FLOPS3.82 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size596.08 MB
Peak swap size594.42 MB
Peak disk usage14.46 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366_0_r158082828_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550747
read mu: 0.550747
read mu_delta: 0.973943
read initial_learning_rate: 0.000743346
read learning_rate: 0.000743346
read learning_rate_delta: 0.914542
read initial_weight_decay: 0.000306793
read weight_decay: 0.000306793
read weight_decay_delta: 0.91346
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.12448
read input_dropout_probability: 0.00170219
read hidden_dropout_probability: 0.0617996
read velocity_reset: 1780
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6366
read normal distribution: '1 1.48912823200226 -0.0867462903261185'
generator_str: '2127596190'
read generator: 2127596190
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 475 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 758959133
post shuffle 1: 860584547
epoch time: 2579.70141601563s, input_fired_time: 26.4657440185547, output_fired_time: 14.3994464874268, propagate_forward_time: 1053.71887207031, propagate_backward_time: 1469.25390625, weight_update_time: 0.642221093177795, other_time: 15.8634033203125
epoch time: 218.147842407227s, input_fired_time: 2.29534196853638, output_fired_time: 0, propagate_forward_time: 213.12825012207, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.7242431640625
validation[          , genome  6366] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10669.50293, best error:     10669.50293 on epoch:     0, epoch:    0/25, mu: 0.5507472157, learning_rate: 0.0007433460, weight_decay: 0.0003067929

epoch time: 2581.7414550781s, input_fired_time: 26.5753860474, output_fired_time: 14.4139041901, propagate_forward_time: 1053.9865722656, propagate_backward_time: 1470.6242675781, weight_update_time: 0.5948225856, other_time: 16.1413574219
epoch time: 217.0072631836s, input_fired_time: 2.2968707085, output_fired_time: 0.0000000000, propagate_forward_time: 211.9455871582, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7648010254
validation[          , genome  6366] predictions:    6112/  10000 (61.12%), best:    6317/10000 (63.17%), error:     11208.93848, best error:     10669.50293 on epoch:     0, epoch:    1/25, mu: 0.5621927381, learning_rate: 0.0006798211, weight_decay: 0.0002802431

epoch time: 2588.8400878906s, input_fired_time: 26.4569263458, output_fired_time: 14.4321613312, propagate_forward_time: 1058.4829101563, propagate_backward_time: 1473.7056884766, weight_update_time: 0.5888594389, other_time: 15.7623291016
epoch time: 217.7496032715s, input_fired_time: 2.2874610424, output_fired_time: 0.0000000000, propagate_forward_time: 212.7396697998, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7224731445
validation[          , genome  6366] predictions:    6258/  10000 (62.58%), best:    6317/10000 (63.17%), error:     10972.34180, best error:     10669.50293 on epoch:     0, epoch:    2/25, mu: 0.5733400583, learning_rate: 0.0006217250, weight_decay: 0.0002559909

epoch time: 2581.5163574219s, input_fired_time: 26.4096488953, output_fired_time: 14.3816747665, propagate_forward_time: 1054.3173828125, propagate_backward_time: 1470.6337890625, weight_update_time: 0.5877999067, other_time: 15.7739257813
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366_0_r158082828_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550747
read mu: 0.584197
read mu_delta: 0.973943
read initial_learning_rate: 0.000743346
read learning_rate: 0.000568594
read learning_rate_delta: 0.914542
read initial_weight_decay: 0.000306793
read weight_decay: 0.000233837
read weight_decay_delta: 0.91346
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.12448
read input_dropout_probability: 0.00170219
read hidden_dropout_probability: 0.0617996
read velocity_reset: 1780
read epoch: 3
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6317
read best_validation_error: 10669.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6366
read normal distribution: '1 1.48912823200226 -0.0867462903261185'
generator_str: '891910944'
read generator: 891910944
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 475 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2586.31030273438s, input_fired_time: 26.4794540405273, output_fired_time: 14.4127244949341, propagate_forward_time: 1056.86669921875, propagate_backward_time: 1472.53991699219, weight_update_time: 0.592044472694397, other_time: 16.0113525390625
epoch time: 218.604370117188s, input_fired_time: 2.33460021018982, output_fired_time: 0, propagate_forward_time: 213.401748657227, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.86802673339844
validation[          , genome  6366] predictions:    6350/  10000 (63.50%), best:    6317/10000 (63.17%), error:     10785.49316, best error:     10669.50293 on epoch:     0, epoch:    3/25, mu: 0.5841968656, learning_rate: 0.0005685936, weight_decay: 0.0002338375

epoch time: 2582.7011718750s, input_fired_time: 26.3415260315, output_fired_time: 14.3813772202, propagate_forward_time: 1055.5076904297, propagate_backward_time: 1470.8613281250, weight_update_time: 0.5862613320, other_time: 15.6092529297
epoch time: 216.8064117432s, input_fired_time: 2.2713906765, output_fired_time: 0.0000000000, propagate_forward_time: 211.8213348389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7136840820
validation[          , genome  6366] predictions:    6223/  10000 (62.23%), best:    6317/10000 (63.17%), error:     10865.17188, best error:     10669.50293 on epoch:     0, epoch:    4/25, mu: 0.5947707891, learning_rate: 0.0005200027, weight_decay: 0.0002136012

epoch time: 2584.7917480469s, input_fired_time: 26.5938796997, output_fired_time: 14.4124355316, propagate_forward_time: 1055.8901367188, propagate_backward_time: 1471.6754150391, weight_update_time: 0.5973057747, other_time: 16.2198486328
epoch time: 216.5579376221s, input_fired_time: 2.3040647507, output_fired_time: 0.0000000000, propagate_forward_time: 211.4295806885, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8242950439
validation[          , genome  6366] predictions:    6185/  10000 (61.85%), best:    6317/10000 (63.17%), error:     11354.02148, best error:     10669.50293 on epoch:     0, epoch:    5/25, mu: 0.6050692201, learning_rate: 0.0004755644, weight_decay: 0.0001951162

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366_0_r158082828_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550747
read mu: 0.615099
read mu_delta: 0.973943
read initial_learning_rate: 0.000743346
read learning_rate: 0.000434924
read learning_rate_delta: 0.914542
read initial_weight_decay: 0.000306793
read weight_decay: 0.000178231
read weight_decay_delta: 0.91346
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.12448
read input_dropout_probability: 0.00170219
read hidden_dropout_probability: 0.0617996
read velocity_reset: 1780
read epoch: 6
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6317
read best_validation_error: 10669.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6366
read normal distribution: '1 1.48912823200226 -0.0867462903261185'
generator_str: '1812406785'
read generator: 1812406785
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 475 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2578.98681640625s, input_fired_time: 26.4553298950195, output_fired_time: 14.383749961853, propagate_forward_time: 1052.86633300781, propagate_backward_time: 1469.3095703125, weight_update_time: 0.591717123985291, other_time: 15.9718017578125
epoch time: 216.413970947266s, input_fired_time: 2.33212327957153, output_fired_time: 0, propagate_forward_time: 211.183349609375, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.89849853515625
validation[          , genome  6366] predictions:    6341/  10000 (63.41%), best:    6341/10000 (63.41%), error:     10609.46387, best error:     10609.46387 on epoch:     6, epoch:    6/25, mu: 0.6150993109, learning_rate: 0.0004349236, weight_decay: 0.0001782308

epoch time: 2580.1501464844s, input_fired_time: 26.4597797394, output_fired_time: 14.3767595291, propagate_forward_time: 1053.4439697266, propagate_backward_time: 1469.8651123047, weight_update_time: 0.5929682255, other_time: 16.0046386719
epoch time: 217.2554321289s, input_fired_time: 2.3433258533, output_fired_time: 0.0000000000, propagate_forward_time: 212.0348205566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8772888184
validation[          , genome  6366] predictions:    6199/  10000 (61.99%), best:    6341/10000 (63.41%), error:     11010.88477, best error:     10609.46387 on epoch:     6, epoch:    7/25, mu: 0.6248680353, learning_rate: 0.0003977559, weight_decay: 0.0001628067

epoch time: 2576.4992675781s, input_fired_time: 26.4851303101, output_fired_time: 14.3936767578, propagate_forward_time: 1051.6364746094, propagate_backward_time: 1467.9249267578, weight_update_time: 0.5952134728, other_time: 16.0592041016
epoch time: 216.9168395996s, input_fired_time: 2.3256773949, output_fired_time: 0.0000000000, propagate_forward_time: 211.7338867188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8572692871
validation[          , genome  6366] predictions:    6297/  10000 (62.97%), best:    6341/10000 (63.41%), error:     10666.59668, best error:     10609.46387 on epoch:     6, epoch:    8/25, mu: 0.6343821883, learning_rate: 0.0003637645, weight_decay: 0.0001487174

epoch time: 2578.8757324219s, input_fired_time: 26.4698982239, output_fired_time: 14.3787517548, propagate_forward_time: 1053.1043701172, propagate_backward_time: 1468.9382324219, weight_update_time: 0.5924548507, other_time: 15.9844970703
epoch time: 217.6797790527s, input_fired_time: 2.2845790386, output_fired_time: 0.0000000000, propagate_forward_time: 212.6790161133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7161865234
validation[          , genome  6366] predictions:    6396/  10000 (63.96%), best:    6396/10000 (63.96%), error:     10402.44141, best error:     10402.44141 on epoch:     9, epoch:    9/25, mu: 0.6436484456, learning_rate: 0.0003326779, weight_decay: 0.0001358475

epoch time: 2574.9470214844s, input_fired_time: 26.3911380768, output_fired_time: 14.3517093658, propagate_forward_time: 1050.9908447266, propagate_backward_time: 1467.3717041016, weight_update_time: 0.5905091763, other_time: 15.8415527344
epoch time: 216.9015655518s, input_fired_time: 2.2915852070, output_fired_time: 0.0000000000, propagate_forward_time: 211.8305053711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7794799805
validation[          , genome  6366] predictions:    6234/  10000 (62.34%), best:    6396/10000 (63.96%), error:     11018.08789, best error:     10402.44141 on epoch:     9, epoch:   10/25, mu: 0.6526732445, learning_rate: 0.0003042479, weight_decay: 0.0001240912

epoch time: 2583.3369140625s, input_fired_time: 26.4255886078, output_fired_time: 14.4014034271, propagate_forward_time: 1055.4645996094, propagate_backward_time: 1471.2603759766, weight_update_time: 0.5902872682, other_time: 15.7850341797
epoch time: 217.7858581543s, input_fired_time: 2.3157942295, output_fired_time: 0.0000000000, propagate_forward_time: 212.6513214111, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8187408447
validation[          , genome  6366] predictions:    6289/  10000 (62.89%), best:    6396/10000 (63.96%), error:     10935.95605, best error:     10402.44141 on epoch:     9, epoch:   11/25, mu: 0.6614629030, learning_rate: 0.0002782475, weight_decay: 0.0001133524

epoch time: 2585.9331054688s, input_fired_time: 26.5064048767, output_fired_time: 14.4036464691, propagate_forward_time: 1056.6419677734, propagate_backward_time: 1472.4033203125, weight_update_time: 0.5941635370, other_time: 15.9779052734
epoch time: 218.1440887451s, input_fired_time: 2.3070263863, output_fired_time: 0.0000000000, propagate_forward_time: 213.0301818848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8068847656
validation[          , genome  6366] predictions:    6222/  10000 (62.22%), best:    6396/10000 (63.96%), error:     10943.90820, best error:     10402.44141 on epoch:     9, epoch:   12/25, mu: 0.6700235605, learning_rate: 0.0002544691, weight_decay: 0.0001035429

epoch time: 2578.7568359375s, input_fired_time: 26.3947734833, output_fired_time: 14.3669023514, propagate_forward_time: 1053.3189697266, propagate_backward_time: 1468.9163818359, weight_update_time: 0.5881626010, other_time: 15.7597656250
epoch time: 218.1744995117s, input_fired_time: 2.2913618088, output_fired_time: 0.0000000000, propagate_forward_time: 213.1226501465, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7604827881
validation[          , genome  6366] predictions:    6250/  10000 (62.50%), best:    6396/10000 (63.96%), error:     10873.48145, best error:     10402.44141 on epoch:     9, epoch:   13/25, mu: 0.6783611178, learning_rate: 0.0002327226, weight_decay: 0.0000945823

epoch time: 2585.4667968750s, input_fired_time: 26.6476211548, output_fired_time: 14.4142131805, propagate_forward_time: 1055.9224853516, propagate_backward_time: 1472.1539306641, weight_update_time: 0.6010389328, other_time: 16.3283691406
epoch time: 218.0024871826s, input_fired_time: 2.3414874077, output_fired_time: 0.0000000000, propagate_forward_time: 212.7473907471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9136047363
validation[          , genome  6366] predictions:    6359/  10000 (63.59%), best:    6396/10000 (63.96%), error:     10523.32617, best error:     10402.44141 on epoch:     9, epoch:   14/25, mu: 0.6864814162, learning_rate: 0.0002128346, weight_decay: 0.0000863971

epoch time: 2582.0458984375s, input_fired_time: 26.5089149475, output_fired_time: 14.3904838562, propagate_forward_time: 1054.4180908203, propagate_backward_time: 1470.6323242188, weight_update_time: 0.5939155221, other_time: 16.0960693359
epoch time: 218.0702972412s, input_fired_time: 2.3471083641, output_fired_time: 0.0000000000, propagate_forward_time: 212.8279876709, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8952026367
validation[          , genome  6366] predictions:    6420/  10000 (64.20%), best:    6396/10000 (63.96%), error:     10424.86328, best error:     10402.44141 on epoch:     9, epoch:   15/25, mu: 0.6943901181, learning_rate: 0.0001946462, weight_decay: 0.0000789203

epoch time: 2579.7250976563s, input_fired_time: 26.5382804871, output_fired_time: 14.4051971436, propagate_forward_time: 1053.2928466797, propagate_backward_time: 1469.4191894531, weight_update_time: 0.5946667790, other_time: 16.0694580078
epoch time: 218.3554687500s, input_fired_time: 2.3462121487, output_fired_time: 0.0000000000, propagate_forward_time: 213.1152038574, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8940582275
validation[          , genome  6366] predictions:    6212/  10000 (62.12%), best:    6396/10000 (63.96%), error:     10959.44629, best error:     10402.44141 on epoch:     9, epoch:   16/25, mu: 0.7020927668, learning_rate: 0.0001780121, weight_decay: 0.0000720906

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_173_6366_0_r158082828_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550747
read mu: 0.709595
read mu_delta: 0.973943
read initial_learning_rate: 0.000743346
read learning_rate: 0.0001628
read learning_rate_delta: 0.914542
read initial_weight_decay: 0.000306793
read weight_decay: 6.58519e-05
read weight_decay_delta: 0.91346
read batch_size: 123
read epsilon: 1e-07
read alpha: 0.12448
read input_dropout_probability: 0.00170219
read hidden_dropout_probability: 0.0617996
read velocity_reset: 1780
read epoch: 17
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 9
read number_validation_images: 10000
read best_validation_predictions: 6396
read best_validation_error: 10402.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6366
read normal distribution: '1 1.48912823200226 -0.0867462903261185'
generator_str: '1019752336'
read generator: 1019752336
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 475 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2591.73828125s, input_fired_time: 26.488073348999, output_fired_time: 14.4400482177734, propagate_forward_time: 1059.29284667969, propagate_backward_time: 1475.72546386719, weight_update_time: 0.591730058193207, other_time: 15.7919921875
epoch time: 218.636581420898s, input_fired_time: 2.35287499427795, output_fired_time: 0, propagate_forward_time: 213.353515625, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.93019104003906
validation[          , genome  6366] predictions:    6527/  10000 (65.27%), best:    6527/10000 (65.27%), error:     10186.90918, best error:     10186.90918 on epoch:    17, epoch:   17/25, mu: 0.7095947266, learning_rate: 0.0001627996, weight_decay: 0.0000658519

epoch time: 2600.2492675781s, input_fired_time: 26.5664005280, output_fired_time: 14.4811725616, propagate_forward_time: 1063.3161621094, propagate_backward_time: 1479.9172363281, weight_update_time: 0.5953204036, other_time: 15.9682617188
epoch time: 218.6632080078s, input_fired_time: 2.2946493626, output_fired_time: 0.0000000000, propagate_forward_time: 213.5976867676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7708740234
validation[          , genome  6366] predictions:    6401/  10000 (64.01%), best:    6527/10000 (65.27%), error:     10466.18652, best error:     10186.90918 on epoch:    17, epoch:   18/25, mu: 0.7169011831, learning_rate: 0.0001488871, weight_decay: 0.0000601530

epoch time: 2578.2075195313s, input_fired_time: 26.4557304382, output_fired_time: 14.4248342514, propagate_forward_time: 1051.8142089844, propagate_backward_time: 1469.1447753906, weight_update_time: 0.5932908654, other_time: 16.3679199219
epoch time: 218.3711090088s, input_fired_time: 2.2954945564, output_fired_time: 0.0000000000, propagate_forward_time: 213.3167724609, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7588348389
validation[          , genome  6366] predictions:    6401/  10000 (64.01%), best:    6527/10000 (65.27%), error:     10367.82031, best error:     10186.90918 on epoch:    17, epoch:   19/25, mu: 0.7240172625, learning_rate: 0.0001361635, weight_decay: 0.0000549474

epoch time: 2587.7011718750s, input_fired_time: 26.6121768951, output_fired_time: 14.4661846161, propagate_forward_time: 1056.7542724609, propagate_backward_time: 1473.2124023438, weight_update_time: 0.5978282094, other_time: 16.6563720703
epoch time: 216.5918273926s, input_fired_time: 2.2407903671, output_fired_time: 0.0000000000, propagate_forward_time: 211.3041992188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0468444824
validation[          , genome  6366] predictions:    6429/  10000 (64.29%), best:    6527/10000 (65.27%), error:     10353.51270, best error:     10186.90918 on epoch:    17, epoch:   20/25, mu: 0.7309479117, learning_rate: 0.0001245272, weight_decay: 0.0000501923

epoch time: 2586.6594238281s, input_fired_time: 26.5482597351, output_fired_time: 14.4499359131, propagate_forward_time: 1055.5910644531, propagate_backward_time: 1473.9791259766, weight_update_time: 0.5933865309, other_time: 16.0909423828
epoch time: 215.5792541504s, input_fired_time: 2.2867960930, output_fired_time: 0.0000000000, propagate_forward_time: 209.9992370605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2932281494
validation[          , genome  6366] predictions:    6419/  10000 (64.19%), best:    6527/10000 (65.27%), error:     10561.92090, best error:     10186.90918 on epoch:    17, epoch:   21/25, mu: 0.7376979589, learning_rate: 0.0001138854, weight_decay: 0.0000458486

epoch time: 2582.5942382813s, input_fired_time: 26.4278945923, output_fired_time: 14.3987035751, propagate_forward_time: 1054.8762207031, propagate_backward_time: 1471.0913085938, weight_update_time: 0.5868407488, other_time: 15.8000488281
epoch time: 217.0476531982s, input_fired_time: 2.3102576733, output_fired_time: 0.0000000000, propagate_forward_time: 211.8515472412, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8858489990
validation[          , genome  6366] predictions:    6414/  10000 (64.14%), best:    6527/10000 (65.27%), error:     10346.58691, best error:     10186.90918 on epoch:    17, epoch:   22/25, mu: 0.7442721725, learning_rate: 0.0001041530, weight_decay: 0.0000418809

epoch time: 2587.8750000000s, input_fired_time: 26.5196723938, output_fired_time: 14.4356193542, propagate_forward_time: 1056.5994873047, propagate_backward_time: 1474.2612304688, weight_update_time: 0.5909475088, other_time: 16.0589599609
epoch time: 217.1808776855s, input_fired_time: 2.2957415581, output_fired_time: 0.0000000000, propagate_forward_time: 211.4296264648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4555053711
validation[          , genome  6366] predictions:    6543/  10000 (65.43%), best:    6543/10000 (65.43%), error:     10106.50879, best error:     10106.50879 on epoch:    23, epoch:   23/25, mu: 0.7506750822, learning_rate: 0.0000952523, weight_decay: 0.0000382565

epoch time: 2594.5268554688s, input_fired_time: 26.7245483398, output_fired_time: 14.4949436188, propagate_forward_time: 1057.3079833984, propagate_backward_time: 1479.1529541016, weight_update_time: 0.6022760272, other_time: 16.8464355469
epoch time: 217.5632171631s, input_fired_time: 2.3360054493, output_fired_time: 0.0000000000, propagate_forward_time: 212.3598632813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8673553467
validation[          , genome  6366] predictions:    6601/  10000 (66.01%), best:    6601/10000 (66.01%), error:     10012.87207, best error:     10012.87207 on epoch:    24, epoch:   24/25, mu: 0.7569111586, learning_rate: 0.0000871122, weight_decay: 0.0000349458

epoch time: 2587.9108886719s, input_fired_time: 26.6611537933, output_fired_time: 14.4271011353, propagate_forward_time: 1056.7280273438, propagate_backward_time: 1473.9312744141, weight_update_time: 0.5950678587, other_time: 16.1634521484
epoch time: 212.6531219482s, input_fired_time: 2.2125141621, output_fired_time: 0.0000000000, propagate_forward_time: 207.1040496826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3365631104
validation[          , genome  6366] predictions:    6589/  10000 (65.89%), best:    6589/10000 (65.89%), error:      9923.11133, best error:      9923.11133 on epoch:    25, epoch:   25/25, mu: 0.7629846931, learning_rate: 0.0000796678, weight_decay: 0.0000319216

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 1075.0096435547s, input_fired_time: 11.4531545639, output_fired_time: 0.0000000000, propagate_forward_time: 1048.8774414063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 14.6790771484
best training[          , genome  6366] predictions:   36225/  50000 (72.45%), best:    6589/10000 (65.89%), error:     41930.35547, best error:      9923.11133 on epoch:    25, epoch:   26/25, mu: 0.7688999772, learning_rate: 0.0000728595, weight_decay: 0.0000291591
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 215.5547790527s, input_fired_time: 2.3109388351, output_fired_time: 0.0000000000, propagate_forward_time: 210.2339477539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0098876953
   testing[          , genome  6366] predictions:    6589/  10000 (65.89%), best:    6589/10000 (65.89%), error:      9923.11133, best error:      9923.11133 on epoch:    25, epoch:   26/25, mu: 0.7688999772, learning_rate: 0.0000728595, weight_decay: 0.0000291591
backpropagation finished successfully!
18:28:22 (12596): called boinc_finish(0)

</stderr_txt>
]]>