Nameexact_genome_1540746903_170_5320_1
Workunit4075223
Created28 Oct 2018, 19:39:20 UTC
Sent28 Oct 2018, 20:13:03 UTC
Report deadline25 Nov 2018, 14:55:30 UTC
Received7 Dec 2018, 17:40:17 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64195
Run time10 hours 30 min 39 sec
CPU time10 hours 19 min 3 sec
Validate stateValid
Credit6,944.87
Device peak FLOPS3.80 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size572.07 MB
Peak swap size591.63 MB
Peak disk usage11.27 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320_1_r1966419571_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.463283
read mu: 0.463283
read mu_delta: 0.937378
read initial_learning_rate: 2.17962e-05
read learning_rate: 2.17962e-05
read learning_rate_delta: 0.906173
read initial_weight_decay: 0.00018413
read weight_decay: 0.00018413
read weight_decay_delta: 0.925541
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0536147
read input_dropout_probability: 0.00151892
read hidden_dropout_probability: 0.077557
read velocity_reset: 1369
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5320
read normal distribution: '0 -1.02765774726868 0.532374441623688'
generator_str: '910376454'
read generator: 910376454
read generated_by_map:
1 crossover 1
reading 104 nodes.
reading 370 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2023561150
post shuffle 1: 1847918996
epoch time: 1044.6240234375s, input_fired_time: 20.9195919036865, output_fired_time: 7.12320518493652, propagate_forward_time: 371.557586669922, propagate_backward_time: 632.710144042969, weight_update_time: 0.248093098402023, other_time: 12.3134155273438
epoch time: 77.9721221923828s, input_fired_time: 1.29238045215607, output_fired_time: 0, propagate_forward_time: 74.5740737915039, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.10566711425781
validation[          , genome  5320] predictions:    6044/  10000 (60.44%), best:    6044/10000 (60.44%), error:     11448.40137, best error:     11448.40137 on epoch:     0, epoch:    0/25, mu: 0.4632826149, learning_rate: 0.0000217962, weight_decay: 0.0001841302

epoch time: 1043.3227539063s, input_fired_time: 20.9075489044, output_fired_time: 7.1229772568, propagate_forward_time: 371.1940002441, propagate_backward_time: 631.7927856445, weight_update_time: 0.2471841425, other_time: 12.3054809570
epoch time: 78.1079406738s, input_fired_time: 1.2946107388, output_fired_time: 0.0000000000, propagate_forward_time: 74.6903991699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1229324341
validation[          , genome  5320] predictions:    6124/  10000 (61.24%), best:    6124/10000 (61.24%), error:     11244.86328, best error:     11244.86328 on epoch:     1, epoch:    1/25, mu: 0.4962666333, learning_rate: 0.0000197512, weight_decay: 0.0001704200

epoch time: 1042.2915039063s, input_fired_time: 20.9220466614, output_fired_time: 7.1237201691, propagate_forward_time: 370.7532653809, propagate_backward_time: 631.2055053711, weight_update_time: 0.2466613948, other_time: 12.2869262695
epoch time: 77.9798126221s, input_fired_time: 1.2934877872, output_fired_time: 0.0000000000, propagate_forward_time: 74.5672760010, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1190490723
validation[          , genome  5320] predictions:    6137/  10000 (61.37%), best:    6124/10000 (61.24%), error:     11245.85742, best error:     11244.86328 on epoch:     1, epoch:    2/25, mu: 0.5271851420, learning_rate: 0.0000178980, weight_decay: 0.0001577306

epoch time: 1042.9841308594s, input_fired_time: 20.9370803833, output_fired_time: 7.1271886826, propagate_forward_time: 370.8655700684, propagate_backward_time: 631.7224731445, weight_update_time: 0.2473991215, other_time: 12.3318481445
epoch time: 77.9892120361s, input_fired_time: 1.2967870235, output_fired_time: 0.0000000000, propagate_forward_time: 74.5807571411, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1116714478
validation[          , genome  5320] predictions:    6170/  10000 (61.70%), best:    6124/10000 (61.24%), error:     11259.74805, best error:     11244.86328 on epoch:     1, epoch:    3/25, mu: 0.5561674833, learning_rate: 0.0000162187, weight_decay: 0.0001459860

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320_1_r1966419571_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.463283
read mu: 0.583335
read mu_delta: 0.937378
read initial_learning_rate: 2.17962e-05
read learning_rate: 1.46969e-05
read learning_rate_delta: 0.906173
read initial_weight_decay: 0.00018413
read weight_decay: 0.000135116
read weight_decay_delta: 0.925541
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0536147
read input_dropout_probability: 0.00151892
read hidden_dropout_probability: 0.077557
read velocity_reset: 1369
read epoch: 4
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 6124
read best_validation_error: 11244.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5320
read normal distribution: '0 -1.02765774726868 0.532374441623688'
generator_str: '2057917850'
read generator: 2057917850
read generated_by_map:
1 crossover 1
reading 104 nodes.
reading 370 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320_1_r1966419571_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.463283
read mu: 0.583335
read mu_delta: 0.937378
read initial_learning_rate: 2.17962e-05
read learning_rate: 1.46969e-05
read learning_rate_delta: 0.906173
read initial_weight_decay: 0.00018413
read weight_decay: 0.000135116
read weight_decay_delta: 0.925541
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0536147
read input_dropout_probability: 0.00151892
read hidden_dropout_probability: 0.077557
read velocity_reset: 1369
read epoch: 4
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 6124
read best_validation_error: 11244.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5320
read normal distribution: '0 -1.02765774726868 0.532374441623688'
generator_str: '2057917850'
read generator: 2057917850
read generated_by_map:
1 crossover 1
reading 104 nodes.
reading 370 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320_1_r1966419571_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.463283
read mu: 0.583335
read mu_delta: 0.937378
read initial_learning_rate: 2.17962e-05
read learning_rate: 1.46969e-05
read learning_rate_delta: 0.906173
read initial_weight_decay: 0.00018413
read weight_decay: 0.000135116
read weight_decay_delta: 0.925541
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0536147
read input_dropout_probability: 0.00151892
read hidden_dropout_probability: 0.077557
read velocity_reset: 1369
read epoch: 4
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 6124
read best_validation_error: 11244.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5320
read normal distribution: '0 -1.02765774726868 0.532374441623688'
generator_str: '2057917850'
read generator: 2057917850
read generated_by_map:
1 crossover 1
reading 104 nodes.
reading 370 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1229.01086425781s, input_fired_time: 23.8705062866211, output_fired_time: 8.39694786071777, propagate_forward_time: 445.451568603516, propagate_backward_time: 736.475524902344, weight_update_time: 0.295989274978638, other_time: 14.8163452148438
epoch time: 140.592224121094s, input_fired_time: 2.03633165359497, output_fired_time: 0, propagate_forward_time: 134.722900390625, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.83299255371094
validation[          , genome  5320] predictions:    6170/  10000 (61.70%), best:    6124/10000 (61.24%), error:     11322.74023, best error:     11244.86328 on epoch:     1, epoch:    4/25, mu: 0.5833348632, learning_rate: 0.0000146969, weight_decay: 0.0001351160

epoch time: 1253.7072753906s, input_fired_time: 23.6101760864, output_fired_time: 8.7030792236, propagate_forward_time: 457.1257019043, propagate_backward_time: 749.5041503906, weight_update_time: 0.2956477702, other_time: 14.7640380859
epoch time: 80.5979843140s, input_fired_time: 1.3496464491, output_fired_time: 0.0000000000, propagate_forward_time: 76.9597244263, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2886123657
validation[          , genome  5320] predictions:    6118/  10000 (61.18%), best:    6124/10000 (61.24%), error:     11438.49316, best error:     11244.86328 on epoch:     1, epoch:    5/25, mu: 0.6088010073, learning_rate: 0.0000133180, weight_decay: 0.0001250553

epoch time: 1099.2869873047s, input_fired_time: 22.1410980225, output_fired_time: 7.5739588737, propagate_forward_time: 390.9614257813, propagate_backward_time: 665.1989135742, weight_update_time: 0.2645189464, other_time: 13.4115600586
epoch time: 80.9753494263s, input_fired_time: 1.3629202843, output_fired_time: 0.0000000000, propagate_forward_time: 77.3129959106, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2994308472
validation[          , genome  5320] predictions:    6161/  10000 (61.61%), best:    6124/10000 (61.24%), error:     11471.67773, best error:     11244.86328 on epoch:     1, epoch:    6/25, mu: 0.6326723695, learning_rate: 0.0000120684, weight_decay: 0.0001157438

epoch time: 1208.2908935547s, input_fired_time: 22.9911575317, output_fired_time: 8.2023096085, propagate_forward_time: 435.7653808594, propagate_backward_time: 727.0847167969, weight_update_time: 0.2831107080, other_time: 14.2473144531
epoch time: 132.9501190186s, input_fired_time: 1.9235625267, output_fired_time: 0.0000000000, propagate_forward_time: 127.4753417969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5512084961
validation[          , genome  5320] predictions:    6152/  10000 (61.52%), best:    6124/10000 (61.24%), error:     11530.76563, best error:     11244.86328 on epoch:     1, epoch:    7/25, mu: 0.6550489068, learning_rate: 0.0000109360, weight_decay: 0.0001071255

epoch time: 1685.6885986328s, input_fired_time: 25.5425853729, output_fired_time: 11.2986345291, propagate_forward_time: 635.2736206055, propagate_backward_time: 995.6580200195, weight_update_time: 0.3776208162, other_time: 17.9157714844
epoch time: 136.7063293457s, input_fired_time: 2.0025522709, output_fired_time: 0.0000000000, propagate_forward_time: 131.3699645996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3338165283
validation[          , genome  5320] predictions:    6152/  10000 (61.52%), best:    6124/10000 (61.24%), error:     11612.08594, best error:     11244.86328 on epoch:     1, epoch:    8/25, mu: 0.6760241389, learning_rate: 0.0000099100, weight_decay: 0.0000991490

epoch time: 1707.0766601563s, input_fired_time: 25.8956165314, output_fired_time: 11.3704252243, propagate_forward_time: 640.6626586914, propagate_backward_time: 1010.2770996094, weight_update_time: 0.3897512555, other_time: 18.8708496094
epoch time: 145.0200500488s, input_fired_time: 2.0529460907, output_fired_time: 0.0000000000, propagate_forward_time: 138.9694366455, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9976654053
validation[          , genome  5320] predictions:    6116/  10000 (61.16%), best:    6124/10000 (61.24%), error:     11693.63184, best error:     11244.86328 on epoch:     1, epoch:    9/25, mu: 0.6956858635, learning_rate: 0.0000089801, weight_decay: 0.0000917664

epoch time: 1658.6895751953s, input_fired_time: 25.3831787109, output_fired_time: 11.0584554672, propagate_forward_time: 617.9797973633, propagate_backward_time: 986.1110229492, weight_update_time: 0.3750946224, other_time: 18.1571044922
epoch time: 139.1208953857s, input_fired_time: 1.9811350107, output_fired_time: 0.0000000000, propagate_forward_time: 133.6777648926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4619903564
validation[          , genome  5320] predictions:    6107/  10000 (61.07%), best:    6124/10000 (61.24%), error:     11770.19531, best error:     11244.86328 on epoch:     1, epoch:   10/25, mu: 0.7141163349, learning_rate: 0.0000081376, weight_decay: 0.0000849335

epoch time: 1752.2342529297s, input_fired_time: 26.0020885468, output_fired_time: 11.4367094040, propagate_forward_time: 661.8927001953, propagate_backward_time: 1033.8226318359, weight_update_time: 0.3960701823, other_time: 19.0800781250
epoch time: 141.1281127930s, input_fired_time: 1.9984908104, output_fired_time: 0.0000000000, propagate_forward_time: 135.8419342041, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2876892090
validation[          , genome  5320] predictions:    6091/  10000 (60.91%), best:    6124/10000 (61.24%), error:     11849.40820, best error:     11244.86328 on epoch:     1, epoch:   11/25, mu: 0.7313926816, learning_rate: 0.0000073740, weight_decay: 0.0000786094

epoch time: 1749.9162597656s, input_fired_time: 25.8529300690, output_fired_time: 11.4940414429, propagate_forward_time: 657.3812866211, propagate_backward_time: 1035.7512207031, weight_update_time: 0.4024972320, other_time: 19.4367675781
epoch time: 143.4234924316s, input_fired_time: 2.0694303513, output_fired_time: 0.0000000000, propagate_forward_time: 137.5510253906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8030395508
validation[          , genome  5320] predictions:    6109/  10000 (61.09%), best:    6124/10000 (61.24%), error:     11892.39453, best error:     11244.86328 on epoch:     1, epoch:   12/25, mu: 0.7475871444, learning_rate: 0.0000066822, weight_decay: 0.0000727562

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_170_5320_1_r1966419571_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.463283
read mu: 0.762767
read mu_delta: 0.937378
read initial_learning_rate: 2.17962e-05
read learning_rate: 6.05519e-06
read learning_rate_delta: 0.906173
read initial_weight_decay: 0.00018413
read weight_decay: 6.73388e-05
read weight_decay_delta: 0.925541
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0536147
read input_dropout_probability: 0.00151892
read hidden_dropout_probability: 0.077557
read velocity_reset: 1369
read epoch: 13
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 6124
read best_validation_error: 11244.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5320
read normal distribution: '0 -1.02765774726868 0.532374441623688'
generator_str: '1222215949'
read generator: 1222215949
read generated_by_map:
1 crossover 1
reading 104 nodes.
reading 370 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2032.47644042969s, input_fired_time: 32.8728523254395, output_fired_time: 9.23744773864746, propagate_forward_time: 736.896118164063, propagate_backward_time: 1233.88549804688, weight_update_time: 0.31337982416153, other_time: 19.5845947265625
epoch time: 108.242706298828s, input_fired_time: 1.65883326530457, output_fired_time: 0, propagate_forward_time: 103.957443237305, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.62642669677734
validation[          , genome  5320] predictions:    6119/  10000 (61.19%), best:    6124/10000 (61.24%), error:     11942.75391, best error:     11244.86328 on epoch:     1, epoch:   13/25, mu: 0.7627674937, learning_rate: 0.0000060552, weight_decay: 0.0000673388

epoch time: 1349.7957763672s, input_fired_time: 23.2177734375, output_fired_time: 8.9652433395, propagate_forward_time: 495.7362365723, propagate_backward_time: 807.0164794922, weight_update_time: 0.3088509738, other_time: 14.8601074219
epoch time: 107.3848190308s, input_fired_time: 1.6409230232, output_fired_time: 0.0000000000, propagate_forward_time: 103.1552581787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5886383057
validation[          , genome  5320] predictions:    6114/  10000 (61.14%), best:    6124/10000 (61.24%), error:     12004.72168, best error:     11244.86328 on epoch:     1, epoch:   14/25, mu: 0.7769972086, learning_rate: 0.0000054871, weight_decay: 0.0000623248

epoch time: 1382.5635986328s, input_fired_time: 23.6026287079, output_fired_time: 9.2552700043, propagate_forward_time: 510.0044250488, propagate_backward_time: 824.7396850586, weight_update_time: 0.3192304969, other_time: 14.9616088867
epoch time: 99.9844207764s, input_fired_time: 1.5570755005, output_fired_time: 0.0000000000, propagate_forward_time: 95.9210586548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5062866211
validation[          , genome  5320] predictions:    6087/  10000 (60.87%), best:    6124/10000 (61.24%), error:     12048.80957, best error:     11244.86328 on epoch:     1, epoch:   15/25, mu: 0.7903358340, learning_rate: 0.0000049722, weight_decay: 0.0000576841

epoch time: 1203.2194824219s, input_fired_time: 22.2955780029, output_fired_time: 8.1001291275, propagate_forward_time: 434.7963562012, propagate_backward_time: 723.8699340820, weight_update_time: 0.2854450643, other_time: 14.1575317383
epoch time: 80.2805786133s, input_fired_time: 1.3986958265, output_fired_time: 0.0000000000, propagate_forward_time: 76.3218765259, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5600051880
validation[          , genome  5320] predictions:    6069/  10000 (60.69%), best:    6124/10000 (61.24%), error:     12128.48633, best error:     11244.86328 on epoch:     1, epoch:   16/25, mu: 0.8028391600, learning_rate: 0.0000045057, weight_decay: 0.0000533890

epoch time: 1172.8112792969s, input_fired_time: 22.3916435242, output_fired_time: 8.0008687973, propagate_forward_time: 421.3133850098, propagate_backward_time: 706.3125000000, weight_update_time: 0.2933441401, other_time: 14.7929687500
epoch time: 151.6135101318s, input_fired_time: 2.0762126446, output_fired_time: 0.0000000000, propagate_forward_time: 146.3837432861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1535491943
validation[          , genome  5320] predictions:    6045/  10000 (60.45%), best:    6124/10000 (61.24%), error:     12069.55273, best error:     11244.86328 on epoch:     1, epoch:   17/25, mu: 0.8145595193, learning_rate: 0.0000040829, weight_decay: 0.0000494137

epoch time: 1864.3927001953s, input_fired_time: 26.0738124847, output_fired_time: 11.9391717911, propagate_forward_time: 704.8434448242, propagate_backward_time: 1103.8393554688, weight_update_time: 0.4031726122, other_time: 17.6967773438
epoch time: 125.9345550537s, input_fired_time: 1.8404726982, output_fired_time: 0.0000000000, propagate_forward_time: 121.1947097778, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8993759155
validation[          , genome  5320] predictions:    6062/  10000 (60.62%), best:    6124/10000 (61.24%), error:     12179.83496, best error:     11244.86328 on epoch:     1, epoch:   18/25, mu: 0.8255459070, learning_rate: 0.0000036999, weight_decay: 0.0000457344

epoch time: 1696.8189697266s, input_fired_time: 25.5390377045, output_fired_time: 11.0144176483, propagate_forward_time: 635.3535156250, propagate_backward_time: 1008.1506958008, weight_update_time: 0.3772161901, other_time: 16.7612915039
epoch time: 122.0163726807s, input_fired_time: 1.8051825762, output_fired_time: 0.0000000000, propagate_forward_time: 117.3430252075, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8681640625
validation[          , genome  5320] predictions:    6046/  10000 (60.46%), best:    6124/10000 (61.24%), error:     12146.78809, best error:     11244.86328 on epoch:     1, epoch:   19/25, mu: 0.8358443379, learning_rate: 0.0000033527, weight_decay: 0.0000423290

epoch time: 1687.8385009766s, input_fired_time: 25.5213375092, output_fired_time: 10.9099979401, propagate_forward_time: 635.0808105469, propagate_backward_time: 999.5789794922, weight_update_time: 0.3731332421, other_time: 16.7473144531
epoch time: 135.7877044678s, input_fired_time: 1.9815777540, output_fired_time: 0.0000000000, propagate_forward_time: 130.7800445557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0260772705
validation[          , genome  5320] predictions:    6058/  10000 (60.58%), best:    6124/10000 (61.24%), error:     12179.05469, best error:     11244.86328 on epoch:     1, epoch:   20/25, mu: 0.8454978466, learning_rate: 0.0000030381, weight_decay: 0.0000391772

epoch time: 1321.4301757813s, input_fired_time: 21.6289653778, output_fired_time: 8.6020126343, propagate_forward_time: 488.2112426758, propagate_backward_time: 788.9497070313, weight_update_time: 0.2982709706, other_time: 14.0382690430
epoch time: 66.4678802490s, input_fired_time: 1.1169638634, output_fired_time: 0.0000000000, propagate_forward_time: 63.4955978394, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8553161621
validation[          , genome  5320] predictions:    6068/  10000 (60.68%), best:    6124/10000 (61.24%), error:     12167.80078, best error:     11244.86328 on epoch:     1, epoch:   21/25, mu: 0.8545468450, learning_rate: 0.0000027531, weight_decay: 0.0000362601

epoch time: 828.9273681641s, input_fired_time: 16.6875038147, output_fired_time: 5.7072677612, propagate_forward_time: 293.5129089355, propagate_backward_time: 502.7746582031, weight_update_time: 0.2022198290, other_time: 10.2449951172
epoch time: 59.7062301636s, input_fired_time: 1.0018639565, output_fired_time: 0.0000000000, propagate_forward_time: 57.0438041687, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6605606079
validation[          , genome  5320] predictions:    6025/  10000 (60.25%), best:    6124/10000 (61.24%), error:     12244.45605, best error:     11244.86328 on epoch:     1, epoch:   22/25, mu: 0.8630291820, learning_rate: 0.0000024948, weight_decay: 0.0000335602

epoch time: 815.3161621094s, input_fired_time: 16.4133472443, output_fired_time: 5.5821480751, propagate_forward_time: 289.4204406738, propagate_backward_time: 494.0486755371, weight_update_time: 0.1937838793, other_time: 9.8515625000
epoch time: 63.0165901184s, input_fired_time: 1.0610057116, output_fired_time: 0.0000000000, propagate_forward_time: 60.1681442261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7874412537
validation[          , genome  5320] predictions:    6071/  10000 (60.71%), best:    6124/10000 (61.24%), error:     12237.09277, best error:     11244.86328 on epoch:     1, epoch:   23/25, mu: 0.8709803224, learning_rate: 0.0000022607, weight_decay: 0.0000310613

epoch time: 815.1794433594s, input_fired_time: 16.4192886353, output_fired_time: 5.5900053978, propagate_forward_time: 289.2744750977, propagate_backward_time: 493.9786682129, weight_update_time: 0.1934001148, other_time: 9.9169616699
epoch time: 62.9546279907s, input_fired_time: 1.0598556995, output_fired_time: 0.0000000000, propagate_forward_time: 60.1219711304, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7728004456
validation[          , genome  5320] predictions:    6024/  10000 (60.24%), best:    6124/10000 (61.24%), error:     12250.21191, best error:     11244.86328 on epoch:     1, epoch:   24/25, mu: 0.8784335256, learning_rate: 0.0000020486, weight_decay: 0.0000287485

epoch time: 900.8104858398s, input_fired_time: 18.1391010284, output_fired_time: 6.1752853394, propagate_forward_time: 319.5645446777, propagate_backward_time: 546.0161132813, weight_update_time: 0.2139907777, other_time: 10.9154052734
epoch time: 78.0889663696s, input_fired_time: 1.3053601980, output_fired_time: 0.0000000000, propagate_forward_time: 74.6008300781, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1827774048
validation[          , genome  5320] predictions:    6055/  10000 (60.55%), best:    6124/10000 (61.24%), error:     12271.41016, best error:     11244.86328 on epoch:     1, epoch:   25/25, mu: 0.8854200244, learning_rate: 0.0000018564, weight_decay: 0.0000266079

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 388.0912170410s, input_fired_time: 6.4801211357, output_fired_time: 0.0000000000, propagate_forward_time: 370.7597961426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.8512878418
best training[          , genome  5320] predictions:   33324/  50000 (66.65%), best:    6124/10000 (61.24%), error:     49808.63672, best error:     11244.86328 on epoch:     1, epoch:   26/25, mu: 0.8919690251, learning_rate: 0.0000016822, weight_decay: 0.0000246267
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 78.1706542969s, input_fired_time: 1.3008614779, output_fired_time: 0.0000000000, propagate_forward_time: 74.6612472534, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2085418701
   testing[          , genome  5320] predictions:    6124/  10000 (61.24%), best:    6124/10000 (61.24%), error:     11244.86328, best error:     11244.86328 on epoch:     1, epoch:   26/25, mu: 0.8919690251, learning_rate: 0.0000016822, weight_decay: 0.0000246267
backpropagation finished successfully!
18:39:49 (2680): called boinc_finish(0)

</stderr_txt>
]]>