Nameexact_genome_1542303603_170_5980_0
Workunit4091043
Created15 Nov 2018, 18:35:47 UTC
Sent15 Nov 2018, 18:49:22 UTC
Report deadline16 Dec 2018, 9:52:15 UTC
Received21 Nov 2018, 6:42:15 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID21895
Run time8 hours 1 min 35 sec
CPU time7 hours 28 min 50 sec
Validate stateValid
Credit7,656.75
Device peak FLOPS4.37 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size573.95 MB
Peak swap size572.19 MB
Peak disk usage12.41 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980_0_r1716263380_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45769
read mu: 0.45769
read mu_delta: 0.935228
read initial_learning_rate: 0.00101115
read learning_rate: 0.00101115
read learning_rate_delta: 0.903798
read initial_weight_decay: 0.000174872
read weight_decay: 0.000174872
read weight_decay_delta: 0.923078
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0491473
read input_dropout_probability: 0.0016063
read hidden_dropout_probability: 0.0770498
read velocity_reset: 1359
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5980
read normal distribution: '1 0 0'
generator_str: '960442389'
read generator: 960442389
read generated_by_map:
1 disable_edge 1
reading 120 nodes.
reading 432 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1668141071
post shuffle 1: 559786437
epoch time: 19972.79296875s, input_fired_time: 20.2992305755615, output_fired_time: 7.53524923324585, propagate_forward_time: 664.948303222656, propagate_backward_time: 19124.060546875, weight_update_time: 0.368853121995926, other_time: 155.94921875
epoch time: 142.805603027344s, input_fired_time: 1.22506237030029, output_fired_time: 0, propagate_forward_time: 111.928771972656, propagate_backward_time: 0, weight_update_time: 0, other_time: 29.6517639160156
validation[          , genome  5980] predictions:    5829/  10000 (58.29%), best:    5829/10000 (58.29%), error:     11845.20996, best error:     11845.20996 on epoch:     0, epoch:    0/25, mu: 0.4576904774, learning_rate: 0.0010111479, weight_decay: 0.0001748716

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980_0_r1716263380_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45769
read mu: 0.492169
read mu_delta: 0.935228
read initial_learning_rate: 0.00101115
read learning_rate: 0.000913874
read learning_rate_delta: 0.903798
read initial_weight_decay: 0.000174872
read weight_decay: 0.00016142
read weight_decay_delta: 0.923078
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0491473
read input_dropout_probability: 0.0016063
read hidden_dropout_probability: 0.0770498
read velocity_reset: 1359
read epoch: 1
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 5829
read best_validation_error: 11845.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5980
read normal distribution: '1 0 0'
generator_str: '872666664'
read generator: 872666664
read generated_by_map:
1 disable_edge 1
reading 120 nodes.
reading 432 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1254.8740234375s, input_fired_time: 20.6805419921875, output_fired_time: 7.53838682174683, propagate_forward_time: 481.890289306641, propagate_backward_time: 733.018127441406, weight_update_time: 0.369903653860092, other_time: 11.7467651367188
epoch time: 73.5206604003906s, input_fired_time: 1.15760004520416, output_fired_time: 0, propagate_forward_time: 70.5671768188477, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.79588317871094
validation[          , genome  5980] predictions:    5944/  10000 (59.44%), best:    5944/10000 (59.44%), error:     11757.56055, best error:     11757.56055 on epoch:     1, epoch:    1/25, mu: 0.4921693802, learning_rate: 0.0009138737, weight_decay: 0.0001614201

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980_0_r1716263380_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45769
read mu: 0.524415
read mu_delta: 0.935228
read initial_learning_rate: 0.00101115
read learning_rate: 0.000825957
read learning_rate_delta: 0.903798
read initial_weight_decay: 0.000174872
read weight_decay: 0.000149003
read weight_decay_delta: 0.923078
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0491473
read input_dropout_probability: 0.0016063
read hidden_dropout_probability: 0.0770498
read velocity_reset: 1359
read epoch: 2
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 1
read number_validation_images: 10000
read best_validation_predictions: 5944
read best_validation_error: 11757.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5980
read normal distribution: '1 0 0'
generator_str: '2068567105'
read generator: 2068567105
read generated_by_map:
1 disable_edge 1
reading 120 nodes.
reading 432 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1852.86865234375s, input_fired_time: 22.7922859191895, output_fired_time: 8.92021083831787, propagate_forward_time: 545.300109863281, propagate_backward_time: 1241.85302734375, weight_update_time: 0.408827930688858, other_time: 34.003173828125
epoch time: 243.683685302734s, input_fired_time: 1.32085490226746, output_fired_time: 0, propagate_forward_time: 240.0224609375, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.34036254882813
validation[          , genome  5980] predictions:    6091/  10000 (60.91%), best:    6091/10000 (60.91%), error:     11163.86523, best error:     11163.86523 on epoch:     2, epoch:    2/25, mu: 0.5244150162, learning_rate: 0.0008259574, weight_decay: 0.0001490034

epoch time: 1041.7099609375s, input_fired_time: 19.5869960785, output_fired_time: 7.1038737297, propagate_forward_time: 370.9279785156, propagate_backward_time: 614.0205078125, weight_update_time: 0.5293267369, other_time: 30.0706176758
epoch time: 78.9512557983s, input_fired_time: 1.2193762064, output_fired_time: 0.0000000000, propagate_forward_time: 75.7885284424, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9433517456
validation[          , genome  5980] predictions:    6015/  10000 (60.15%), best:    6091/10000 (60.91%), error:     11392.22168, best error:     11163.86523 on epoch:     2, epoch:    3/25, mu: 0.5545720458, learning_rate: 0.0007464988, weight_decay: 0.0001375417

epoch time: 966.1186523438s, input_fired_time: 18.7226276398, output_fired_time: 6.8415703773, propagate_forward_time: 352.5072021484, propagate_backward_time: 576.9938354492, weight_update_time: 0.3546568453, other_time: 11.0534057617
epoch time: 73.9274291992s, input_fired_time: 1.1796411276, output_fired_time: 0.0000000000, propagate_forward_time: 70.9324417114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8153457642
validation[          , genome  5980] predictions:    6047/  10000 (60.47%), best:    6091/10000 (60.91%), error:     11392.38867, best error:     11163.86523 on epoch:     2, epoch:    4/25, mu: 0.5827757120, learning_rate: 0.0006746843, weight_decay: 0.0001269617

epoch time: 966.0256347656s, input_fired_time: 18.7757759094, output_fired_time: 6.9971160889, propagate_forward_time: 354.2460021973, propagate_backward_time: 575.1064453125, weight_update_time: 0.3480991125, other_time: 10.9002685547
epoch time: 74.4902648926s, input_fired_time: 1.1733244658, output_fired_time: 0.0000000000, propagate_forward_time: 71.5066070557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8103332520
validation[          , genome  5980] predictions:    6045/  10000 (60.45%), best:    6091/10000 (60.91%), error:     11534.53125, best error:     11163.86523 on epoch:     2, epoch:    5/25, mu: 0.6091525555, learning_rate: 0.0006097784, weight_decay: 0.0001171956

epoch time: 1019.4855957031s, input_fired_time: 29.4454421997, output_fired_time: 6.9668803215, propagate_forward_time: 380.9940490723, propagate_backward_time: 590.8972167969, weight_update_time: 0.3545703590, other_time: 11.1820068359
epoch time: 232.2615814209s, input_fired_time: 1.5621192455, output_fired_time: 0.0000000000, propagate_forward_time: 217.9999542236, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.6995086670
validation[          , genome  5980] predictions:    6130/  10000 (61.30%), best:    6130/10000 (61.30%), error:     11103.16309, best error:     11103.16309 on epoch:     6, epoch:    6/25, mu: 0.6338209510, learning_rate: 0.0005511167, weight_decay: 0.0001081807

epoch time: 1016.3340454102s, input_fired_time: 18.8313846588, output_fired_time: 6.7846617699, propagate_forward_time: 353.3926391602, propagate_backward_time: 626.3408813477, weight_update_time: 0.3526817560, other_time: 10.9844970703
epoch time: 72.8362121582s, input_fired_time: 1.1631653309, output_fired_time: 0.0000000000, propagate_forward_time: 69.8922424316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7808074951
validation[          , genome  5980] predictions:    6175/  10000 (61.75%), best:    6130/10000 (61.30%), error:     11116.08594, best error:     11103.16309 on epoch:     6, epoch:    7/25, mu: 0.6568914652, learning_rate: 0.0004980983, weight_decay: 0.0000998592

epoch time: 984.9326171875s, input_fired_time: 18.9405841827, output_fired_time: 6.8858366013, propagate_forward_time: 358.9764709473, propagate_backward_time: 589.1179199219, weight_update_time: 0.3493897319, other_time: 11.0117187500
epoch time: 72.6758041382s, input_fired_time: 1.1511332989, output_fired_time: 0.0000000000, propagate_forward_time: 69.7525863647, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7720870972
validation[          , genome  5980] predictions:    6066/  10000 (60.66%), best:    6130/10000 (61.30%), error:     11412.60449, best error:     11103.16309 on epoch:     6, epoch:    8/25, mu: 0.6784676909, learning_rate: 0.0004501803, weight_decay: 0.0000921778

epoch time: 969.9277954102s, input_fired_time: 19.7375736237, output_fired_time: 6.9611272812, propagate_forward_time: 352.8442993164, propagate_backward_time: 579.4200439453, weight_update_time: 0.3501717448, other_time: 10.9647827148
epoch time: 76.0177993774s, input_fired_time: 1.1982388496, output_fired_time: 0.0000000000, propagate_forward_time: 72.9820480347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8375091553
validation[          , genome  5980] predictions:    6077/  10000 (60.77%), best:    6130/10000 (61.30%), error:     11424.63770, best error:     11103.16309 on epoch:     6, epoch:    9/25, mu: 0.6986463666, learning_rate: 0.0004068722, weight_decay: 0.0000850873

epoch time: 1127.6573486328s, input_fired_time: 19.8994693756, output_fired_time: 7.1767487526, propagate_forward_time: 436.3398742676, propagate_backward_time: 652.5606689453, weight_update_time: 0.3697793186, other_time: 11.6806640625
epoch time: 72.6657257080s, input_fired_time: 1.2074292898, output_fired_time: 0.0000000000, propagate_forward_time: 69.6646347046, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7936630249
validation[          , genome  5980] predictions:    6089/  10000 (60.89%), best:    6130/10000 (61.30%), error:     11208.45020, best error:     11103.16309 on epoch:     6, epoch:   10/25, mu: 0.7175180316, learning_rate: 0.0003677303, weight_decay: 0.0000785422

epoch time: 963.2739868164s, input_fired_time: 18.7968196869, output_fired_time: 6.8078579903, propagate_forward_time: 352.1170654297, propagate_backward_time: 574.5533447266, weight_update_time: 0.3646830618, other_time: 10.9989013672
epoch time: 73.8084411621s, input_fired_time: 1.2024919987, output_fired_time: 0.0000000000, propagate_forward_time: 70.7042083740, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9017410278
validation[          , genome  5980] predictions:    6140/  10000 (61.40%), best:    6130/10000 (61.30%), error:     11133.64258, best error:     11103.16309 on epoch:     6, epoch:   11/25, mu: 0.7351673245, learning_rate: 0.0003323540, weight_decay: 0.0000725006

epoch time: 983.0369873047s, input_fired_time: 19.0908145905, output_fired_time: 6.8959693909, propagate_forward_time: 351.8481445313, propagate_backward_time: 594.2952270508, weight_update_time: 0.3438349962, other_time: 10.9067993164
epoch time: 72.6037597656s, input_fired_time: 1.1693358421, output_fired_time: 0.0000000000, propagate_forward_time: 69.6470718384, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7873535156
validation[          , genome  5980] predictions:    6186/  10000 (61.86%), best:    6186/10000 (61.86%), error:     10934.32715, best error:     10934.32715 on epoch:    12, epoch:   12/25, mu: 0.7516734004, learning_rate: 0.0003003810, weight_decay: 0.0000669237

epoch time: 968.3501586914s, input_fired_time: 18.6402225494, output_fired_time: 6.7451915741, propagate_forward_time: 349.3320007324, propagate_backward_time: 582.7778320313, weight_update_time: 0.3440580964, other_time: 10.8549804688
epoch time: 73.4643707275s, input_fired_time: 1.1959856749, output_fired_time: 0.0000000000, propagate_forward_time: 70.4726791382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7957077026
validation[          , genome  5980] predictions:    6272/  10000 (62.72%), best:    6272/10000 (62.72%), error:     10815.91797, best error:     10815.91797 on epoch:    13, epoch:   13/25, mu: 0.7671103477, learning_rate: 0.0002714838, weight_decay: 0.0000617758

epoch time: 1010.5869750977s, input_fired_time: 19.0577201843, output_fired_time: 6.9611968994, propagate_forward_time: 367.5566101074, propagate_backward_time: 605.6377563477, weight_update_time: 0.3577224314, other_time: 11.3737182617
epoch time: 74.0162734985s, input_fired_time: 1.2205548286, output_fired_time: 0.0000000000, propagate_forward_time: 70.9799728394, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8157424927
validation[          , genome  5980] predictions:    6238/  10000 (62.38%), best:    6272/10000 (62.72%), error:     10829.88184, best error:     10815.91797 on epoch:    13, epoch:   14/25, mu: 0.7815474272, learning_rate: 0.0002453665, weight_decay: 0.0000570239

epoch time: 999.8461303711s, input_fired_time: 19.6920471191, output_fired_time: 7.0856852531, propagate_forward_time: 368.4036865234, propagate_backward_time: 593.3466796875, weight_update_time: 0.3543890715, other_time: 11.3179931641
epoch time: 74.3796005249s, input_fired_time: 1.2260245085, output_fired_time: 0.0000000000, propagate_forward_time: 71.3441314697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8094482422
validation[          , genome  5980] predictions:    6239/  10000 (62.39%), best:    6272/10000 (62.72%), error:     10840.27246, best error:     10815.91797 on epoch:    13, epoch:   15/25, mu: 0.7950493693, learning_rate: 0.0002217618, weight_decay: 0.0000526375

epoch time: 1062.3973388672s, input_fired_time: 19.9146804810, output_fired_time: 7.1200647354, propagate_forward_time: 381.9833068848, propagate_backward_time: 641.4047851563, weight_update_time: 0.3487202823, other_time: 11.9744873047
epoch time: 86.7456970215s, input_fired_time: 1.2054872513, output_fired_time: 0.0000000000, propagate_forward_time: 83.7097854614, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8304214478
validation[          , genome  5980] predictions:    6304/  10000 (63.04%), best:    6304/10000 (63.04%), error:     10702.71094, best error:     10702.71094 on epoch:    16, epoch:   16/25, mu: 0.8076767921, learning_rate: 0.0002004280, weight_decay: 0.0000485885

epoch time: 1012.8331909180s, input_fired_time: 18.9008769989, output_fired_time: 6.9014053345, propagate_forward_time: 374.5770874023, propagate_backward_time: 601.3948974609, weight_update_time: 0.3539494574, other_time: 11.0588989258
epoch time: 72.7174682617s, input_fired_time: 1.1870290041, output_fired_time: 0.0000000000, propagate_forward_time: 69.7547531128, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7756881714
validation[          , genome  5980] predictions:    6151/  10000 (61.51%), best:    6304/10000 (63.04%), error:     11009.01953, best error:     10702.71094 on epoch:    16, epoch:   17/25, mu: 0.8194862604, learning_rate: 0.0001811464, weight_decay: 0.0000448510

epoch time: 997.1408691406s, input_fired_time: 18.9243831635, output_fired_time: 6.8040618896, propagate_forward_time: 362.9100341797, propagate_backward_time: 597.4260864258, weight_update_time: 0.3443109989, other_time: 11.0762939453
epoch time: 73.5907745361s, input_fired_time: 1.1868731976, output_fired_time: 0.0000000000, propagate_forward_time: 70.5782775879, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8256225586
validation[          , genome  5980] predictions:    6378/  10000 (63.78%), best:    6378/10000 (63.78%), error:     10488.95996, best error:     10488.95996 on epoch:    18, epoch:   18/25, mu: 0.8305308223, learning_rate: 0.0001637198, weight_decay: 0.0000414010

epoch time: 975.5607299805s, input_fired_time: 19.1255435944, output_fired_time: 6.8537120819, propagate_forward_time: 353.0331726074, propagate_backward_time: 585.6547851563, weight_update_time: 0.3491671979, other_time: 10.8935546875
epoch time: 73.6510238647s, input_fired_time: 1.1819881201, output_fired_time: 0.0000000000, propagate_forward_time: 70.6228103638, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8462219238
validation[          , genome  5980] predictions:    6369/  10000 (63.69%), best:    6378/10000 (63.78%), error:     10516.95215, best error:     10488.95996 on epoch:    18, epoch:   19/25, mu: 0.8408600092, learning_rate: 0.0001479697, weight_decay: 0.0000382163

epoch time: 957.0578002930s, input_fired_time: 18.6470909119, output_fired_time: 6.7290177345, propagate_forward_time: 349.1042480469, propagate_backward_time: 571.7786865234, weight_update_time: 0.3435994685, other_time: 10.7987670898
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980_0_r1716263380_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45769
read mu: 0.85052
read mu_delta: 0.935228
read initial_learning_rate: 0.00101115
read learning_rate: 0.000133735
read learning_rate_delta: 0.903798
read initial_weight_decay: 0.000174872
read weight_decay: 3.52766e-05
read weight_decay_delta: 0.923078
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0491473
read input_dropout_probability: 0.0016063
read hidden_dropout_probability: 0.0770498
read velocity_reset: 1359
read epoch: 20
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 18
read number_validation_images: 10000
read best_validation_predictions: 6378
read best_validation_error: 10489
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5980
read normal distribution: '1 0 0'
generator_str: '1819504785'
read generator: 1819504785
read generated_by_map:
1 disable_edge 1
reading 120 nodes.
reading 432 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 956.370056152344s, input_fired_time: 18.7083148956299, output_fired_time: 6.73345565795898, propagate_forward_time: 349.39990234375, propagate_backward_time: 570.767517089844, weight_update_time: 0.343203783035278, other_time: 10.7608642578125
epoch time: 72.7681198120117s, input_fired_time: 1.17589366436005, output_fired_time: 0, propagate_forward_time: 69.8225173950195, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.76970672607422
validation[          , genome  5980] predictions:    6351/  10000 (63.51%), best:    6378/10000 (63.78%), error:     10638.55859, best error:     10488.95996 on epoch:    18, epoch:   20/25, mu: 0.8505201340, learning_rate: 0.0001337347, weight_decay: 0.0000352766

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542303603_170_5980_0_r1716263380_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.45769
read mu: 0.859555
read mu_delta: 0.935228
read initial_learning_rate: 0.00101115
read learning_rate: 0.000120869
read learning_rate_delta: 0.903798
read initial_weight_decay: 0.000174872
read weight_decay: 3.25631e-05
read weight_decay_delta: 0.923078
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0491473
read input_dropout_probability: 0.0016063
read hidden_dropout_probability: 0.0770498
read velocity_reset: 1359
read epoch: 21
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 18
read number_validation_images: 10000
read best_validation_predictions: 6378
read best_validation_error: 10489
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5980
read normal distribution: '1 0 0'
generator_str: '1267461696'
read generator: 1267461696
read generated_by_map:
1 disable_edge 1
reading 120 nodes.
reading 432 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1241.34704589844s, input_fired_time: 19.9125595092773, output_fired_time: 7.06194114685059, propagate_forward_time: 505.773101806641, propagate_backward_time: 696.953247070313, weight_update_time: 0.369022458791733, other_time: 11.646240234375
epoch time: 72.9292984008789s, input_fired_time: 1.17133915424347, output_fired_time: 0, propagate_forward_time: 69.958625793457, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.79933166503906
validation[          , genome  5980] predictions:    6373/  10000 (63.73%), best:    6373/10000 (63.73%), error:     10466.68555, best error:     10466.68555 on epoch:    21, epoch:   21/25, mu: 0.8595545888, learning_rate: 0.0001208692, weight_decay: 0.0000325631

epoch time: 1217.2669677734s, input_fired_time: 19.1483554840, output_fired_time: 7.0001587868, propagate_forward_time: 357.5122680664, propagate_backward_time: 822.6007690430, weight_update_time: 0.3481262028, other_time: 11.0054931641
epoch time: 72.7231216431s, input_fired_time: 1.1802600622, output_fired_time: 0.0000000000, propagate_forward_time: 69.7683181763, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7745437622
validation[          , genome  5980] predictions:    6389/  10000 (63.89%), best:    6389/10000 (63.89%), error:     10394.02930, best error:     10394.02930 on epoch:    22, epoch:   22/25, mu: 0.8680038452, learning_rate: 0.0001092414, weight_decay: 0.0000300583

epoch time: 936.4898681641s, input_fired_time: 18.3735103607, output_fired_time: 6.6457047462, propagate_forward_time: 342.1312255859, propagate_backward_time: 558.8543090820, weight_update_time: 0.3375018239, other_time: 10.4851074219
epoch time: 71.9011383057s, input_fired_time: 1.1703644991, output_fired_time: 0.0000000000, propagate_forward_time: 68.9948425293, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7359313965
validation[          , genome  5980] predictions:    6379/  10000 (63.79%), best:    6389/10000 (63.89%), error:     10410.15527, best error:     10394.02930 on epoch:    22, epoch:   23/25, mu: 0.8759058118, learning_rate: 0.0000987322, weight_decay: 0.0000277461

epoch time: 935.8018188477s, input_fired_time: 18.3667869568, output_fired_time: 6.6361222267, propagate_forward_time: 341.9683532715, propagate_backward_time: 558.3099365234, weight_update_time: 0.3393946290, other_time: 10.5206298828
epoch time: 71.6954956055s, input_fired_time: 1.1627267599, output_fired_time: 0.0000000000, propagate_forward_time: 68.7942504883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7385177612
validation[          , genome  5980] predictions:    6386/  10000 (63.86%), best:    6389/10000 (63.89%), error:     10559.57324, best error:     10394.02930 on epoch:    22, epoch:   24/25, mu: 0.8832959533, learning_rate: 0.0000892340, weight_decay: 0.0000256118

epoch time: 935.4565429688s, input_fired_time: 18.3541011810, output_fired_time: 6.6205396652, propagate_forward_time: 341.7322692871, propagate_backward_time: 558.2828369141, weight_update_time: 0.3374221921, other_time: 10.4667968750
epoch time: 71.6386871338s, input_fired_time: 1.1664359570, output_fired_time: 0.0000000000, propagate_forward_time: 68.7299346924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7423171997
validation[          , genome  5980] predictions:    6481/  10000 (64.81%), best:    6481/10000 (64.81%), error:     10328.99121, best error:     10328.99121 on epoch:    25, epoch:   25/25, mu: 0.8902074099, learning_rate: 0.0000806495, weight_decay: 0.0000236417

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 356.3430786133s, input_fired_time: 5.8203902245, output_fired_time: 0.0000000000, propagate_forward_time: 341.8214111328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.7012634277
best training[          , genome  5980] predictions:   35810/  50000 (71.62%), best:    6481/10000 (64.81%), error:     43085.51172, best error:     10328.99121 on epoch:    25, epoch:   26/25, mu: 0.8966712356, learning_rate: 0.0000728909, weight_decay: 0.0000218232
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 71.7335739136s, input_fired_time: 1.1676410437, output_fired_time: 0.0000000000, propagate_forward_time: 68.8335571289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7323760986
   testing[          , genome  5980] predictions:    6481/  10000 (64.81%), best:    6481/10000 (64.81%), error:     10328.99121, best error:     10328.99121 on epoch:    25, epoch:   26/25, mu: 0.8966712356, learning_rate: 0.0000728909, weight_decay: 0.0000218232
backpropagation finished successfully!
23:41:40 (6704): called boinc_finish(0)

</stderr_txt>
]]>