Nameexact_genome_1542495604_171_6221_3
Workunit4096540
Created24 Nov 2018, 10:15:36 UTC
Sent24 Nov 2018, 10:15:46 UTC
Report deadline25 Dec 2018, 8:29:10 UTC
Received26 Nov 2018, 7:32:39 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID78968
Run time18 hours 1 min 20 sec
CPU time17 hours 41 min 27 sec
Validate stateValid
Credit7,731.49
Device peak FLOPS2.20 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size585.12 MB
Peak swap size604.96 MB
Peak disk usage13.38 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221_3_r1019629578_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.477481
read mu: 0.477481
read mu_delta: 0.964024
read initial_learning_rate: 0.000165301
read learning_rate: 0.000165301
read learning_rate_delta: 0.923389
read initial_weight_decay: 0.000224214
read weight_decay: 0.000224214
read weight_decay_delta: 0.93638
read batch_size: 117
read epsilon: 1e-07
read alpha: 0.0964378
read input_dropout_probability: 0.001321
read hidden_dropout_probability: 0.118944
read velocity_reset: 1478
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6221
read normal distribution: '1 -0.415217757225037 -0.0329449623823166'
generator_str: '1100803519'
read generator: 1100803519
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 430 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 633124928
post shuffle 1: 585810841
epoch time: 2173.66186523438s, input_fired_time: 35.1585998535156, output_fired_time: 14.2498836517334, propagate_forward_time: 815.447021484375, propagate_backward_time: 1285.84387207031, weight_update_time: 0.55372416973114, other_time: 22.9622802734375
epoch time: 179.944702148438s, input_fired_time: 2.54827070236206, output_fired_time: 0, propagate_forward_time: 172.974502563477, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.42193603515625
validation[          , genome  6221] predictions:    6548/  10000 (65.48%), best:    6548/10000 (65.48%), error:     10203.11133, best error:     10203.11133 on epoch:     0, epoch:    0/25, mu: 0.4774805307, learning_rate: 0.0001653014, weight_decay: 0.0002242141

epoch time: 2187.5629882813s, input_fired_time: 35.5982551575, output_fired_time: 14.3485660553, propagate_forward_time: 814.7982177734, propagate_backward_time: 1299.2142333984, weight_update_time: 0.5783466697, other_time: 23.6037597656
epoch time: 169.1197662354s, input_fired_time: 2.4071550369, output_fired_time: 0.0000000000, propagate_forward_time: 162.5645141602, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1481018066
validation[          , genome  6221] predictions:    6538/  10000 (65.38%), best:    6548/10000 (65.48%), error:     10302.88965, best error:     10203.11133 on epoch:     0, epoch:    1/25, mu: 0.4959188998, learning_rate: 0.0001526375, weight_decay: 0.0002099495

epoch time: 2248.3825683594s, input_fired_time: 35.8190574646, output_fired_time: 14.6312294006, propagate_forward_time: 838.7909545898, propagate_backward_time: 1335.0532226563, weight_update_time: 0.5787118077, other_time: 24.0878906250
epoch time: 178.5238647461s, input_fired_time: 2.5602209568, output_fired_time: 0.0000000000, propagate_forward_time: 171.5985107422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3651275635
validation[          , genome  6221] predictions:    6524/  10000 (65.24%), best:    6548/10000 (65.48%), error:     10368.74805, best error:     10203.11133 on epoch:     0, epoch:    2/25, mu: 0.5136939287, learning_rate: 0.0001409437, weight_decay: 0.0001965924

epoch time: 2209.0192871094s, input_fired_time: 34.9526634216, output_fired_time: 14.3738527298, propagate_forward_time: 821.5839233398, propagate_backward_time: 1314.2757568359, weight_update_time: 0.5595294237, other_time: 23.8331298828
epoch time: 165.9208374023s, input_fired_time: 2.3859190941, output_fired_time: 0.0000000000, propagate_forward_time: 159.5832061768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9517059326
validation[          , genome  6221] predictions:    6481/  10000 (64.81%), best:    6548/10000 (65.48%), error:     10390.83008, best error:     10203.11133 on epoch:     0, epoch:    3/25, mu: 0.5308294892, learning_rate: 0.0001301459, weight_decay: 0.0001840852

epoch time: 2239.8576660156s, input_fired_time: 35.2617721558, output_fired_time: 14.6957693100, propagate_forward_time: 835.8942260742, propagate_backward_time: 1329.9017333984, weight_update_time: 0.5663671494, other_time: 24.1041259766
epoch time: 173.8071899414s, input_fired_time: 2.4515388012, output_fired_time: 0.0000000000, propagate_forward_time: 167.2213287354, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1343231201
validation[          , genome  6221] predictions:    6414/  10000 (64.14%), best:    6548/10000 (65.48%), error:     10476.74023, best error:     10203.11133 on epoch:     0, epoch:    4/25, mu: 0.5473485589, learning_rate: 0.0001201752, weight_decay: 0.0001723736

epoch time: 2310.6437988281s, input_fired_time: 36.0279388428, output_fired_time: 15.1596813202, propagate_forward_time: 869.5848388672, propagate_backward_time: 1365.3371582031, weight_update_time: 0.6045350432, other_time: 24.5343017578
epoch time: 183.7415466309s, input_fired_time: 2.6412005424, output_fired_time: 0.0000000000, propagate_forward_time: 176.3981781006, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7021636963
validation[          , genome  6221] predictions:    6497/  10000 (64.97%), best:    6548/10000 (65.48%), error:     10470.87500, best error:     10203.11133 on epoch:     0, epoch:    5/25, mu: 0.5632733703, learning_rate: 0.0001109685, weight_decay: 0.0001614071

epoch time: 2296.2070312500s, input_fired_time: 36.6472930908, output_fired_time: 15.4342174530, propagate_forward_time: 859.0875244141, propagate_backward_time: 1358.5998535156, weight_update_time: 0.6226021647, other_time: 26.4381103516
epoch time: 187.1663818359s, input_fired_time: 2.6574542522, output_fired_time: 0.0000000000, propagate_forward_time: 180.0738830566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4350433350
validation[          , genome  6221] predictions:    6361/  10000 (63.61%), best:    6548/10000 (65.48%), error:     10676.30176, best error:     10203.11133 on epoch:     0, epoch:    6/25, mu: 0.5786252618, learning_rate: 0.0001024671, weight_decay: 0.0001511383

epoch time: 2254.9497070313s, input_fired_time: 35.4858131409, output_fired_time: 14.6447229385, propagate_forward_time: 845.8425292969, propagate_backward_time: 1334.6212158203, weight_update_time: 0.5593215227, other_time: 24.3553466797
epoch time: 187.2696228027s, input_fired_time: 2.6000001431, output_fired_time: 0.0000000000, propagate_forward_time: 180.2238311768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4457855225
validation[          , genome  6221] predictions:    6447/  10000 (64.47%), best:    6548/10000 (65.48%), error:     10573.06738, best error:     10203.11133 on epoch:     0, epoch:    7/25, mu: 0.5934248567, learning_rate: 0.0000946169, weight_decay: 0.0001415229

epoch time: 2323.3491210938s, input_fired_time: 37.8046112061, output_fired_time: 16.6220512390, propagate_forward_time: 877.1668701172, propagate_backward_time: 1364.3809814453, weight_update_time: 0.6351550817, other_time: 27.3745117188
epoch time: 176.3506164551s, input_fired_time: 2.7992155552, output_fired_time: 0.0000000000, propagate_forward_time: 169.0378723145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5135345459
validation[          , genome  6221] predictions:    6275/  10000 (62.75%), best:    6548/10000 (65.48%), error:     10872.89844, best error:     10203.11133 on epoch:     0, epoch:    8/25, mu: 0.6076920033, learning_rate: 0.0000873682, weight_decay: 0.0001325191

epoch time: 2258.6489257813s, input_fired_time: 35.9349594116, output_fired_time: 15.5114421844, propagate_forward_time: 846.8208007813, propagate_backward_time: 1335.7182617188, weight_update_time: 0.5805873871, other_time: 24.6633300781
epoch time: 168.9545593262s, input_fired_time: 2.4969787598, output_fired_time: 0.0000000000, propagate_forward_time: 162.2263946533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2311859131
validation[          , genome  6221] predictions:    6199/  10000 (61.99%), best:    6548/10000 (65.48%), error:     11098.10449, best error:     10203.11133 on epoch:     0, epoch:    9/25, mu: 0.6214458942, learning_rate: 0.0000806748, weight_decay: 0.0001240882

epoch time: 2209.4914550781s, input_fired_time: 35.5608139038, output_fired_time: 14.7536764145, propagate_forward_time: 825.2291259766, propagate_backward_time: 1309.6566162109, weight_update_time: 0.6107252836, other_time: 24.2912597656
epoch time: 178.8074493408s, input_fired_time: 2.5221126080, output_fired_time: 0.0000000000, propagate_forward_time: 171.9483337402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3370056152
validation[          , genome  6221] predictions:    6294/  10000 (62.94%), best:    6548/10000 (65.48%), error:     10975.20313, best error:     10203.11133 on epoch:     0, epoch:   10/25, mu: 0.6347049475, learning_rate: 0.0000744943, weight_decay: 0.0001161937

epoch time: 2276.5275878906s, input_fired_time: 40.1458663940, output_fired_time: 17.2735671997, propagate_forward_time: 847.8354492188, propagate_backward_time: 1340.6066894531, weight_update_time: 0.6795734167, other_time: 30.6660156250
epoch time: 192.0371856689s, input_fired_time: 2.6872949600, output_fired_time: 0.0000000000, propagate_forward_time: 184.9037628174, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4461212158
validation[          , genome  6221] predictions:    6249/  10000 (62.49%), best:    6548/10000 (65.48%), error:     10923.86621, best error:     10203.11133 on epoch:     0, epoch:   11/25, mu: 0.6474870443, learning_rate: 0.0000687872, weight_decay: 0.0001088014

epoch time: 2235.5139160156s, input_fired_time: 35.1504592896, output_fired_time: 14.4496793747, propagate_forward_time: 835.9334716797, propagate_backward_time: 1326.1567382813, weight_update_time: 0.5511109829, other_time: 23.8236083984
epoch time: 189.0392913818s, input_fired_time: 2.5573930740, output_fired_time: 0.0000000000, propagate_forward_time: 182.0790863037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4028167725
validation[          , genome  6221] predictions:    6285/  10000 (62.85%), best:    6548/10000 (65.48%), error:     11059.41699, best error:     10203.11133 on epoch:     0, epoch:   12/25, mu: 0.6598092914, learning_rate: 0.0000635173, weight_decay: 0.0001018794

epoch time: 2334.1298828125s, input_fired_time: 37.4559593201, output_fired_time: 15.9530553818, propagate_forward_time: 867.4485473633, propagate_backward_time: 1386.6425781250, weight_update_time: 0.6125971675, other_time: 26.6296386719
epoch time: 189.2817077637s, input_fired_time: 2.8899500370, output_fired_time: 0.0000000000, propagate_forward_time: 181.5670776367, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8246765137
validation[          , genome  6221] predictions:    6340/  10000 (63.40%), best:    6548/10000 (65.48%), error:     10856.37793, best error:     10203.11133 on epoch:     0, epoch:   13/25, mu: 0.6716881990, learning_rate: 0.0000586512, weight_decay: 0.0000953978

epoch time: 2261.8198242188s, input_fired_time: 36.4663200378, output_fired_time: 14.9458274841, propagate_forward_time: 842.7224121094, propagate_backward_time: 1343.0445556641, weight_update_time: 0.6060595512, other_time: 24.6407470703
epoch time: 180.1953430176s, input_fired_time: 2.4701831341, output_fired_time: 0.0000000000, propagate_forward_time: 173.4676818848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2574768066
validation[          , genome  6221] predictions:    6049/  10000 (60.49%), best:    6548/10000 (65.48%), error:     11664.61719, best error:     10203.11133 on epoch:     0, epoch:   14/25, mu: 0.6831397414, learning_rate: 0.0000541578, weight_decay: 0.0000893285

epoch time: 2340.5817871094s, input_fired_time: 37.4612655640, output_fired_time: 15.8574666977, propagate_forward_time: 870.1176147461, propagate_backward_time: 1391.1787109375, weight_update_time: 0.5980916619, other_time: 25.9667968750
epoch time: 180.6261444092s, input_fired_time: 2.4652652740, output_fired_time: 0.0000000000, propagate_forward_time: 173.8258972168, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3349761963
validation[          , genome  6221] predictions:    6178/  10000 (61.78%), best:    6548/10000 (65.48%), error:     11204.46582, best error:     10203.11133 on epoch:     0, epoch:   15/25, mu: 0.6941792965, learning_rate: 0.0000500087, weight_decay: 0.0000836454

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221_3_r1019629578_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.477481
read mu: 0.704822
read mu_delta: 0.964024
read initial_learning_rate: 0.000165301
read learning_rate: 4.61775e-05
read learning_rate_delta: 0.923389
read initial_weight_decay: 0.000224214
read weight_decay: 7.83239e-05
read weight_decay_delta: 0.93638
read batch_size: 117
read epsilon: 1e-07
read alpha: 0.0964378
read input_dropout_probability: 0.001321
read hidden_dropout_probability: 0.118944
read velocity_reset: 1478
read epoch: 16
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6548
read best_validation_error: 10203.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6221
read normal distribution: '1 -0.415217757225037 -0.0329449623823166'
generator_str: '1524039999'
read generator: 1524039999
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 430 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2116.39135742188s, input_fired_time: 34.1544227600098, output_fired_time: 13.6919384002686, propagate_forward_time: 785.665466308594, propagate_backward_time: 1259.93395996094, weight_update_time: 0.550131022930145, other_time: 22.9454345703125
epoch time: 167.138092041016s, input_fired_time: 2.44096493721008, output_fired_time: 0, propagate_forward_time: 160.581024169922, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.11610412597656
validation[          , genome  6221] predictions:    6330/  10000 (63.30%), best:    6548/10000 (65.48%), error:     10927.86328, best error:     10203.11133 on epoch:     0, epoch:   16/25, mu: 0.7048217058, learning_rate: 0.0000461775, weight_decay: 0.0000783239

epoch time: 2238.1953125000s, input_fired_time: 35.4614715576, output_fired_time: 14.7601919174, propagate_forward_time: 836.4505615234, propagate_backward_time: 1327.2994384766, weight_update_time: 0.5608934760, other_time: 24.2236328125
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221_3_r1019629578_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.477481
read mu: 0.715081
read mu_delta: 0.964024
read initial_learning_rate: 0.000165301
read learning_rate: 4.26398e-05
read learning_rate_delta: 0.923389
read initial_weight_decay: 0.000224214
read weight_decay: 7.33409e-05
read weight_decay_delta: 0.93638
read batch_size: 117
read epsilon: 1e-07
read alpha: 0.0964378
read input_dropout_probability: 0.001321
read hidden_dropout_probability: 0.118944
read velocity_reset: 1478
read epoch: 17
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6548
read best_validation_error: 10203.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6221
read normal distribution: '1 -0.415217757225037 -0.0329449623823166'
generator_str: '465180048'
read generator: 465180048
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 430 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221_3_r1019629578_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.477481
read mu: 0.715081
read mu_delta: 0.964024
read initial_learning_rate: 0.000165301
read learning_rate: 4.26398e-05
read learning_rate_delta: 0.923389
read initial_weight_decay: 0.000224214
read weight_decay: 7.33409e-05
read weight_decay_delta: 0.93638
read batch_size: 117
read epsilon: 1e-07
read alpha: 0.0964378
read input_dropout_probability: 0.001321
read hidden_dropout_probability: 0.118944
read velocity_reset: 1478
read epoch: 17
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6548
read best_validation_error: 10203.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6221
read normal distribution: '1 -0.415217757225037 -0.0329449623823166'
generator_str: '465180048'
read generator: 465180048
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 430 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2129.36840820313s, input_fired_time: 34.0191116333008, output_fired_time: 13.8584079742432, propagate_forward_time: 793.60498046875, propagate_backward_time: 1258.21350097656, weight_update_time: 0.521701753139496, other_time: 29.6724853515625
epoch time: 170.86003112793s, input_fired_time: 2.37153673171997, output_fired_time: 0, propagate_forward_time: 163.354904174805, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.13359069824219
validation[          , genome  6221] predictions:    6313/  10000 (63.13%), best:    6548/10000 (65.48%), error:     10899.94629, best error:     10203.11133 on epoch:     0, epoch:   17/25, mu: 0.7150812745, learning_rate: 0.0000426398, weight_decay: 0.0000733409

epoch time: 2017.2094726563s, input_fired_time: 33.3515625000, output_fired_time: 12.9553794861, propagate_forward_time: 746.4868164063, propagate_backward_time: 1203.0251464844, weight_update_time: 0.4984299242, other_time: 21.3906250000
epoch time: 166.0633087158s, input_fired_time: 2.3952109814, output_fired_time: 0.0000000000, propagate_forward_time: 159.7590332031, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9090576172
validation[          , genome  6221] predictions:    6339/  10000 (63.39%), best:    6548/10000 (65.48%), error:     10887.57324, best error:     10203.11133 on epoch:     0, epoch:   18/25, mu: 0.7249717116, learning_rate: 0.0000393731, weight_decay: 0.0000686749

epoch time: 2266.9697265625s, input_fired_time: 36.7163047791, output_fired_time: 14.8213043213, propagate_forward_time: 849.5545654297, propagate_backward_time: 1341.3431396484, weight_update_time: 0.5796728730, other_time: 24.5344238281
epoch time: 182.5924835205s, input_fired_time: 2.6243476868, output_fired_time: 0.0000000000, propagate_forward_time: 175.4429931641, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5251464844
validation[          , genome  6221] predictions:    6208/  10000 (62.08%), best:    6548/10000 (65.48%), error:     11076.86230, best error:     10203.11133 on epoch:     0, epoch:   19/25, mu: 0.7345063686, learning_rate: 0.0000363567, weight_decay: 0.0000643058

epoch time: 2312.9106445313s, input_fired_time: 37.1860656738, output_fired_time: 15.3142709732, propagate_forward_time: 864.2688598633, propagate_backward_time: 1371.0765380859, weight_update_time: 0.5759267807, other_time: 25.0650634766
epoch time: 188.9713287354s, input_fired_time: 2.6540730000, output_fired_time: 0.0000000000, propagate_forward_time: 181.5345458984, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7827148438
validation[          , genome  6221] predictions:    6266/  10000 (62.66%), best:    6548/10000 (65.48%), error:     10983.60938, best error:     10203.11133 on epoch:     0, epoch:   20/25, mu: 0.7436980009, learning_rate: 0.0000335714, weight_decay: 0.0000602146

epoch time: 2389.8361816406s, input_fired_time: 37.5992622375, output_fired_time: 15.8522615433, propagate_forward_time: 900.8292846680, propagate_backward_time: 1408.8264160156, weight_update_time: 0.6346930861, other_time: 26.7287597656
epoch time: 182.8297882080s, input_fired_time: 2.6016893387, output_fired_time: 0.0000000000, propagate_forward_time: 175.4588165283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.7692871094
validation[          , genome  6221] predictions:    6098/  10000 (60.98%), best:    6548/10000 (65.48%), error:     11241.81445, best error:     10203.11133 on epoch:     0, epoch:   21/25, mu: 0.7525589466, learning_rate: 0.0000309994, weight_decay: 0.0000563837

epoch time: 2403.2036132813s, input_fired_time: 38.2324676514, output_fired_time: 16.3859558105, propagate_forward_time: 900.2260742188, propagate_backward_time: 1421.4193115234, weight_update_time: 0.6266364455, other_time: 26.9398193359
epoch time: 180.9978027344s, input_fired_time: 2.6379523277, output_fired_time: 0.0000000000, propagate_forward_time: 173.8132324219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5466156006
validation[          , genome  6221] predictions:    6081/  10000 (60.81%), best:    6548/10000 (65.48%), error:     11374.10059, best error:     10203.11133 on epoch:     0, epoch:   22/25, mu: 0.7611011267, learning_rate: 0.0000286245, weight_decay: 0.0000527966

epoch time: 2353.5285644531s, input_fired_time: 39.4284286499, output_fired_time: 17.0733108521, propagate_forward_time: 880.4141235352, propagate_backward_time: 1387.0921630859, weight_update_time: 0.6696254015, other_time: 29.5206298828
epoch time: 190.3767852783s, input_fired_time: 3.0081293583, output_fired_time: 0.0000000000, propagate_forward_time: 182.4171600342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9514923096
validation[          , genome  6221] predictions:    6131/  10000 (61.31%), best:    6548/10000 (65.48%), error:     11552.72168, best error:     10203.11133 on epoch:     0, epoch:   23/25, mu: 0.7693359852, learning_rate: 0.0000264316, weight_decay: 0.0000494376

epoch time: 2468.6713867188s, input_fired_time: 39.2560081482, output_fired_time: 16.7057056427, propagate_forward_time: 925.8753662109, propagate_backward_time: 1458.2467041016, weight_update_time: 0.6381030679, other_time: 28.5874023438
epoch time: 186.8065338135s, input_fired_time: 2.6737837791, output_fired_time: 0.0000000000, propagate_forward_time: 179.3145446777, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8182067871
validation[          , genome  6221] predictions:    6089/  10000 (60.89%), best:    6548/10000 (65.48%), error:     11286.14355, best error:     10203.11133 on epoch:     0, epoch:   24/25, mu: 0.7772746086, learning_rate: 0.0000244066, weight_decay: 0.0000462924

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_171_6221_3_r1019629578_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.477481
read mu: 0.784928
read mu_delta: 0.964024
read initial_learning_rate: 0.000165301
read learning_rate: 2.25368e-05
read learning_rate_delta: 0.923389
read initial_weight_decay: 0.000224214
read weight_decay: 4.33473e-05
read weight_decay_delta: 0.93638
read batch_size: 117
read epsilon: 1e-07
read alpha: 0.0964378
read input_dropout_probability: 0.001321
read hidden_dropout_probability: 0.118944
read velocity_reset: 1478
read epoch: 25
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6548
read best_validation_error: 10203.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6221
read normal distribution: '1 -0.415217757225037 -0.0329449623823166'
generator_str: '1150769592'
read generator: 1150769592
read generated_by_map:
1 split_edge 1
reading 112 nodes.
reading 430 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2388.95141601563s, input_fired_time: 37.1506729125977, output_fired_time: 15.7153882980347, propagate_forward_time: 902.910705566406, propagate_backward_time: 1407.06396484375, weight_update_time: 0.67842572927475, other_time: 26.11083984375
epoch time: 183.728210449219s, input_fired_time: 2.56199169158936, output_fired_time: 0, propagate_forward_time: 176.607925415039, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.55828857421875
validation[          , genome  6221] predictions:    6361/  10000 (63.61%), best:    6548/10000 (65.48%), error:     10933.18457, best error:     10203.11133 on epoch:     0, epoch:   25/25, mu: 0.7849276066, learning_rate: 0.0000225368, weight_decay: 0.0000433473

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 935.4046020508s, input_fired_time: 16.0327529907, output_fired_time: 0.0000000000, propagate_forward_time: 890.2772827148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 29.0945434570
best training[          , genome  6221] predictions:   36632/  50000 (73.26%), best:    6548/10000 (65.48%), error:     41934.00781, best error:     10203.11133 on epoch:     0, epoch:   26/25, mu: 0.7923052907, learning_rate: 0.0000208102, weight_decay: 0.0000405895
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 192.9091186523s, input_fired_time: 4.0424857140, output_fired_time: 0.0000000000, propagate_forward_time: 181.9468078613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.9198303223
   testing[          , genome  6221] predictions:    6548/  10000 (65.48%), best:    6548/10000 (65.48%), error:     10203.11133, best error:     10203.11133 on epoch:     0, epoch:   26/25, mu: 0.7923052907, learning_rate: 0.0000208102, weight_decay: 0.0000405895
backpropagation finished successfully!
22:31:21 (8992): called boinc_finish(0)

</stderr_txt>
]]>