Nameexact_genome_1542692703_187_6381_0
Workunit4100425
Created20 Nov 2018, 7:41:57 UTC
Sent20 Nov 2018, 8:02:54 UTC
Report deadline28 Feb 2019, 18:54:10 UTC
Received27 Nov 2018, 12:35:05 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID75701
Run time1 days 1 hours 43 min 40 sec
CPU time1 days 1 hours 27 min 23 sec
Validate stateValid
Credit25,113.07
Device peak FLOPS3.58 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34

Stderr output

<core_client_version>7.0.80</core_client_version>
<![CDATA[
<stderr_txt>
rd_time: 813.5309448242, propagate_backward_time: 1161.2957763672, weight_update_time: 0.4705229700, other_time: 28.2166748047
epoch time: 167.3794097900s, input_fired_time: 2.9781250954, output_fired_time: 0.0000000000, propagate_forward_time: 161.3526153564, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0486755371
validation[          , genome  6381] predictions:    6152/  10000 (61.52%), best:    6134/10000 (61.34%), error:     11295.73242, best error:     11130.23438 on epoch:    38, epoch:   42/100, mu: 0.9414788485, learning_rate: 0.0000150604, weight_decay: 0.0000576476

epoch time: 2046.1651611328s, input_fired_time: 27.7025337219, output_fired_time: 16.0582580566, propagate_forward_time: 807.8917236328, propagate_backward_time: 1157.0212402344, weight_update_time: 2.4419286251, other_time: 37.4914550781
epoch time: 170.2331237793s, input_fired_time: 0.9032803178, output_fired_time: 0.0000000000, propagate_forward_time: 165.2994689941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0303802490
validation[          , genome  6381] predictions:    6130/  10000 (61.30%), best:    6130/10000 (61.30%), error:     11127.73730, best error:     11127.73730 on epoch:    43, epoch:   43/100, mu: 0.9440714717, learning_rate: 0.0000139806, weight_decay: 0.0000545949

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381_0_r358195053_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502954
read mu: 0.946526
read mu_delta: 0.946567
read initial_learning_rate: 0.000342713
read learning_rate: 1.29782e-05
read learning_rate_delta: 0.9283
read initial_weight_decay: 0.000566535
read weight_decay: 5.17038e-05
read weight_decay_delta: 0.947044
read batch_size: 88
read epsilon: 1e-07
read alpha: 0.0780389
read input_dropout_probability: 0.0011895
read hidden_dropout_probability: 0.0957836
read velocity_reset: 1904
read epoch: 44
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 43
read number_validation_images: 10000
read best_validation_predictions: 6130
read best_validation_error: 11127.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6381
read normal distribution: '1 0.418137460947037 0.409919023513794'
generator_str: '1725415769'
read generator: 1725415769
read generated_by_map:
1 change_size 1
reading 89 nodes.
reading 371 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2144.62866210938s, input_fired_time: 41.4531097412109, output_fired_time: 12.2687473297119, propagate_forward_time: 836.7978515625, propagate_backward_time: 1226.20166015625, weight_update_time: 0.51615971326828, other_time: 27.9072265625
epoch time: 180.148178100586s, input_fired_time: 1.04338657855988, output_fired_time: 0, propagate_forward_time: 173.852172851563, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.25262451171875
validation[          , genome  6381] predictions:    6222/  10000 (62.22%), best:    6222/10000 (62.22%), error:     11036.10254, best error:     11036.10254 on epoch:    44, epoch:   44/100, mu: 0.9465255737, learning_rate: 0.0000129782, weight_decay: 0.0000517038

epoch time: 2092.1481933594s, input_fired_time: 29.1889286041, output_fired_time: 21.1413402557, propagate_forward_time: 812.1053466797, propagate_backward_time: 1199.1506347656, weight_update_time: 0.5101698041, other_time: 30.5618896484
epoch time: 168.3224639893s, input_fired_time: 3.9403591156, output_fired_time: 0.0000000000, propagate_forward_time: 160.2350158691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1470947266
validation[          , genome  6381] predictions:    6126/  10000 (61.26%), best:    6222/10000 (62.22%), error:     11254.60449, best error:     11036.10254 on epoch:    44, epoch:   45/100, mu: 0.9488485456, learning_rate: 0.0000120476, weight_decay: 0.0000489658

epoch time: 2054.4165039063s, input_fired_time: 34.7742500305, output_fired_time: 16.1179485321, propagate_forward_time: 809.6118164063, propagate_backward_time: 1174.6394042969, weight_update_time: 0.5094686151, other_time: 19.2730712891
epoch time: 169.5554199219s, input_fired_time: 0.9194986224, output_fired_time: 0.0000000000, propagate_forward_time: 161.5608825684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0750427246
validation[          , genome  6381] predictions:    6264/  10000 (62.64%), best:    6264/10000 (62.64%), error:     10887.84961, best error:     10887.84961 on epoch:    46, epoch:   46/100, mu: 0.9510473609, learning_rate: 0.0000111838, weight_decay: 0.0000463727

epoch time: 2065.9562988281s, input_fired_time: 29.0555858612, output_fired_time: 15.9758129120, propagate_forward_time: 809.9042358398, propagate_backward_time: 1185.3463134766, weight_update_time: 0.5201941729, other_time: 25.6744384766
epoch time: 169.2882080078s, input_fired_time: 0.9368827939, output_fired_time: 0.0000000000, propagate_forward_time: 160.1825866699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1687316895
validation[          , genome  6381] predictions:    6086/  10000 (60.86%), best:    6264/10000 (62.64%), error:     11242.71582, best error:     10887.84961 on epoch:    46, epoch:   47/100, mu: 0.9531286955, learning_rate: 0.0000103819, weight_decay: 0.0000439171

epoch time: 2105.8737792969s, input_fired_time: 37.1564674377, output_fired_time: 12.2401494980, propagate_forward_time: 804.8620605469, propagate_backward_time: 1211.7270507813, weight_update_time: 2.6137802601, other_time: 39.8879394531
epoch time: 168.6247253418s, input_fired_time: 2.8928384781, output_fired_time: 0.0000000000, propagate_forward_time: 163.6202392578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1116485596
validation[          , genome  6381] predictions:    6214/  10000 (62.14%), best:    6264/10000 (62.64%), error:     10968.68457, best error:     10887.84961 on epoch:    46, epoch:   48/100, mu: 0.9550988078, learning_rate: 0.0000096375, weight_decay: 0.0000415914

epoch time: 2125.9934082031s, input_fired_time: 35.3372039795, output_fired_time: 15.0803937912, propagate_forward_time: 839.0498046875, propagate_backward_time: 1194.4024658203, weight_update_time: 0.5519452095, other_time: 42.1236572266
epoch time: 175.2620544434s, input_fired_time: 1.9865804911, output_fired_time: 0.0000000000, propagate_forward_time: 165.1371002197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1383666992
validation[          , genome  6381] predictions:    5971/  10000 (59.71%), best:    6264/10000 (62.64%), error:     11627.35449, best error:     10887.84961 on epoch:    46, epoch:   49/100, mu: 0.9569636583, learning_rate: 0.0000089465, weight_decay: 0.0000393889

epoch time: 2082.7495117188s, input_fired_time: 28.7168579102, output_fired_time: 12.9983358383, propagate_forward_time: 832.4113769531, propagate_backward_time: 1175.9552001953, weight_update_time: 4.7098660469, other_time: 32.6678466797
epoch time: 167.9716491699s, input_fired_time: 4.2009520531, output_fired_time: 0.0000000000, propagate_forward_time: 159.7235412598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0471496582
validation[          , genome  6381] predictions:    6223/  10000 (62.23%), best:    6264/10000 (62.64%), error:     11002.42383, best error:     10887.84961 on epoch:    46, epoch:   50/100, mu: 0.9587289095, learning_rate: 0.0000083051, weight_decay: 0.0000373030

epoch time: 2075.5488281250s, input_fired_time: 42.7680053711, output_fired_time: 11.0619440079, propagate_forward_time: 798.0640258789, propagate_backward_time: 1195.1774902344, weight_update_time: 1.5392234325, other_time: 28.4772949219
epoch time: 170.0181732178s, input_fired_time: 2.0095379353, output_fired_time: 0.0000000000, propagate_forward_time: 162.0175476074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9910888672
validation[          , genome  6381] predictions:    6145/  10000 (61.45%), best:    6264/10000 (62.64%), error:     11005.10938, best error:     10887.84961 on epoch:    46, epoch:   51/100, mu: 0.9603998065, learning_rate: 0.0000077096, weight_decay: 0.0000353276

epoch time: 2058.3264160156s, input_fired_time: 43.7857894897, output_fired_time: 11.8397302628, propagate_forward_time: 796.4187622070, propagate_backward_time: 1172.8577880859, weight_update_time: 3.6126534939, other_time: 33.4244384766
epoch time: 170.2239990234s, input_fired_time: 0.9216282964, output_fired_time: 0.0000000000, propagate_forward_time: 162.1105499268, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1918182373
validation[          , genome  6381] predictions:    6281/  10000 (62.81%), best:    6264/10000 (62.64%), error:     10978.38281, best error:     10887.84961 on epoch:    46, epoch:   52/100, mu: 0.9619814157, learning_rate: 0.0000071568, weight_decay: 0.0000334568

epoch time: 2060.5913085938s, input_fired_time: 29.9165115356, output_fired_time: 19.3726158142, propagate_forward_time: 827.1411132813, propagate_backward_time: 1156.6315917969, weight_update_time: 0.5659320951, other_time: 27.5295410156
epoch time: 169.1096343994s, input_fired_time: 0.9212958217, output_fired_time: 0.0000000000, propagate_forward_time: 159.0251617432, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.1631774902
validation[          , genome  6381] predictions:    6281/  10000 (62.81%), best:    6281/10000 (62.81%), error:     10846.31152, best error:     10846.31152 on epoch:    53, epoch:   53/100, mu: 0.9634785056, learning_rate: 0.0000066437, weight_decay: 0.0000316851

epoch time: 2123.6867675781s, input_fired_time: 37.2075653076, output_fired_time: 14.9961433411, propagate_forward_time: 825.9591674805, propagate_backward_time: 1217.9157714844, weight_update_time: 3.5687532425, other_time: 27.6081542969
epoch time: 167.7413482666s, input_fired_time: 0.9227150083, output_fired_time: 0.0000000000, propagate_forward_time: 163.7913360596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0272979736
validation[          , genome  6381] predictions:    6240/  10000 (62.40%), best:    6281/10000 (62.81%), error:     10911.87305, best error:     10846.31152 on epoch:    53, epoch:   54/100, mu: 0.9648956060, learning_rate: 0.0000061673, weight_decay: 0.0000300072

epoch time: 2052.9660644531s, input_fired_time: 36.7615699768, output_fired_time: 14.1172199249, propagate_forward_time: 822.6524658203, propagate_backward_time: 1163.1907958984, weight_update_time: 0.5808016658, other_time: 16.2440185547
epoch time: 168.9605102539s, input_fired_time: 1.9042205811, output_fired_time: 0.0000000000, propagate_forward_time: 162.0579833984, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9983062744
validation[          , genome  6381] predictions:    6200/  10000 (62.00%), best:    6281/10000 (62.81%), error:     10975.84863, best error:     10846.31152 on epoch:    53, epoch:   55/100, mu: 0.9662370086, learning_rate: 0.0000057251, weight_decay: 0.0000284182

epoch time: 2053.3181152344s, input_fired_time: 24.8781738281, output_fired_time: 21.1123046875, propagate_forward_time: 805.4767456055, propagate_backward_time: 1167.4559326172, weight_update_time: 2.6612319946, other_time: 34.3948974609
epoch time: 168.7680206299s, input_fired_time: 0.9195549488, output_fired_time: 0.0000000000, propagate_forward_time: 160.7734069824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0750579834
validation[          , genome  6381] predictions:    6197/  10000 (61.97%), best:    6281/10000 (62.81%), error:     11060.14746, best error:     10846.31152 on epoch:    53, epoch:   56/100, mu: 0.9675067067, learning_rate: 0.0000053146, weight_decay: 0.0000269133

epoch time: 2049.6669921875s, input_fired_time: 30.6527576447, output_fired_time: 14.9922828674, propagate_forward_time: 820.8469848633, propagate_backward_time: 1155.6738281250, weight_update_time: 0.5908589959, other_time: 27.5012207031
epoch time: 169.9585418701s, input_fired_time: 1.9043240547, output_fired_time: 0.0000000000, propagate_forward_time: 157.1596374512, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.8945770264
validation[          , genome  6381] predictions:    6281/  10000 (62.81%), best:    6281/10000 (62.81%), error:     10840.27441, best error:     10840.27441 on epoch:    57, epoch:   57/100, mu: 0.9687085748, learning_rate: 0.0000049336, weight_decay: 0.0000254881

epoch time: 2053.1589355469s, input_fired_time: 38.9702720642, output_fired_time: 10.7309341431, propagate_forward_time: 792.1853637695, propagate_backward_time: 1174.7078857422, weight_update_time: 6.7155008316, other_time: 36.5645751953
epoch time: 167.7546691895s, input_fired_time: 2.8872866631, output_fired_time: 0.0000000000, propagate_forward_time: 159.8672332764, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0001525879
validation[          , genome  6381] predictions:    6306/  10000 (63.06%), best:    6281/10000 (62.81%), error:     10903.06152, best error:     10840.27441 on epoch:    57, epoch:   58/100, mu: 0.9698462486, learning_rate: 0.0000045798, weight_decay: 0.0000241383

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381_0_r358195053_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502954
read mu: 0.970923
read mu_delta: 0.946567
read initial_learning_rate: 0.000342713
read learning_rate: 4.25145e-06
read learning_rate_delta: 0.9283
read initial_weight_decay: 0.000566535
read weight_decay: 2.28601e-05
read weight_decay_delta: 0.947044
read batch_size: 88
read epsilon: 1e-07
read alpha: 0.0780389
read input_dropout_probability: 0.0011895
read hidden_dropout_probability: 0.0957836
read velocity_reset: 1904
read epoch: 59
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 57
read number_validation_images: 10000
read best_validation_predictions: 6281
read best_validation_error: 10840.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6381
read normal distribution: '1 0.418137460947037 0.409919023513794'
generator_str: '1238780700'
read generator: 1238780700
read generated_by_map:
1 change_size 1
reading 89 nodes.
reading 371 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2129.66967773438s, input_fired_time: 30.3100547790527, output_fired_time: 10.1157884597778, propagate_forward_time: 836.365661621094, propagate_backward_time: 1229.03125, weight_update_time: 0.639851450920105, other_time: 23.846923828125
epoch time: 172.102951049805s, input_fired_time: 1.94626832008362, output_fired_time: 0, propagate_forward_time: 165.045974731445, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.11070251464844
validation[          , genome  6381] predictions:    6244/  10000 (62.44%), best:    6281/10000 (62.81%), error:     10843.52734, best error:     10840.27441 on epoch:    57, epoch:   59/100, mu: 0.9709231257, learning_rate: 0.0000042514, weight_decay: 0.0000228601

epoch time: 2096.8395996094s, input_fired_time: 30.9677982330, output_fired_time: 13.9451112747, propagate_forward_time: 825.8388061523, propagate_backward_time: 1194.5399169922, weight_update_time: 1.6232311726, other_time: 31.5479736328
epoch time: 173.0766601563s, input_fired_time: 0.9444438815, output_fired_time: 0.0000000000, propagate_forward_time: 166.0283813477, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1038360596
validation[          , genome  6381] predictions:    6324/  10000 (63.24%), best:    6281/10000 (62.81%), error:     10897.56250, best error:     10840.27441 on epoch:    57, epoch:   60/100, mu: 0.9719424844, learning_rate: 0.0000039466, weight_decay: 0.0000216495

epoch time: 2105.3171386719s, input_fired_time: 39.0052909851, output_fired_time: 9.9586296082, propagate_forward_time: 830.2495727539, propagate_backward_time: 1188.5462646484, weight_update_time: 0.6285090446, other_time: 37.5572509766
epoch time: 172.0627899170s, input_fired_time: 0.9425565600, output_fired_time: 0.0000000000, propagate_forward_time: 167.0172576904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1029815674
validation[          , genome  6381] predictions:    6186/  10000 (61.86%), best:    6281/10000 (62.81%), error:     11007.43066, best error:     10840.27441 on epoch:    57, epoch:   61/100, mu: 0.9729073644, learning_rate: 0.0000036636, weight_decay: 0.0000205030

epoch time: 2112.2307128906s, input_fired_time: 41.0988960266, output_fired_time: 16.0148105621, propagate_forward_time: 823.4945678711, propagate_backward_time: 1198.7703857422, weight_update_time: 0.6375426054, other_time: 32.8519287109
epoch time: 167.7017364502s, input_fired_time: 1.9076775312, output_fired_time: 0.0000000000, propagate_forward_time: 157.5004882813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2935638428
validation[          , genome  6381] predictions:    6261/  10000 (62.61%), best:    6281/10000 (62.81%), error:     10921.87988, best error:     10840.27441 on epoch:    57, epoch:   62/100, mu: 0.9738206863, learning_rate: 0.0000034010, weight_decay: 0.0000194173

epoch time: 2045.2639160156s, input_fired_time: 41.3739662170, output_fired_time: 18.0608940125, propagate_forward_time: 798.7429199219, propagate_backward_time: 1154.4952392578, weight_update_time: 0.6174325943, other_time: 32.5908203125
epoch time: 167.7319488525s, input_fired_time: 0.9215533137, output_fired_time: 0.0000000000, propagate_forward_time: 162.6804809570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1299133301
validation[          , genome  6381] predictions:    6274/  10000 (62.74%), best:    6274/10000 (62.74%), error:     10788.81445, best error:     10788.81445 on epoch:    63, epoch:   63/100, mu: 0.9746851921, learning_rate: 0.0000031571, weight_decay: 0.0000183890

epoch time: 2097.7719726563s, input_fired_time: 42.2838706970, output_fired_time: 19.0777416229, propagate_forward_time: 816.9188232422, propagate_backward_time: 1192.6782226563, weight_update_time: 2.6042983532, other_time: 26.8132324219
epoch time: 172.9506072998s, input_fired_time: 2.9786827564, output_fired_time: 0.0000000000, propagate_forward_time: 166.7783660889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1935577393
validation[          , genome  6381] predictions:    6256/  10000 (62.56%), best:    6274/10000 (62.74%), error:     10904.33008, best error:     10788.81445 on epoch:    63, epoch:   64/100, mu: 0.9755035043, learning_rate: 0.0000029307, weight_decay: 0.0000174152

epoch time: 2054.8205566406s, input_fired_time: 38.9157371521, output_fired_time: 8.8570899963, propagate_forward_time: 819.9804077148, propagate_backward_time: 1167.6501464844, weight_update_time: 0.6256591678, other_time: 19.4172363281
epoch time: 167.9193420410s, input_fired_time: 4.8532629013, output_fired_time: 0.0000000000, propagate_forward_time: 155.8956298828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1704559326
validation[          , genome  6381] predictions:    6267/  10000 (62.67%), best:    6274/10000 (62.74%), error:     10868.35352, best error:     10788.81445 on epoch:    63, epoch:   65/100, mu: 0.9762780666, learning_rate: 0.0000027206, weight_decay: 0.0000164930

epoch time: 2049.9294433594s, input_fired_time: 40.1271896362, output_fired_time: 14.8605432510, propagate_forward_time: 803.2189941406, propagate_backward_time: 1165.2243652344, weight_update_time: 0.6217253804, other_time: 26.4982910156
epoch time: 167.3716278076s, input_fired_time: 3.8723938465, output_fired_time: 0.0000000000, propagate_forward_time: 159.4892883301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0099487305
validation[          , genome  6381] predictions:    6340/  10000 (63.40%), best:    6340/10000 (63.40%), error:     10740.54785, best error:     10740.54785 on epoch:    66, epoch:   66/100, mu: 0.9770112634, learning_rate: 0.0000025255, weight_decay: 0.0000156196

epoch time: 2043.4389648438s, input_fired_time: 38.8256225586, output_fired_time: 12.8947944641, propagate_forward_time: 782.2963867188, propagate_backward_time: 1168.9433593750, weight_update_time: 1.6060458422, other_time: 40.4787597656
epoch time: 174.2104492188s, input_fired_time: 2.9465296268, output_fired_time: 0.0000000000, propagate_forward_time: 163.1848144531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.0791015625
validation[          , genome  6381] predictions:    6290/  10000 (62.90%), best:    6340/10000 (63.40%), error:     10789.68457, best error:     10740.54785 on epoch:    66, epoch:   67/100, mu: 0.9777052999, learning_rate: 0.0000023445, weight_decay: 0.0000147925

epoch time: 2048.5258789063s, input_fired_time: 43.8040428162, output_fired_time: 17.9140262604, propagate_forward_time: 794.4879150391, propagate_backward_time: 1157.8037109375, weight_update_time: 2.7076129913, other_time: 34.5161132813
epoch time: 171.6838378906s, input_fired_time: 1.9070606232, output_fired_time: 0.0000000000, propagate_forward_time: 166.6435699463, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1332092285
validation[          , genome  6381] predictions:    6189/  10000 (61.89%), best:    6340/10000 (63.40%), error:     11058.41797, best error:     10740.54785 on epoch:    66, epoch:   68/100, mu: 0.9783622622, learning_rate: 0.0000021764, weight_decay: 0.0000140091

epoch time: 2115.5251464844s, input_fired_time: 42.1818008423, output_fired_time: 13.1861886978, propagate_forward_time: 837.1777343750, propagate_backward_time: 1198.0882568359, weight_update_time: 0.6447134614, other_time: 24.8909912109
epoch time: 167.7529144287s, input_fired_time: 1.9098881483, output_fired_time: 0.0000000000, propagate_forward_time: 160.8363800049, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0066528320
validation[          , genome  6381] predictions:    6248/  10000 (62.48%), best:    6340/10000 (63.40%), error:     10860.98340, best error:     10740.54785 on epoch:    66, epoch:   69/100, mu: 0.9789841175, learning_rate: 0.0000020203, weight_decay: 0.0000132673

epoch time: 2046.9210205078s, input_fired_time: 27.5998802185, output_fired_time: 20.8882770538, propagate_forward_time: 794.1052246094, propagate_backward_time: 1177.2637939453, weight_update_time: 2.5906519890, other_time: 27.0638427734
epoch time: 167.4804687500s, input_fired_time: 7.0401015282, output_fired_time: 0.0000000000, propagate_forward_time: 156.4342041016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0061645508
validation[          , genome  6381] predictions:    6301/  10000 (63.01%), best:    6340/10000 (63.40%), error:     10767.77344, best error:     10740.54785 on epoch:    66, epoch:   70/100, mu: 0.9795727134, learning_rate: 0.0000018755, weight_decay: 0.0000125647

epoch time: 2047.9686279297s, input_fired_time: 25.6283130646, output_fired_time: 10.7097454071, propagate_forward_time: 825.4349365234, propagate_backward_time: 1159.6882324219, weight_update_time: 0.6244300008, other_time: 26.5074462891
epoch time: 169.0330505371s, input_fired_time: 2.0214874744, output_fired_time: 0.0000000000, propagate_forward_time: 158.8554534912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.1561126709
validation[          , genome  6381] predictions:    6277/  10000 (62.77%), best:    6340/10000 (63.40%), error:     10828.73047, best error:     10740.54785 on epoch:    66, epoch:   71/100, mu: 0.9801298976, learning_rate: 0.0000017410, weight_decay: 0.0000118993

epoch time: 2045.2156982422s, input_fired_time: 31.7588920593, output_fired_time: 20.0288085938, propagate_forward_time: 799.9536743164, propagate_backward_time: 1164.9548339844, weight_update_time: 0.6268047690, other_time: 28.5195312500
epoch time: 167.6992340088s, input_fired_time: 2.8929073811, output_fired_time: 0.0000000000, propagate_forward_time: 160.6860656738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1202545166
validation[          , genome  6381] predictions:    6294/  10000 (62.94%), best:    6340/10000 (63.40%), error:     10814.02246, best error:     10740.54785 on epoch:    66, epoch:   72/100, mu: 0.9806572795, learning_rate: 0.0000016162, weight_decay: 0.0000112692

epoch time: 2043.3201904297s, input_fired_time: 27.7109928131, output_fired_time: 10.8201036453, propagate_forward_time: 810.2832641602, propagate_backward_time: 1148.7874755859, weight_update_time: 3.6812603474, other_time: 45.7183837891
epoch time: 167.7140655518s, input_fired_time: 3.0041105747, output_fired_time: 0.0000000000, propagate_forward_time: 162.6660461426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0439147949
validation[          , genome  6381] predictions:    6329/  10000 (63.29%), best:    6340/10000 (63.40%), error:     10803.51855, best error:     10740.54785 on epoch:    66, epoch:   73/100, mu: 0.9811564684, learning_rate: 0.0000015003, weight_decay: 0.0000106724

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381_0_r358195053_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502954
read mu: 0.981629
read mu_delta: 0.946567
read initial_learning_rate: 0.000342713
read learning_rate: 1.39271e-06
read learning_rate_delta: 0.9283
read initial_weight_decay: 0.000566535
read weight_decay: 1.01072e-05
read weight_decay_delta: 0.947044
read batch_size: 88
read epsilon: 1e-07
read alpha: 0.0780389
read input_dropout_probability: 0.0011895
read hidden_dropout_probability: 0.0957836
read velocity_reset: 1904
read epoch: 74
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 66
read number_validation_images: 10000
read best_validation_predictions: 6340
read best_validation_error: 10740.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6381
read normal distribution: '1 0.418137460947037 0.409919023513794'
generator_str: '353799180'
read generator: 353799180
read generated_by_map:
1 change_size 1
reading 89 nodes.
reading 371 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2123.50122070313s, input_fired_time: 42.0524406433105, output_fired_time: 11.0457983016968, propagate_forward_time: 821.694519042969, propagate_backward_time: 1215.58410644531, weight_update_time: 2.7244918346405, other_time: 33.1241455078125
epoch time: 167.820846557617s, input_fired_time: 3.00462698936462, output_fired_time: 0, propagate_forward_time: 158.628448486328, propagate_backward_time: 0, weight_update_time: 0, other_time: 6.18777465820313
validation[          , genome  6381] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10730.36230, best error:     10730.36230 on epoch:    74, epoch:   74/100, mu: 0.9816290140, learning_rate: 0.0000013927, weight_decay: 0.0000101072

epoch time: 2043.2701416016s, input_fired_time: 31.6623401642, output_fired_time: 12.7851057053, propagate_forward_time: 804.4385375977, propagate_backward_time: 1157.2319335938, weight_update_time: 2.5884568691, other_time: 37.1520996094
epoch time: 168.8602447510s, input_fired_time: 2.8931028843, output_fired_time: 0.0000000000, propagate_forward_time: 160.9835662842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9835815430
validation[          , genome  6381] predictions:    6286/  10000 (62.86%), best:    6317/10000 (63.17%), error:     10841.55957, best error:     10730.36230 on epoch:    74, epoch:   75/100, mu: 0.9820762873, learning_rate: 0.0000012929, weight_decay: 0.0000095720

epoch time: 2042.8802490234s, input_fired_time: 26.8462734222, output_fired_time: 11.7985229492, propagate_forward_time: 817.8109130859, propagate_backward_time: 1159.1007080078, weight_update_time: 2.6968717575, other_time: 27.3237304688
epoch time: 164.7148590088s, input_fired_time: 0.9286538363, output_fired_time: 0.0000000000, propagate_forward_time: 160.7725067139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0137023926
validation[          , genome  6381] predictions:    6289/  10000 (62.89%), best:    6317/10000 (63.17%), error:     10763.12109, best error:     10730.36230 on epoch:    74, epoch:   76/100, mu: 0.9824996591, learning_rate: 0.0000012002, weight_decay: 0.0000090651

epoch time: 2042.0810546875s, input_fired_time: 36.0294799805, output_fired_time: 19.8810367584, propagate_forward_time: 785.5004882813, propagate_backward_time: 1165.3779296875, weight_update_time: 0.6223174334, other_time: 35.2921142578
epoch time: 168.7704620361s, input_fired_time: 1.9114559889, output_fired_time: 0.0000000000, propagate_forward_time: 163.7290649414, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1299438477
validation[          , genome  6381] predictions:    6295/  10000 (62.95%), best:    6317/10000 (63.17%), error:     10795.25195, best error:     10730.36230 on epoch:    74, epoch:   77/100, mu: 0.9829004407, learning_rate: 0.0000011141, weight_decay: 0.0000085851

epoch time: 2040.5341796875s, input_fired_time: 45.0930976868, output_fired_time: 12.9999170303, propagate_forward_time: 798.3938598633, propagate_backward_time: 1157.8048095703, weight_update_time: 0.6219153404, other_time: 26.2425537109
epoch time: 168.1011352539s, input_fired_time: 3.0050404072, output_fired_time: 0.0000000000, propagate_forward_time: 162.0788726807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0172271729
validation[          , genome  6381] predictions:    6353/  10000 (63.53%), best:    6353/10000 (63.53%), error:     10724.00098, best error:     10724.00098 on epoch:    78, epoch:   78/100, mu: 0.9832797647, learning_rate: 0.0000010342, weight_decay: 0.0000081304

epoch time: 2045.3140869141s, input_fired_time: 38.8649063110, output_fired_time: 14.8545837402, propagate_forward_time: 806.4901733398, propagate_backward_time: 1156.6997070313, weight_update_time: 0.6214050651, other_time: 28.4047851563
epoch time: 167.3507690430s, input_fired_time: 0.9281107783, output_fired_time: 0.0000000000, propagate_forward_time: 162.4231414795, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9995117188
validation[          , genome  6381] predictions:    6340/  10000 (63.40%), best:    6353/10000 (63.53%), error:     10735.12793, best error:     10724.00098 on epoch:    78, epoch:   79/100, mu: 0.9836388230, learning_rate: 0.0000009601, weight_decay: 0.0000076999

epoch time: 2042.8728027344s, input_fired_time: 28.6975879669, output_fired_time: 9.8283157349, propagate_forward_time: 805.9881591797, propagate_backward_time: 1173.9886474609, weight_update_time: 1.6049405336, other_time: 24.3699951172
epoch time: 167.4500732422s, input_fired_time: 0.9281686544, output_fired_time: 0.0000000000, propagate_forward_time: 159.3629455566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1589660645
validation[          , genome  6381] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10703.47266, best error:     10703.47266 on epoch:    80, epoch:   80/100, mu: 0.9839787483, learning_rate: 0.0000008912, weight_decay: 0.0000072921

epoch time: 2045.9328613281s, input_fired_time: 33.9377784729, output_fired_time: 18.7818641663, propagate_forward_time: 796.2109375000, propagate_backward_time: 1152.5445556641, weight_update_time: 0.6218107939, other_time: 44.4577636719
epoch time: 168.7647399902s, input_fired_time: 0.9274062514, output_fired_time: 0.0000000000, propagate_forward_time: 158.9220581055, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.9152832031
validation[          , genome  6381] predictions:    6346/  10000 (63.46%), best:    6317/10000 (63.17%), error:     10753.08203, best error:     10703.47266 on epoch:    80, epoch:   81/100, mu: 0.9843004942, learning_rate: 0.0000008273, weight_decay: 0.0000069060

epoch time: 2045.7830810547s, input_fired_time: 35.0357131958, output_fired_time: 15.8311958313, propagate_forward_time: 787.3784790039, propagate_backward_time: 1166.7817382813, weight_update_time: 2.6971511841, other_time: 40.7558593750
epoch time: 167.5736999512s, input_fired_time: 1.9102023840, output_fired_time: 0.0000000000, propagate_forward_time: 161.6669616699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9965362549
validation[          , genome  6381] predictions:    6332/  10000 (63.32%), best:    6332/10000 (63.32%), error:     10692.43164, best error:     10692.43164 on epoch:    82, epoch:   82/100, mu: 0.9846050143, learning_rate: 0.0000007680, weight_decay: 0.0000065403

epoch time: 2046.0748291016s, input_fired_time: 27.7163448334, output_fired_time: 19.8733806610, propagate_forward_time: 823.1635131836, propagate_backward_time: 1150.8546142578, weight_update_time: 1.6048498154, other_time: 24.4669189453
epoch time: 167.5798187256s, input_fired_time: 3.8780322075, output_fired_time: 0.0000000000, propagate_forward_time: 159.7048187256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9969635010
validation[          , genome  6381] predictions:    6303/  10000 (63.03%), best:    6332/10000 (63.32%), error:     10752.75781, best error:     10692.43164 on epoch:    82, epoch:   83/100, mu: 0.9848932624, learning_rate: 0.0000007129, weight_decay: 0.0000061939

epoch time: 2090.6318359375s, input_fired_time: 33.4281921387, output_fired_time: 9.0528230667, propagate_forward_time: 829.3364257813, propagate_backward_time: 1180.2357177734, weight_update_time: 3.7106866837, other_time: 38.5787353516
epoch time: 170.4693908691s, input_fired_time: 0.9515107274, output_fired_time: 0.0000000000, propagate_forward_time: 162.3512725830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.1666107178
validation[          , genome  6381] predictions:    6364/  10000 (63.64%), best:    6332/10000 (63.32%), error:     10703.22852, best error:     10692.43164 on epoch:    82, epoch:   84/100, mu: 0.9851661325, learning_rate: 0.0000006618, weight_decay: 0.0000058659

epoch time: 2058.2209472656s, input_fired_time: 32.9569969177, output_fired_time: 9.7943277359, propagate_forward_time: 804.5493774414, propagate_backward_time: 1184.6546630859, weight_update_time: 0.6268709898, other_time: 26.2655029297
epoch time: 168.8520660400s, input_fired_time: 1.9127929211, output_fired_time: 0.0000000000, propagate_forward_time: 162.8300170898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1092529297
validation[          , genome  6381] predictions:    6275/  10000 (62.75%), best:    6332/10000 (63.32%), error:     10746.71680, best error:     10692.43164 on epoch:    82, epoch:   85/100, mu: 0.9854243994, learning_rate: 0.0000006144, weight_decay: 0.0000055553

epoch time: 2049.5322265625s, input_fired_time: 43.1603126526, output_fired_time: 11.9518337250, propagate_forward_time: 800.8193969727, propagate_backward_time: 1160.1546630859, weight_update_time: 2.6991744041, other_time: 33.4461669922
epoch time: 167.2392120361s, input_fired_time: 0.9410682917, output_fired_time: 0.0000000000, propagate_forward_time: 162.0494384766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2487030029
validation[          , genome  6381] predictions:    6335/  10000 (63.35%), best:    6335/10000 (63.35%), error:     10658.11426, best error:     10658.11426 on epoch:    86, epoch:   86/100, mu: 0.9856688976, learning_rate: 0.0000005703, weight_decay: 0.0000052611

epoch time: 2048.0791015625s, input_fired_time: 25.7096672058, output_fired_time: 14.7605819702, propagate_forward_time: 802.5406494141, propagate_backward_time: 1165.8291015625, weight_update_time: 1.6049141884, other_time: 39.2390136719
epoch time: 167.8937835693s, input_fired_time: 2.8950800896, output_fired_time: 0.0000000000, propagate_forward_time: 161.9812011719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0175018311
validation[          , genome  6381] predictions:    6319/  10000 (63.19%), best:    6335/10000 (63.35%), error:     10712.13477, best error:     10658.11426 on epoch:    86, epoch:   87/100, mu: 0.9859003425, learning_rate: 0.0000005294, weight_decay: 0.0000049825

epoch time: 2051.3269042969s, input_fired_time: 33.5373878479, output_fired_time: 20.2083206177, propagate_forward_time: 794.3970336914, propagate_backward_time: 1170.7309570313, weight_update_time: 2.5875041485, other_time: 32.4531250000
epoch time: 167.7845611572s, input_fired_time: 0.9325492978, output_fired_time: 0.0000000000, propagate_forward_time: 163.7226562500, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1293487549
validation[          , genome  6381] predictions:    6348/  10000 (63.48%), best:    6335/10000 (63.35%), error:     10685.09473, best error:     10658.11426 on epoch:    86, epoch:   88/100, mu: 0.9861193895, learning_rate: 0.0000004915, weight_decay: 0.0000047187

epoch time: 2048.3754882813s, input_fired_time: 33.6395111084, output_fired_time: 9.8328170776, propagate_forward_time: 801.0095825195, propagate_backward_time: 1168.4838867188, weight_update_time: 2.6974649429, other_time: 35.4096679688
epoch time: 167.7730407715s, input_fired_time: 0.9304391742, output_fired_time: 0.0000000000, propagate_forward_time: 160.8720703125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9705352783
validation[          , genome  6381] predictions:    6362/  10000 (63.62%), best:    6335/10000 (63.35%), error:     10751.28809, best error:     10658.11426 on epoch:    86, epoch:   89/100, mu: 0.9863267541, learning_rate: 0.0000004562, weight_decay: 0.0000044688

epoch time: 2044.7485351563s, input_fired_time: 33.9543724060, output_fired_time: 18.1308135986, propagate_forward_time: 816.3185424805, propagate_backward_time: 1149.1456298828, weight_update_time: 0.6218749881, other_time: 27.1990966797
epoch time: 167.4762725830s, input_fired_time: 0.9316318035, output_fired_time: 0.0000000000, propagate_forward_time: 163.5280914307, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0165557861
validation[          , genome  6381] predictions:    6370/  10000 (63.70%), best:    6335/10000 (63.35%), error:     10696.42676, best error:     10658.11426 on epoch:    86, epoch:   90/100, mu: 0.9865230322, learning_rate: 0.0000004235, weight_decay: 0.0000042321

epoch time: 2047.6207275391s, input_fired_time: 38.1053657532, output_fired_time: 15.0712003708, propagate_forward_time: 800.7581787109, propagate_backward_time: 1162.2352294922, weight_update_time: 0.6210076213, other_time: 31.4508056641
epoch time: 167.3561096191s, input_fired_time: 0.9326069355, output_fired_time: 0.0000000000, propagate_forward_time: 162.3195343018, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1039733887
validation[          , genome  6381] predictions:    6358/  10000 (63.58%), best:    6335/10000 (63.35%), error:     10731.45117, best error:     10658.11426 on epoch:    86, epoch:   91/100, mu: 0.9867088199, learning_rate: 0.0000003932, weight_decay: 0.0000040080

epoch time: 2047.4248046875s, input_fired_time: 26.6372680664, output_fired_time: 18.0172519684, propagate_forward_time: 808.2947387695, propagate_backward_time: 1173.1757812500, weight_update_time: 0.6218997240, other_time: 21.2998046875
epoch time: 167.2674102783s, input_fired_time: 1.9148081541, output_fired_time: 0.0000000000, propagate_forward_time: 159.2775878906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0750122070
validation[          , genome  6381] predictions:    6360/  10000 (63.60%), best:    6335/10000 (63.35%), error:     10723.87402, best error:     10658.11426 on epoch:    86, epoch:   92/100, mu: 0.9868846536, learning_rate: 0.0000003650, weight_decay: 0.0000037958

epoch time: 2043.0537109375s, input_fired_time: 36.9119606018, output_fired_time: 18.3416156769, propagate_forward_time: 785.3035888672, propagate_backward_time: 1168.3198242188, weight_update_time: 0.6216700673, other_time: 34.1766357422
epoch time: 168.6686706543s, input_fired_time: 4.1010098457, output_fired_time: 0.0000000000, propagate_forward_time: 159.5867767334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9808807373
validation[          , genome  6381] predictions:    6361/  10000 (63.61%), best:    6335/10000 (63.35%), error:     10680.68164, best error:     10658.11426 on epoch:    86, epoch:   93/100, mu: 0.9870511293, learning_rate: 0.0000003388, weight_decay: 0.0000035948

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_187_6381_0_r358195053_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.502954
read mu: 0.987209
read mu_delta: 0.946567
read initial_learning_rate: 0.000342713
read learning_rate: 3.14504e-07
read learning_rate_delta: 0.9283
read initial_weight_decay: 0.000566535
read weight_decay: 3.4044e-06
read weight_decay_delta: 0.947044
read batch_size: 88
read epsilon: 1e-07
read alpha: 0.0780389
read input_dropout_probability: 0.0011895
read hidden_dropout_probability: 0.0957836
read velocity_reset: 1904
read epoch: 94
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 86
read number_validation_images: 10000
read best_validation_predictions: 6335
read best_validation_error: 10658.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6381
read normal distribution: '1 0.418137460947037 0.409919023513794'
generator_str: '1965430118'
read generator: 1965430118
read generated_by_map:
1 change_size 1
reading 89 nodes.
reading 371 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2122.73657226563s, input_fired_time: 42.463996887207, output_fired_time: 11.1070499420166, propagate_forward_time: 856.669006347656, propagate_backward_time: 1184.80517578125, weight_update_time: 2.61296153068542, other_time: 27.69140625
epoch time: 168.924102783203s, input_fired_time: 2.90250658988953, output_fired_time: 0, propagate_forward_time: 160.772491455078, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.24909973144531
validation[          , genome  6381] predictions:    6330/  10000 (63.30%), best:    6335/10000 (63.35%), error:     10710.21582, best error:     10658.11426 on epoch:    86, epoch:   94/100, mu: 0.9872087240, learning_rate: 0.0000003145, weight_decay: 0.0000034044

epoch time: 2053.0932617188s, input_fired_time: 36.2065773010, output_fired_time: 8.8992042542, propagate_forward_time: 820.4298706055, propagate_backward_time: 1160.2302246094, weight_update_time: 0.6271538138, other_time: 27.3273925781
epoch time: 170.0456085205s, input_fired_time: 2.0281336308, output_fired_time: 0.0000000000, propagate_forward_time: 160.0454864502, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9719848633
validation[          , genome  6381] predictions:    6345/  10000 (63.45%), best:    6335/10000 (63.35%), error:     10729.62012, best error:     10658.11426 on epoch:    86, epoch:   95/100, mu: 0.9873578548, learning_rate: 0.0000002920, weight_decay: 0.0000032241

epoch time: 2048.7907714844s, input_fired_time: 43.1645965576, output_fired_time: 16.1808128357, propagate_forward_time: 785.1452026367, propagate_backward_time: 1165.5561523438, weight_update_time: 0.6252593994, other_time: 38.7441406250
epoch time: 167.8038940430s, input_fired_time: 2.0245354176, output_fired_time: 0.0000000000, propagate_forward_time: 163.7146301270, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0647277832
validation[          , genome  6381] predictions:    6375/  10000 (63.75%), best:    6335/10000 (63.35%), error:     10707.29688, best error:     10658.11426 on epoch:    86, epoch:   96/100, mu: 0.9874990582, learning_rate: 0.0000002710, weight_decay: 0.0000030534

epoch time: 2048.5061035156s, input_fired_time: 44.8279914856, output_fired_time: 12.9296293259, propagate_forward_time: 792.8869628906, propagate_backward_time: 1172.0970458984, weight_update_time: 0.6230179071, other_time: 25.7644042969
epoch time: 168.0692596436s, input_fired_time: 0.9431288242, output_fired_time: 0.0000000000, propagate_forward_time: 165.0527343750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0733947754
validation[          , genome  6381] predictions:    6357/  10000 (63.57%), best:    6335/10000 (63.35%), error:     10718.05176, best error:     10658.11426 on epoch:    86, epoch:   97/100, mu: 0.9876326919, learning_rate: 0.0000002516, weight_decay: 0.0000028917

epoch time: 2043.9307861328s, input_fired_time: 34.1043052673, output_fired_time: 14.7009887695, propagate_forward_time: 810.0208740234, propagate_backward_time: 1162.7172851563, weight_update_time: 2.5930178165, other_time: 22.3873291016
epoch time: 167.7869262695s, input_fired_time: 2.8985154629, output_fired_time: 0.0000000000, propagate_forward_time: 161.7275085449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1609039307
validation[          , genome  6381] predictions:    6331/  10000 (63.31%), best:    6335/10000 (63.35%), error:     10707.98926, best error:     10658.11426 on epoch:    86, epoch:   98/100, mu: 0.9877591729, learning_rate: 0.0000002335, weight_decay: 0.0000027386

epoch time: 2049.1174316406s, input_fired_time: 36.0361824036, output_fired_time: 10.8222265244, propagate_forward_time: 811.5130004883, propagate_backward_time: 1167.2144775391, weight_update_time: 0.6224191189, other_time: 23.5316162109
epoch time: 167.8430175781s, input_fired_time: 3.0068225861, output_fired_time: 0.0000000000, propagate_forward_time: 161.7791137695, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0570831299
validation[          , genome  6381] predictions:    6373/  10000 (63.73%), best:    6335/10000 (63.35%), error:     10689.54004, best error:     10658.11426 on epoch:    86, epoch:   99/100, mu: 0.9878789186, learning_rate: 0.0000002168, weight_decay: 0.0000025935

epoch time: 2053.3825683594s, input_fired_time: 35.9348220825, output_fired_time: 18.0319271088, propagate_forward_time: 789.7561645508, propagate_backward_time: 1171.0635986328, weight_update_time: 3.6819453239, other_time: 38.5960693359
epoch time: 167.4146423340s, input_fired_time: 2.8986628056, output_fired_time: 0.0000000000, propagate_forward_time: 158.4261169434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0898590088
validation[          , genome  6381] predictions:    6328/  10000 (63.28%), best:    6335/10000 (63.35%), error:     10766.26953, best error:     10658.11426 on epoch:    86, epoch:  100/100, mu: 0.9879922271, learning_rate: 0.0000002013, weight_decay: 0.0000024562

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 836.9775390625s, input_fired_time: 12.7366771698, output_fired_time: 0.0000000000, propagate_forward_time: 789.8132324219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 34.4276123047
best training[          , genome  6381] predictions:   33511/  50000 (67.02%), best:    6335/10000 (63.35%), error:     50107.94531, best error:     10658.11426 on epoch:    86, epoch:  101/100, mu: 0.9880995154, learning_rate: 0.0000001868, weight_decay: 0.0000023261
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 168.6724090576s, input_fired_time: 0.9315229654, output_fired_time: 0.0000000000, propagate_forward_time: 165.6857604980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0551300049
   testing[          , genome  6381] predictions:    6335/  10000 (63.35%), best:    6335/10000 (63.35%), error:     10658.11426, best error:     10658.11426 on epoch:    86, epoch:  101/100, mu: 0.9880995154, learning_rate: 0.0000001868, weight_decay: 0.0000023261
backpropagation finished successfully!
13:33:46 (2868): called boinc_finish(0)

</stderr_txt>
]]>