Nameexact_genome_1542495604_170_6199_0
Workunit4096250
Created18 Nov 2018, 4:25:57 UTC
Sent18 Nov 2018, 5:02:38 UTC
Report deadline14 Dec 2018, 4:09:16 UTC
Received19 Nov 2018, 20:48:27 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID13757
Run time6 hours 44 min 20 sec
CPU time6 hours 31 min 13 sec
Validate stateValid
Credit6,490.74
Device peak FLOPS4.19 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size569.43 MB
Peak swap size589.24 MB
Peak disk usage11.37 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199_0_r464034326_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.462051
read mu: 0.462051
read mu_delta: 0.93722
read initial_learning_rate: 1e-05
read learning_rate: 1e-05
read learning_rate_delta: 0.90631
read initial_weight_decay: 0.000185071
read weight_decay: 0.000185071
read weight_decay_delta: 0.925315
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0511267
read input_dropout_probability: 0.00153756
read hidden_dropout_probability: 0.0786986
read velocity_reset: 1392
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6199
read normal distribution: '1 0 0'
generator_str: '1634961895'
read generator: 1634961895
read generated_by_map:
1 enable_node 1
reading 104 nodes.
reading 381 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1751305900
post shuffle 1: 570328872
epoch time: 843.225891113281s, input_fired_time: 15.6666107177734, output_fired_time: 5.59378862380981, propagate_forward_time: 308.556549072266, propagate_backward_time: 504.00341796875, weight_update_time: 0.229972809553146, other_time: 9.405517578125
epoch time: 65.8825531005859s, input_fired_time: 1.03133118152618, output_fired_time: 0, propagate_forward_time: 63.2444763183594, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.60674285888672
validation[          , genome  6199] predictions:    6294/  10000 (62.94%), best:    6294/10000 (62.94%), error:     10763.84082, best error:     10763.84082 on epoch:     0, epoch:    0/25, mu: 0.4620509446, learning_rate: 0.0000100000, weight_decay: 0.0001850711

epoch time: 838.2504272461s, input_fired_time: 15.5252742767, output_fired_time: 5.5534744263, propagate_forward_time: 306.7961730957, propagate_backward_time: 501.1124267578, weight_update_time: 0.2278996408, other_time: 9.2630920410
epoch time: 63.5345191956s, input_fired_time: 0.9658684731, output_fired_time: 0.0000000000, propagate_forward_time: 61.0413017273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5273475647
validation[          , genome  6199] predictions:    6286/  10000 (62.86%), best:    6294/10000 (62.94%), error:     10821.78418, best error:     10763.84082 on epoch:     0, epoch:    1/25, mu: 0.4951954186, learning_rate: 0.0000090631, weight_decay: 0.0001712492

epoch time: 852.9738769531s, input_fired_time: 15.9280195236, output_fired_time: 5.7088441849, propagate_forward_time: 311.8562927246, propagate_backward_time: 509.9369506836, weight_update_time: 0.2329846919, other_time: 9.5437622070
epoch time: 63.9872703552s, input_fired_time: 1.0330761671, output_fired_time: 0.0000000000, propagate_forward_time: 61.4365692139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5176239014
validation[          , genome  6199] predictions:    6285/  10000 (62.85%), best:    6294/10000 (62.94%), error:     10943.55664, best error:     10763.84082 on epoch:     0, epoch:    2/25, mu: 0.5262590647, learning_rate: 0.0000082140, weight_decay: 0.0001584595

epoch time: 844.9815673828s, input_fired_time: 15.7357301712, output_fired_time: 5.6449866295, propagate_forward_time: 308.4223632813, propagate_backward_time: 505.8104858398, weight_update_time: 0.2291938066, other_time: 9.3680419922
epoch time: 65.6050338745s, input_fired_time: 1.0175367594, output_fired_time: 0.0000000000, propagate_forward_time: 62.9585418701, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6289520264
validation[          , genome  6199] predictions:    6288/  10000 (62.88%), best:    6294/10000 (62.94%), error:     11040.05762, best error:     10763.84082 on epoch:     0, epoch:    3/25, mu: 0.5553725362, learning_rate: 0.0000074444, weight_decay: 0.0001466250

epoch time: 835.2047119141s, input_fired_time: 15.4695339203, output_fired_time: 5.4952054024, propagate_forward_time: 305.2558898926, propagate_backward_time: 499.7593383789, weight_update_time: 0.2335387319, other_time: 9.2247619629
epoch time: 65.7279510498s, input_fired_time: 1.0014693737, output_fired_time: 0.0000000000, propagate_forward_time: 63.1574897766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5689888000
validation[          , genome  6199] predictions:    6289/  10000 (62.89%), best:    6294/10000 (62.94%), error:     11174.58398, best error:     10763.84082 on epoch:     0, epoch:    4/25, mu: 0.5826582909, learning_rate: 0.0000067469, weight_decay: 0.0001356744

epoch time: 845.3628540039s, input_fired_time: 15.7199068069, output_fired_time: 5.6136636734, propagate_forward_time: 309.1246337891, propagate_backward_time: 505.5235290527, weight_update_time: 0.2318883836, other_time: 9.3811340332
epoch time: 63.4065361023s, input_fired_time: 0.9652270675, output_fired_time: 0.0000000000, propagate_forward_time: 60.9251632690, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5161476135
validation[          , genome  6199] predictions:    6268/  10000 (62.68%), best:    6294/10000 (62.94%), error:     11330.34863, best error:     10763.84082 on epoch:     0, epoch:    5/25, mu: 0.6082310677, learning_rate: 0.0000061148, weight_decay: 0.0001255416

epoch time: 848.1543579102s, input_fired_time: 15.8519763947, output_fired_time: 5.6681175232, propagate_forward_time: 310.2312927246, propagate_backward_time: 506.9284667969, weight_update_time: 0.2310642600, other_time: 9.4744873047
epoch time: 65.6072769165s, input_fired_time: 1.0220470428, output_fired_time: 0.0000000000, propagate_forward_time: 62.9873237610, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5979042053
validation[          , genome  6199] predictions:    6257/  10000 (62.57%), best:    6294/10000 (62.94%), error:     11441.21875, best error:     10763.84082 on epoch:     0, epoch:    6/25, mu: 0.6321983933, learning_rate: 0.0000055419, weight_decay: 0.0001161656

epoch time: 833.2966308594s, input_fired_time: 15.4287643433, output_fired_time: 5.4861574173, propagate_forward_time: 304.6954040527, propagate_backward_time: 498.4638977051, weight_update_time: 0.2256255895, other_time: 9.2224121094
epoch time: 65.5654067993s, input_fired_time: 1.0069065094, output_fired_time: 0.0000000000, propagate_forward_time: 62.9604530334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5980491638
validation[          , genome  6199] predictions:    6266/  10000 (62.66%), best:    6294/10000 (62.94%), error:     11568.76758, best error:     10763.84082 on epoch:     0, epoch:    7/25, mu: 0.6546610594, learning_rate: 0.0000050227, weight_decay: 0.0001074899

epoch time: 840.8851928711s, input_fired_time: 15.6276626587, output_fired_time: 5.5834431648, propagate_forward_time: 307.7846069336, propagate_backward_time: 502.5327758789, weight_update_time: 0.2291661799, other_time: 9.3566894531
epoch time: 63.8623008728s, input_fired_time: 0.9658331275, output_fired_time: 0.0000000000, propagate_forward_time: 61.3724517822, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5240173340
validation[          , genome  6199] predictions:    6245/  10000 (62.45%), best:    6294/10000 (62.94%), error:     11700.39258, best error:     10763.84082 on epoch:     0, epoch:    8/25, mu: 0.6757135391, learning_rate: 0.0000045521, weight_decay: 0.0000994620

epoch time: 850.4874877930s, input_fired_time: 15.9046020508, output_fired_time: 5.7114715576, propagate_forward_time: 311.2042846680, propagate_backward_time: 508.1065368652, weight_update_time: 0.2335894555, other_time: 9.5605773926
epoch time: 63.4517021179s, input_fired_time: 0.9698942900, output_fired_time: 0.0000000000, propagate_forward_time: 60.9657325745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5160751343
validation[          , genome  6199] predictions:    6218/  10000 (62.18%), best:    6294/10000 (62.94%), error:     11830.32422, best error:     10763.84082 on epoch:     0, epoch:    9/25, mu: 0.6954443455, learning_rate: 0.0000041256, weight_decay: 0.0000920338

epoch time: 838.6326904297s, input_fired_time: 15.5363311768, output_fired_time: 5.5712227821, propagate_forward_time: 306.7525939941, propagate_backward_time: 501.4400634766, weight_update_time: 0.2283398956, other_time: 9.3324890137
epoch time: 65.6284637451s, input_fired_time: 1.0192693472, output_fired_time: 0.0000000000, propagate_forward_time: 62.9844970703, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6246948242
validation[          , genome  6199] predictions:    6214/  10000 (62.14%), best:    6294/10000 (62.94%), error:     11919.71973, best error:     10763.84082 on epoch:     0, epoch:   10/25, mu: 0.7139364481, learning_rate: 0.0000037391, weight_decay: 0.0000851603

epoch time: 837.6890258789s, input_fired_time: 15.5288438797, output_fired_time: 5.5664939880, propagate_forward_time: 306.2662963867, propagate_backward_time: 501.0137023926, weight_update_time: 0.2280811071, other_time: 9.3136901855
epoch time: 63.5087051392s, input_fired_time: 0.9710615277, output_fired_time: 0.0000000000, propagate_forward_time: 61.0161972046, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5214462280
validation[          , genome  6199] predictions:    6221/  10000 (62.21%), best:    6294/10000 (62.94%), error:     11984.38867, best error:     10763.84082 on epoch:     0, epoch:   11/25, mu: 0.7312676311, learning_rate: 0.0000033888, weight_decay: 0.0000788001

epoch time: 848.4821166992s, input_fired_time: 15.7767066956, output_fired_time: 5.6899333000, propagate_forward_time: 310.3321838379, propagate_backward_time: 507.2123413086, weight_update_time: 0.2336763591, other_time: 9.4708862305
epoch time: 63.5226402283s, input_fired_time: 0.9857836366, output_fired_time: 0.0000000000, propagate_forward_time: 61.0098609924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5269966125
validation[          , genome  6199] predictions:    6207/  10000 (62.07%), best:    6294/10000 (62.94%), error:     12106.47949, best error:     10763.84082 on epoch:     0, epoch:   12/25, mu: 0.7475107908, learning_rate: 0.0000030713, weight_decay: 0.0000729150

epoch time: 841.6986694336s, input_fired_time: 15.6964387894, output_fired_time: 5.5749692917, propagate_forward_time: 308.2208862305, propagate_backward_time: 502.8659362793, weight_update_time: 0.2295392454, other_time: 9.3404846191
epoch time: 65.6811218262s, input_fired_time: 1.0107011795, output_fired_time: 0.0000000000, propagate_forward_time: 63.0704460144, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5999717712
validation[          , genome  6199] predictions:    6174/  10000 (61.74%), best:    6294/10000 (62.94%), error:     12197.69727, best error:     10763.84082 on epoch:     0, epoch:   13/25, mu: 0.7627341747, learning_rate: 0.0000027835, weight_decay: 0.0000674693

epoch time: 833.7945556641s, input_fired_time: 15.4371652603, output_fired_time: 5.4945354462, propagate_forward_time: 304.9347534180, propagate_backward_time: 498.7179260254, weight_update_time: 0.2258294076, other_time: 9.2101745605
epoch time: 65.3483200073s, input_fired_time: 0.9913627505, output_fired_time: 0.0000000000, propagate_forward_time: 62.7768402100, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5801162720
validation[          , genome  6199] predictions:    6172/  10000 (61.72%), best:    6294/10000 (62.94%), error:     12249.15918, best error:     10763.84082 on epoch:     0, epoch:   14/25, mu: 0.7770018578, learning_rate: 0.0000025227, weight_decay: 0.0000624304

epoch time: 847.3334350586s, input_fired_time: 15.8033800125, output_fired_time: 5.6508097649, propagate_forward_time: 310.2868347168, propagate_backward_time: 506.1173095703, weight_update_time: 0.2315004319, other_time: 9.4750976563
epoch time: 63.5158348083s, input_fired_time: 0.9675256610, output_fired_time: 0.0000000000, propagate_forward_time: 61.0252990723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5230102539
validation[          , genome  6199] predictions:    6173/  10000 (61.73%), best:    6294/10000 (62.94%), error:     12333.77344, best error:     10763.84082 on epoch:     0, epoch:   15/25, mu: 0.7903738022, learning_rate: 0.0000022864, weight_decay: 0.0000577678

epoch time: 859.0866699219s, input_fired_time: 15.8714962006, output_fired_time: 5.6110482216, propagate_forward_time: 314.4251403809, propagate_backward_time: 513.6456909180, weight_update_time: 0.2297969908, other_time: 9.5332641602
epoch time: 66.3019943237s, input_fired_time: 1.0159066916, output_fired_time: 0.0000000000, propagate_forward_time: 63.6520118713, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6340751648
validation[          , genome  6199] predictions:    6139/  10000 (61.39%), best:    6294/10000 (62.94%), error:     12409.12207, best error:     10763.84082 on epoch:     0, epoch:   16/25, mu: 0.8029062748, learning_rate: 0.0000020722, weight_decay: 0.0000534535

epoch time: 870.0540161133s, input_fired_time: 16.0834999084, output_fired_time: 5.7137880325, propagate_forward_time: 318.2469177246, propagate_backward_time: 520.2985229492, weight_update_time: 0.2338806391, other_time: 9.7112426758
epoch time: 67.1621322632s, input_fired_time: 1.0195306540, output_fired_time: 0.0000000000, propagate_forward_time: 64.4876174927, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6549835205
validation[          , genome  6199] predictions:    6115/  10000 (61.15%), best:    6294/10000 (62.94%), error:     12451.27832, best error:     10763.84082 on epoch:     0, epoch:   17/25, mu: 0.8146519661, learning_rate: 0.0000018780, weight_decay: 0.0000494613

epoch time: 869.8010253906s, input_fired_time: 15.9614248276, output_fired_time: 5.7617821693, propagate_forward_time: 318.0231018066, propagate_backward_time: 520.2210693359, weight_update_time: 0.2403663248, other_time: 9.8336181641
epoch time: 66.6936569214s, input_fired_time: 1.0207369328, output_fired_time: 0.0000000000, propagate_forward_time: 64.0039978027, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6689224243
validation[          , genome  6199] predictions:    6131/  10000 (61.31%), best:    6294/10000 (62.94%), error:     12499.87793, best error:     10763.84082 on epoch:     0, epoch:   18/25, mu: 0.8256602287, learning_rate: 0.0000017021, weight_decay: 0.0000457673

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199_0_r464034326_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.462051
read mu: 0.835977
read mu_delta: 0.93722
read initial_learning_rate: 1e-05
read learning_rate: 1.5426e-06
read learning_rate_delta: 0.90631
read initial_weight_decay: 0.000185071
read weight_decay: 4.23492e-05
read weight_decay_delta: 0.925315
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0511267
read input_dropout_probability: 0.00153756
read hidden_dropout_probability: 0.0786986
read velocity_reset: 1392
read epoch: 19
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6294
read best_validation_error: 10763.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6199
read normal distribution: '1 0 0'
generator_str: '1054725057'
read generator: 1054725057
read generated_by_map:
1 enable_node 1
reading 104 nodes.
reading 381 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 863.547973632813s, input_fired_time: 15.9385414123535, output_fired_time: 5.76007318496704, propagate_forward_time: 316.028137207031, propagate_backward_time: 516.095642089844, weight_update_time: 0.235214054584503, other_time: 9.7255859375
epoch time: 65.8438949584961s, input_fired_time: 1.00848686695099, output_fired_time: 0, propagate_forward_time: 63.2140693664551, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.62134170532227
validation[          , genome  6199] predictions:    6144/  10000 (61.44%), best:    6294/10000 (62.94%), error:     12540.85059, best error:     10763.84082 on epoch:     0, epoch:   19/25, mu: 0.8359774351, learning_rate: 0.0000015426, weight_decay: 0.0000423492

epoch time: 864.2138671875s, input_fired_time: 15.9844112396, output_fired_time: 5.7257108688, propagate_forward_time: 316.0959167480, propagate_backward_time: 516.6977539063, weight_update_time: 0.2356462181, other_time: 9.7100830078
epoch time: 66.3374710083s, input_fired_time: 1.0151191950, output_fired_time: 0.0000000000, propagate_forward_time: 63.6689262390, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6534233093
validation[          , genome  6199] predictions:    6109/  10000 (61.09%), best:    6294/10000 (62.94%), error:     12579.82910, best error:     10763.84082 on epoch:     0, epoch:   20/25, mu: 0.8456469178, learning_rate: 0.0000013981, weight_decay: 0.0000391864

epoch time: 865.1982421875s, input_fired_time: 15.9749956131, output_fired_time: 5.7168807983, propagate_forward_time: 316.6384582520, propagate_backward_time: 517.0938110352, weight_update_time: 0.2314726859, other_time: 9.7741088867
epoch time: 66.6098556519s, input_fired_time: 1.0183564425, output_fired_time: 0.0000000000, propagate_forward_time: 63.9451866150, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6463127136
validation[          , genome  6199] predictions:    6085/  10000 (60.85%), best:    6294/10000 (62.94%), error:     12609.73730, best error:     10763.84082 on epoch:     0, epoch:   21/25, mu: 0.8547093272, learning_rate: 0.0000012671, weight_decay: 0.0000362598

epoch time: 877.7576904297s, input_fired_time: 16.1553840637, output_fired_time: 5.8071837425, propagate_forward_time: 321.4182739258, propagate_backward_time: 524.4757690430, weight_update_time: 0.2398894876, other_time: 9.9010620117
epoch time: 67.3689346313s, input_fired_time: 1.0260795355, output_fired_time: 0.0000000000, propagate_forward_time: 64.6565933228, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6862640381
validation[          , genome  6199] predictions:    6094/  10000 (60.94%), best:    6294/10000 (62.94%), error:     12648.56934, best error:     10763.84082 on epoch:     0, epoch:   22/25, mu: 0.8632028103, learning_rate: 0.0000011484, weight_decay: 0.0000335517

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542495604_170_6199_0_r464034326_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.462051
read mu: 0.871163
read mu_delta: 0.93722
read initial_learning_rate: 1e-05
read learning_rate: 1.04078e-06
read learning_rate_delta: 0.90631
read initial_weight_decay: 0.000185071
read weight_decay: 3.10459e-05
read weight_decay_delta: 0.925315
read batch_size: 105
read epsilon: 1e-07
read alpha: 0.0511267
read input_dropout_probability: 0.00153756
read hidden_dropout_probability: 0.0786986
read velocity_reset: 1392
read epoch: 23
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 6294
read best_validation_error: 10763.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6199
read normal distribution: '1 0 0'
generator_str: '1311125774'
read generator: 1311125774
read generated_by_map:
1 enable_node 1
reading 104 nodes.
reading 381 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 842.369873046875s, input_fired_time: 15.5619802474976, output_fired_time: 5.50586080551147, propagate_forward_time: 308.643218994141, propagate_backward_time: 502.96826171875, weight_update_time: 0.230037957429886, other_time: 9.6905517578125
epoch time: 64.8418045043945s, input_fired_time: 0.981450617313385, output_fired_time: 0, propagate_forward_time: 62.2048568725586, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.65549850463867
validation[          , genome  6199] predictions:    6087/  10000 (60.87%), best:    6294/10000 (62.94%), error:     12663.98828, best error:     10763.84082 on epoch:     0, epoch:   23/25, mu: 0.8711630702, learning_rate: 0.0000010408, weight_decay: 0.0000310459

epoch time: 844.9436035156s, input_fired_time: 15.5951509476, output_fired_time: 5.5025000572, propagate_forward_time: 308.6563110352, propagate_backward_time: 505.4520263672, weight_update_time: 0.2309707105, other_time: 9.7376098633
epoch time: 64.7267913818s, input_fired_time: 0.9916856289, output_fired_time: 0.0000000000, propagate_forward_time: 62.1009368896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6341705322
validation[          , genome  6199] predictions:    6103/  10000 (61.03%), best:    6294/10000 (62.94%), error:     12699.82617, best error:     10763.84082 on epoch:     0, epoch:   24/25, mu: 0.8786236048, learning_rate: 0.0000009433, weight_decay: 0.0000287273

epoch time: 843.4480590820s, input_fired_time: 15.5744609833, output_fired_time: 5.5145325661, propagate_forward_time: 308.2839355469, propagate_backward_time: 504.3149719238, weight_update_time: 0.2391557395, other_time: 9.7601623535
epoch time: 64.9675979614s, input_fired_time: 0.9878866076, output_fired_time: 0.0000000000, propagate_forward_time: 62.3496437073, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6300659180
validation[          , genome  6199] predictions:    6086/  10000 (60.86%), best:    6294/10000 (62.94%), error:     12741.69434, best error:     10763.84082 on epoch:     0, epoch:   25/25, mu: 0.8856157660, learning_rate: 0.0000008549, weight_decay: 0.0000265818

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 322.0080566406s, input_fired_time: 4.9612517357, output_fired_time: 0.0000000000, propagate_forward_time: 308.8243408203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2224731445
best training[          , genome  6199] predictions:   34850/  50000 (69.70%), best:    6294/10000 (62.94%), error:     45907.87891, best error:     10763.84082 on epoch:     0, epoch:   26/25, mu: 0.8921689987, learning_rate: 0.0000007748, weight_decay: 0.0000245966
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 64.8325119019s, input_fired_time: 0.9865265489, output_fired_time: 0.0000000000, propagate_forward_time: 62.2085151672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6374702454
   testing[          , genome  6199] predictions:    6294/  10000 (62.94%), best:    6294/10000 (62.94%), error:     10763.84082, best error:     10763.84082 on epoch:     0, epoch:   26/25, mu: 0.8921689987, learning_rate: 0.0000007748, weight_decay: 0.0000245966
backpropagation finished successfully!
14:39:33 (9980): called boinc_finish(0)

</stderr_txt>
]]>