Nameexact_genome_1542069603_172_5732_1
Workunit4084824
Created13 Nov 2018, 6:29:26 UTC
Sent13 Nov 2018, 6:53:56 UTC
Report deadline10 Dec 2018, 8:41:31 UTC
Received13 Nov 2018, 22:35:20 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60920
Run time12 hours 25 min 39 sec
CPU time12 hours 21 min 36 sec
Validate stateValid
Credit6,768.68
Device peak FLOPS3.88 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size564.97 MB
Peak swap size563.24 MB
Peak disk usage12.74 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_172_5732.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_172_5732_1_r2121396871_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542069603_172_5732.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.495044
read mu: 0.495044
read mu_delta: 0.931319
read initial_learning_rate: 0.000517007
read learning_rate: 0.000517007
read learning_rate_delta: 0.928471
read initial_weight_decay: 0.000232595
read weight_decay: 0.000232595
read weight_decay_delta: 0.940432
read batch_size: 104
read epsilon: 1e-07
read alpha: 0.0916003
read input_dropout_probability: 0.00140804
read hidden_dropout_probability: 0.0779236
read velocity_reset: 1766
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5732
read normal distribution: '0 0.678724765777588 -1.27143037319183'
generator_str: '1778203107'
read generator: 1778203107
read generated_by_map:
1 split_edge 1
reading 101 nodes.
reading 419 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1877187697
post shuffle 1: 791913448
epoch time: 1499.80834960938s, input_fired_time: 21.5298118591309, output_fired_time: 10.6248302459717, propagate_forward_time: 573.711791992188, propagate_backward_time: 880.238647460938, weight_update_time: 0.440581202507019, other_time: 13.7032470703125
epoch time: 119.372146606445s, input_fired_time: 1.68430256843567, output_fired_time: 0, propagate_forward_time: 115.307929992676, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.37991333007813
validation[          , genome  5732] predictions:    6011/  10000 (60.11%), best:    6011/10000 (60.11%), error:     11638.40820, best error:     11638.40820 on epoch:     0, epoch:    0/25, mu: 0.4950443506, learning_rate: 0.0005170074, weight_decay: 0.0002325946

epoch time: 1501.2645263672s, input_fired_time: 21.5291271210, output_fired_time: 10.6240997314, propagate_forward_time: 574.2033691406, propagate_backward_time: 881.2843627930, weight_update_time: 0.4375465214, other_time: 13.6234741211
epoch time: 120.0696258545s, input_fired_time: 1.6903450489, output_fired_time: 0.0000000000, propagate_forward_time: 115.9881591797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3911209106
validation[          , genome  5732] predictions:    5865/  10000 (58.65%), best:    6011/10000 (60.11%), error:     11852.04980, best error:     11638.40820 on epoch:     0, epoch:    1/25, mu: 0.5290383697, learning_rate: 0.0004800263, weight_decay: 0.0002187394

epoch time: 1503.6634521484s, input_fired_time: 21.5235977173, output_fired_time: 10.6426143646, propagate_forward_time: 574.9147338867, propagate_backward_time: 882.8258056641, weight_update_time: 0.4430342615, other_time: 13.7567749023
epoch time: 119.8957672119s, input_fired_time: 1.6892207861, output_fired_time: 0.0000000000, propagate_forward_time: 115.8042526245, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4022903442
validation[          , genome  5732] predictions:    5972/  10000 (59.72%), best:    5972/10000 (59.72%), error:     11540.53125, best error:     11540.53125 on epoch:     2, epoch:    2/25, mu: 0.5606976151, learning_rate: 0.0004456905, weight_decay: 0.0002057095

epoch time: 1503.6993408203s, input_fired_time: 21.5327739716, output_fired_time: 10.6713914871, propagate_forward_time: 575.5771484375, propagate_backward_time: 882.1969604492, weight_update_time: 0.4422671199, other_time: 13.7211303711
epoch time: 119.5414505005s, input_fired_time: 1.7027293444, output_fired_time: 0.0000000000, propagate_forward_time: 115.4325256348, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4061965942
validation[          , genome  5732] predictions:    6103/  10000 (61.03%), best:    6103/10000 (61.03%), error:     11277.42969, best error:     11277.42969 on epoch:     3, epoch:    3/25, mu: 0.5901824832, learning_rate: 0.0004138107, weight_decay: 0.0001934558

epoch time: 1503.6354980469s, input_fired_time: 21.5802822113, output_fired_time: 10.6949853897, propagate_forward_time: 574.9115600586, propagate_backward_time: 882.7777709961, weight_update_time: 0.4367642999, other_time: 13.6708984375
epoch time: 119.7204589844s, input_fired_time: 1.6858539581, output_fired_time: 0.0000000000, propagate_forward_time: 115.6473617554, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3872451782
validation[          , genome  5732] predictions:    6083/  10000 (60.83%), best:    6103/10000 (61.03%), error:     11385.15820, best error:     11277.42969 on epoch:     3, epoch:    4/25, mu: 0.6176422834, learning_rate: 0.0003842112, weight_decay: 0.0001819320

epoch time: 1504.8057861328s, input_fired_time: 21.5533981323, output_fired_time: 10.6759490967, propagate_forward_time: 575.3084716797, propagate_backward_time: 883.5316772461, weight_update_time: 0.4390621483, other_time: 13.7363891602
epoch time: 119.9648895264s, input_fired_time: 1.6936962605, output_fired_time: 0.0000000000, propagate_forward_time: 115.8660278320, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4051666260
validation[          , genome  5732] predictions:    6085/  10000 (60.85%), best:    6103/10000 (61.03%), error:     11307.63281, best error:     11277.42969 on epoch:     3, epoch:    5/25, mu: 0.6432161331, learning_rate: 0.0003567290, weight_decay: 0.0001710947

epoch time: 1503.7174072266s, input_fired_time: 21.5307369232, output_fired_time: 10.6474828720, propagate_forward_time: 575.5122070313, propagate_backward_time: 882.3123168945, weight_update_time: 0.4434055984, other_time: 13.7146606445
epoch time: 119.9366607666s, input_fired_time: 1.6819307804, output_fired_time: 0.0000000000, propagate_forward_time: 115.8706741333, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3840560913
validation[          , genome  5732] predictions:    6082/  10000 (60.82%), best:    6103/10000 (61.03%), error:     11377.32324, best error:     11277.42969 on epoch:     3, epoch:    6/25, mu: 0.6670335531, learning_rate: 0.0003312125, weight_decay: 0.0001609029

epoch time: 1510.4182128906s, input_fired_time: 21.6043586731, output_fired_time: 10.6873521805, propagate_forward_time: 577.8733520508, propagate_backward_time: 886.5144653320, weight_update_time: 0.4406715930, other_time: 13.7386474609
epoch time: 120.1977233887s, input_fired_time: 1.6873886585, output_fired_time: 0.0000000000, propagate_forward_time: 116.1282119751, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3821258545
validation[          , genome  5732] predictions:    6013/  10000 (60.13%), best:    6103/10000 (61.03%), error:     11560.76074, best error:     11277.42969 on epoch:     3, epoch:    7/25, mu: 0.6892151833, learning_rate: 0.0003075212, weight_decay: 0.0001513182

epoch time: 1502.9631347656s, input_fired_time: 21.5205307007, output_fired_time: 10.6547946930, propagate_forward_time: 573.9683837891, propagate_backward_time: 883.1140136719, weight_update_time: 0.4440140724, other_time: 13.7054443359
epoch time: 120.0414733887s, input_fired_time: 1.6837214231, output_fired_time: 0.0000000000, propagate_forward_time: 115.9816207886, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3761291504
validation[          , genome  5732] predictions:    6095/  10000 (60.95%), best:    6095/10000 (60.95%), error:     11240.37207, best error:     11240.37207 on epoch:     8, epoch:    8/25, mu: 0.7098733783, learning_rate: 0.0002855245, weight_decay: 0.0001423045

epoch time: 1503.3593750000s, input_fired_time: 21.5546722412, output_fired_time: 10.6328887939, propagate_forward_time: 574.9994506836, propagate_backward_time: 882.3955688477, weight_update_time: 0.4401267469, other_time: 13.7767333984
epoch time: 120.0187377930s, input_fired_time: 1.6894699335, output_fired_time: 0.0000000000, propagate_forward_time: 115.9471359253, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3821334839
validation[          , genome  5732] predictions:    6248/  10000 (62.48%), best:    6248/10000 (62.48%), error:     10816.49219, best error:     10816.49219 on epoch:     9, epoch:    9/25, mu: 0.7291127443, learning_rate: 0.0002651012, weight_decay: 0.0001338276

epoch time: 1502.8677978516s, input_fired_time: 21.5466537476, output_fired_time: 10.6444950104, propagate_forward_time: 574.7793579102, propagate_backward_time: 882.2459716797, weight_update_time: 0.4365624487, other_time: 13.6513061523
epoch time: 119.8323211670s, input_fired_time: 1.6853978634, output_fired_time: 0.0000000000, propagate_forward_time: 115.7671813965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3797454834
validation[          , genome  5732] predictions:    6227/  10000 (62.27%), best:    6248/10000 (62.48%), error:     10902.27148, best error:     10816.49219 on epoch:     9, epoch:   10/25, mu: 0.7470307350, learning_rate: 0.0002461388, weight_decay: 0.0001258558

epoch time: 1502.7735595703s, input_fired_time: 21.5670070648, output_fired_time: 10.6549644470, propagate_forward_time: 574.5932006836, propagate_backward_time: 882.2304077148, weight_update_time: 0.4437176585, other_time: 13.7280273438
epoch time: 120.1867446899s, input_fired_time: 1.6841131449, output_fired_time: 0.0000000000, propagate_forward_time: 116.0920181274, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4106140137
validation[          , genome  5732] predictions:    6033/  10000 (60.33%), best:    6248/10000 (62.48%), error:     11274.98340, best error:     10816.49219 on epoch:     9, epoch:   11/25, mu: 0.7637180686, learning_rate: 0.0002285327, weight_decay: 0.0001183588

epoch time: 1502.5103759766s, input_fired_time: 21.5195064545, output_fired_time: 10.6391439438, propagate_forward_time: 574.1201171875, propagate_backward_time: 882.5367431641, weight_update_time: 0.4403134882, other_time: 13.6948242188
epoch time: 119.8673858643s, input_fired_time: 1.6939041615, output_fired_time: 0.0000000000, propagate_forward_time: 115.7629394531, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4105453491
validation[          , genome  5732] predictions:    6287/  10000 (62.87%), best:    6287/10000 (62.87%), error:     10630.23438, best error:     10630.23438 on epoch:    12, epoch:   12/25, mu: 0.7792593241, learning_rate: 0.0002121860, weight_decay: 0.0001113084

epoch time: 1503.4477539063s, input_fired_time: 21.5758666992, output_fired_time: 10.6478300095, propagate_forward_time: 574.6374511719, propagate_backward_time: 882.9132080078, weight_update_time: 0.4433259964, other_time: 13.6733398438
epoch time: 120.7473983765s, input_fired_time: 1.6949590445, output_fired_time: 0.0000000000, propagate_forward_time: 116.6394119263, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4130249023
validation[          , genome  5732] predictions:    6201/  10000 (62.01%), best:    6287/10000 (62.87%), error:     11033.59180, best error:     10630.23438 on epoch:    12, epoch:   13/25, mu: 0.7937331796, learning_rate: 0.0001970085, weight_decay: 0.0001046780

epoch time: 1505.5512695313s, input_fired_time: 21.5700893402, output_fired_time: 10.6497364044, propagate_forward_time: 575.6128540039, propagate_backward_time: 883.9555053711, weight_update_time: 0.4479606450, other_time: 13.7630615234
epoch time: 120.1853790283s, input_fired_time: 1.6987942457, output_fired_time: 0.0000000000, propagate_forward_time: 116.1112899780, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3752975464
validation[          , genome  5732] predictions:    6225/  10000 (62.25%), best:    6287/10000 (62.87%), error:     11000.47754, best error:     10630.23438 on epoch:    12, epoch:   14/25, mu: 0.8072129488, learning_rate: 0.0001829167, weight_decay: 0.0000984425

epoch time: 1504.1695556641s, input_fired_time: 21.6027259827, output_fired_time: 10.6324777603, propagate_forward_time: 574.7224121094, propagate_backward_time: 883.4846191406, weight_update_time: 0.4371361136, other_time: 13.7272949219
epoch time: 120.2786865234s, input_fired_time: 1.6898670197, output_fired_time: 0.0000000000, propagate_forward_time: 116.1829528809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4058685303
validation[          , genome  5732] predictions:    6104/  10000 (61.04%), best:    6287/10000 (62.87%), error:     11241.72168, best error:     10630.23438 on epoch:    12, epoch:   15/25, mu: 0.8197669387, learning_rate: 0.0001698328, weight_decay: 0.0000925785

epoch time: 1567.6635742188s, input_fired_time: 22.1969547272, output_fired_time: 11.1688013077, propagate_forward_time: 600.0239257813, propagate_backward_time: 920.0532836914, weight_update_time: 0.4550286829, other_time: 14.2206420898
epoch time: 121.5201110840s, input_fired_time: 1.7122621536, output_fired_time: 0.0000000000, propagate_forward_time: 117.3820037842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4258422852
validation[          , genome  5732] predictions:    6199/  10000 (61.99%), best:    6287/10000 (62.87%), error:     10895.33691, best error:     10630.23438 on epoch:    12, epoch:   16/25, mu: 0.8314586878, learning_rate: 0.0001576849, weight_decay: 0.0000870637

epoch time: 1505.3687744141s, input_fired_time: 21.6088714600, output_fired_time: 10.6731576920, propagate_forward_time: 575.6571655273, propagate_backward_time: 883.6817016602, weight_update_time: 0.4417119026, other_time: 13.7478027344
epoch time: 120.0641250610s, input_fired_time: 1.6950976849, output_fired_time: 0.0000000000, propagate_forward_time: 115.9759063721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3931198120
validation[          , genome  5732] predictions:    6272/  10000 (62.72%), best:    6287/10000 (62.87%), error:     10750.17773, best error:     10630.23438 on epoch:    12, epoch:   17/25, mu: 0.8423474431, learning_rate: 0.0001464058, weight_decay: 0.0000818775

epoch time: 1506.3261718750s, input_fired_time: 21.6064300537, output_fired_time: 10.6693019867, propagate_forward_time: 575.8679809570, propagate_backward_time: 884.5006103516, weight_update_time: 0.4419415891, other_time: 13.6818237305
epoch time: 120.3749847412s, input_fired_time: 1.7041009665, output_fired_time: 0.0000000000, propagate_forward_time: 116.2897949219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3810882568
validation[          , genome  5732] predictions:    6316/  10000 (63.16%), best:    6316/10000 (63.16%), error:     10601.02344, best error:     10601.02344 on epoch:    18, epoch:   18/25, mu: 0.8524883389, learning_rate: 0.0001359335, weight_decay: 0.0000770002

epoch time: 1515.4152832031s, input_fired_time: 21.6184844971, output_fired_time: 10.6504735947, propagate_forward_time: 575.6425170898, propagate_backward_time: 893.7980957031, weight_update_time: 0.4414969683, other_time: 13.7056274414
epoch time: 120.1195983887s, input_fired_time: 1.7016295195, output_fired_time: 0.0000000000, propagate_forward_time: 116.0196685791, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3983001709
validation[          , genome  5732] predictions:    6242/  10000 (62.42%), best:    6316/10000 (63.16%), error:     10777.18945, best error:     10601.02344 on epoch:    18, epoch:   19/25, mu: 0.8619327545, learning_rate: 0.0001262103, weight_decay: 0.0000724135

epoch time: 1506.2412109375s, input_fired_time: 21.5997295380, output_fired_time: 10.6718721390, propagate_forward_time: 575.9716186523, propagate_backward_time: 884.2212524414, weight_update_time: 0.4412854016, other_time: 13.7767333984
epoch time: 120.1484222412s, input_fired_time: 1.7012150288, output_fired_time: 0.0000000000, propagate_forward_time: 116.0449981689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4022064209
validation[          , genome  5732] predictions:    6250/  10000 (62.50%), best:    6316/10000 (63.16%), error:     10732.33789, best error:     10601.02344 on epoch:    18, epoch:   20/25, mu: 0.8707285523, learning_rate: 0.0001171826, weight_decay: 0.0000680999

epoch time: 1517.1021728516s, input_fired_time: 21.7223434448, output_fired_time: 10.7996349335, propagate_forward_time: 579.9489135742, propagate_backward_time: 890.8137207031, weight_update_time: 0.4445150197, other_time: 13.8175659180
epoch time: 119.8070449829s, input_fired_time: 1.6993285418, output_fired_time: 0.0000000000, propagate_forward_time: 115.7130584717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3946609497
validation[          , genome  5732] predictions:    6317/  10000 (63.17%), best:    6316/10000 (63.16%), error:     10617.95313, best error:     10601.02344 on epoch:    18, epoch:   21/25, mu: 0.8789201975, learning_rate: 0.0001088007, weight_decay: 0.0000640433

epoch time: 1505.6217041016s, input_fired_time: 21.6580562592, output_fired_time: 10.6470966339, propagate_forward_time: 575.9829101563, propagate_backward_time: 883.6263427734, weight_update_time: 0.4371821284, other_time: 13.7072753906
epoch time: 119.6027755737s, input_fired_time: 1.6906803846, output_fired_time: 0.0000000000, propagate_forward_time: 115.5276336670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3844604492
validation[          , genome  5732] predictions:    6279/  10000 (62.79%), best:    6316/10000 (63.16%), error:     10773.21582, best error:     10601.02344 on epoch:    18, epoch:   22/25, mu: 0.8865492344, learning_rate: 0.0001010183, weight_decay: 0.0000602284

epoch time: 1507.5334472656s, input_fired_time: 21.6410484314, output_fired_time: 10.7246313095, propagate_forward_time: 576.5582885742, propagate_backward_time: 884.8688354492, weight_update_time: 0.4445009530, other_time: 13.7407226563
epoch time: 120.5670394897s, input_fired_time: 1.7102590799, output_fired_time: 0.0000000000, propagate_forward_time: 116.4706192017, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3861618042
validation[          , genome  5732] predictions:    6388/  10000 (63.88%), best:    6388/10000 (63.88%), error:     10495.62012, best error:     10495.62012 on epoch:    23, epoch:   23/25, mu: 0.8936543465, learning_rate: 0.0000937925, weight_decay: 0.0000566407

epoch time: 1510.7000732422s, input_fired_time: 21.6922073364, output_fired_time: 10.7703399658, propagate_forward_time: 578.7142944336, propagate_backward_time: 885.8142089844, weight_update_time: 0.4441975057, other_time: 13.7089233398
epoch time: 120.8297805786s, input_fired_time: 1.7056082487, output_fired_time: 0.0000000000, propagate_forward_time: 116.7300491333, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3941268921
validation[          , genome  5732] predictions:    6297/  10000 (62.97%), best:    6388/10000 (63.88%), error:     10756.00586, best error:     10495.62012 on epoch:    23, epoch:   24/25, mu: 0.9002714753, learning_rate: 0.0000870836, weight_decay: 0.0000532667

epoch time: 1514.5673828125s, input_fired_time: 21.7475471497, output_fired_time: 10.7288227081, propagate_forward_time: 579.9417114258, propagate_backward_time: 888.4318237305, weight_update_time: 0.4421984255, other_time: 13.7174072266
epoch time: 120.5094223022s, input_fired_time: 1.7143816948, output_fired_time: 0.0000000000, propagate_forward_time: 116.4096832275, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3853607178
validation[          , genome  5732] predictions:    6462/  10000 (64.62%), best:    6462/10000 (64.62%), error:     10257.37012, best error:     10257.37012 on epoch:    25, epoch:   25/25, mu: 0.9064341187, learning_rate: 0.0000808546, weight_decay: 0.0000500937

evaluating best weights on full training data.
evaluting training set with running mean/variance:
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_172_5732.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_172_5732_1_r2121396871_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.495044
read mu: 0.912174
read mu_delta: 0.931319
read initial_learning_rate: 0.000517007
read learning_rate: 7.50712e-05
read learning_rate_delta: 0.928471
read initial_weight_decay: 0.000232595
read weight_decay: 4.71097e-05
read weight_decay_delta: 0.940432
read batch_size: 104
read epsilon: 1e-07
read alpha: 0.0916003
read input_dropout_probability: 0.00140804
read hidden_dropout_probability: 0.0779236
read velocity_reset: 1766
read epoch: 26
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 25
read number_validation_images: 10000
read best_validation_predictions: 6462
read best_validation_error: 10257.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5732
read normal distribution: '0 0.678724765777588 -1.27143037319183'
generator_str: '1102518505'
read generator: 1102518505
read generated_by_map:
1 split_edge 1
reading 101 nodes.
reading 419 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1528.484375s, input_fired_time: 22.0371761322021, output_fired_time: 11.0049390792847, propagate_forward_time: 593.160278320313, propagate_backward_time: 888.545166015625, weight_update_time: 0.459201544523239, other_time: 13.73681640625
epoch time: 123.941055297852s, input_fired_time: 1.72988629341125, output_fired_time: 0, propagate_forward_time: 119.821319580078, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.38984680175781
validation[          , genome  5732] predictions:    6348/  10000 (63.48%), best:    6462/10000 (64.62%), error:     10466.79492, best error:     10257.37012 on epoch:    25, epoch:   26/25, mu: 0.9121735096, learning_rate: 0.0000750712, weight_decay: 0.0000471097

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 613.5064697266s, input_fired_time: 8.6186227798, output_fired_time: 0.0000000000, propagate_forward_time: 592.9859008789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.9019165039
best training[          , genome  5732] predictions:   35176/  50000 (70.35%), best:    6462/10000 (64.62%), error:     45031.45313, best error:     10257.37012 on epoch:    25, epoch:   27/25, mu: 0.9175186753, learning_rate: 0.0000697014, weight_decay: 0.0000443035
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 123.4264221191s, input_fired_time: 1.7243461609, output_fired_time: 0.0000000000, propagate_forward_time: 119.3534317017, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3486404419
   testing[          , genome  5732] predictions:    6462/  10000 (64.62%), best:    6462/10000 (64.62%), error:     10257.37012, best error:     10257.37012 on epoch:    25, epoch:   27/25, mu: 0.9175186753, learning_rate: 0.0000697014, weight_decay: 0.0000443035
backpropagation finished successfully!
15:33:04 (17512): called boinc_finish(0)

</stderr_txt>
]]>