Nameexact_genome_1542692703_182_6397_4
Workunit4100542
Created20 Nov 2018, 9:17:00 UTC
Sent20 Nov 2018, 9:58:45 UTC
Report deadline5 Feb 2019, 2:00:25 UTC
Received27 Nov 2018, 13:03:10 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88000
Run time1 days 13 hours 54 min 17 sec
CPU time1 days 13 hours 52 min 53 sec
Validate stateValid
Credit19,166.96
Device peak FLOPS3.82 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size519.05 MB
Peak swap size517.43 MB
Peak disk usage9.30 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1238.8193359375s, input_fired_time: 10.6153421402, output_fired_time: 9.8282155991, propagate_forward_time: 493.0237731934, propagate_backward_time: 712.3767700195, weight_update_time: 1.4071539640, other_time: 12.9751586914
epoch time: 102.3906555176s, input_fired_time: 1.3840354681, output_fired_time: 0.0000000000, propagate_forward_time: 98.7399749756, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2666473389
validation[          , genome  6397] predictions:    5225/  10000 (52.25%), best:    5595/10000 (55.95%), error:     15552.35059, best error:     13110.45117 on epoch:    28, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 1235.5461425781s, input_fired_time: 10.6258916855, output_fired_time: 9.8279981613, propagate_forward_time: 491.1229858398, propagate_backward_time: 710.5883789063, weight_update_time: 1.4094974995, other_time: 13.3809204102
epoch time: 102.4248580933s, input_fired_time: 1.3518369198, output_fired_time: 0.0000000000, propagate_forward_time: 98.8650054932, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2080154419
validation[          , genome  6397] predictions:    5544/  10000 (55.44%), best:    5595/10000 (55.95%), error:     13706.17383, best error:     13110.45117 on epoch:    28, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 1235.9304199219s, input_fired_time: 10.6749610901, output_fired_time: 9.8281965256, propagate_forward_time: 491.1602783203, propagate_backward_time: 710.7627563477, weight_update_time: 1.4159060717, other_time: 13.5042114258
epoch time: 103.2386627197s, input_fired_time: 1.3920830488, output_fired_time: 0.0000000000, propagate_forward_time: 99.4973373413, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3492431641
validation[          , genome  6397] predictions:    5651/  10000 (56.51%), best:    5595/10000 (55.95%), error:     13346.85352, best error:     13110.45117 on epoch:    28, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 1239.5166015625s, input_fired_time: 10.7129898071, output_fired_time: 9.8388891220, propagate_forward_time: 493.0358581543, propagate_backward_time: 712.7868652344, weight_update_time: 1.4124354124, other_time: 13.1419677734
epoch time: 103.5931854248s, input_fired_time: 1.3635385036, output_fired_time: 0.0000000000, propagate_forward_time: 99.9254455566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3041992188
validation[          , genome  6397] predictions:    5738/  10000 (57.38%), best:    5595/10000 (55.95%), error:     13113.90527, best error:     13110.45117 on epoch:    28, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 1242.9934082031s, input_fired_time: 10.7441864014, output_fired_time: 9.8826532364, propagate_forward_time: 494.9405212402, propagate_backward_time: 714.2181396484, weight_update_time: 1.4166314602, other_time: 13.2078857422
epoch time: 101.7185745239s, input_fired_time: 1.3729300499, output_fired_time: 0.0000000000, propagate_forward_time: 98.0941238403, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2515182495
validation[          , genome  6397] predictions:    5561/  10000 (55.61%), best:    5595/10000 (55.95%), error:     13739.48730, best error:     13110.45117 on epoch:    28, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 1242.7066650391s, input_fired_time: 10.6664228439, output_fired_time: 9.8636922836, propagate_forward_time: 494.8545532227, propagate_backward_time: 714.2591552734, weight_update_time: 1.4117814302, other_time: 13.0629272461
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_182_6397.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_182_6397_4_r944138045_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.904336
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00218531
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 8.74123e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 34
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 28
read number_validation_images: 10000
read best_validation_predictions: 5595
read best_validation_error: 13110.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6397
read normal distribution: '1 -0.641208350658417 2.29480671882629'
generator_str: '1515169467'
read generator: 1515169467
read generated_by_map:
1 split_node 1
reading 75 nodes.
reading 255 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1246.03295898438s, input_fired_time: 10.8732891082764, output_fired_time: 9.89279556274414, propagate_forward_time: 496.442840576172, propagate_backward_time: 715.442687988281, weight_update_time: 1.42184221744537, other_time: 13.3812866210938
epoch time: 104.784744262695s, input_fired_time: 1.37319350242615, output_fired_time: 0, propagate_forward_time: 101.058540344238, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.35301208496094
validation[          , genome  6397] predictions:    5649/  10000 (56.49%), best:    5595/10000 (55.95%), error:     13556.08691, best error:     13110.45117 on epoch:    28, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 1244.6468505859s, input_fired_time: 10.6396713257, output_fired_time: 9.8863267899, propagate_forward_time: 496.1059265137, propagate_backward_time: 715.0651855469, weight_update_time: 1.4074907303, other_time: 12.9497070313
epoch time: 103.0490188599s, input_fired_time: 1.3682165146, output_fired_time: 0.0000000000, propagate_forward_time: 99.4025192261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2782821655
validation[          , genome  6397] predictions:    5617/  10000 (56.17%), best:    5595/10000 (55.95%), error:     13250.87402, best error:     13110.45117 on epoch:    28, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 1244.5589599609s, input_fired_time: 10.6725826263, output_fired_time: 9.8799257278, propagate_forward_time: 495.8948974609, propagate_backward_time: 715.1212158203, weight_update_time: 1.4094308615, other_time: 12.9903564453
epoch time: 101.4984817505s, input_fired_time: 1.3970637321, output_fired_time: 0.0000000000, propagate_forward_time: 97.8413772583, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2600402832
validation[          , genome  6397] predictions:    5720/  10000 (57.20%), best:    5720/10000 (57.20%), error:     13005.23535, best error:     13005.23535 on epoch:    36, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 1246.8537597656s, input_fired_time: 10.7502107620, output_fired_time: 9.8859224319, propagate_forward_time: 496.7522583008, propagate_backward_time: 716.2932128906, weight_update_time: 1.4154484272, other_time: 13.1721801758
epoch time: 102.0455322266s, input_fired_time: 1.4039018154, output_fired_time: 0.0000000000, propagate_forward_time: 98.3165740967, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3250579834
validation[          , genome  6397] predictions:    5685/  10000 (56.85%), best:    5720/10000 (57.20%), error:     13111.83496, best error:     13005.23535 on epoch:    36, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 1240.2602539063s, input_fired_time: 10.6540460587, output_fired_time: 9.8502388000, propagate_forward_time: 493.0956726074, propagate_backward_time: 713.3408203125, weight_update_time: 1.4092525244, other_time: 13.3194580078
epoch time: 102.1295471191s, input_fired_time: 1.4266636372, output_fired_time: 0.0000000000, propagate_forward_time: 98.3890838623, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3137969971
validation[          , genome  6397] predictions:    5785/  10000 (57.85%), best:    5720/10000 (57.20%), error:     13023.72070, best error:     13005.23535 on epoch:    36, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 1239.7858886719s, input_fired_time: 10.9696302414, output_fired_time: 9.8804416656, propagate_forward_time: 492.3217163086, propagate_backward_time: 711.3911132813, weight_update_time: 1.4288758039, other_time: 15.2229614258
epoch time: 101.5780944824s, input_fired_time: 1.4113388062, output_fired_time: 0.0000000000, propagate_forward_time: 97.5011596680, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6655960083
validation[          , genome  6397] predictions:    5848/  10000 (58.48%), best:    5848/10000 (58.48%), error:     12440.37402, best error:     12440.37402 on epoch:    39, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 1246.9335937500s, input_fired_time: 10.7863988876, output_fired_time: 9.9089727402, propagate_forward_time: 496.5717773438, propagate_backward_time: 716.3563842773, weight_update_time: 1.4166167974, other_time: 13.3101196289
epoch time: 101.6038894653s, input_fired_time: 1.3944857121, output_fired_time: 0.0000000000, propagate_forward_time: 96.2832565308, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9261474609
validation[          , genome  6397] predictions:    5715/  10000 (57.15%), best:    5848/10000 (58.48%), error:     12991.37793, best error:     12440.37402 on epoch:    39, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 1241.0971679688s, input_fired_time: 10.8858203888, output_fired_time: 9.8812627792, propagate_forward_time: 492.9539489746, propagate_backward_time: 713.3060302734, weight_update_time: 1.4209700823, other_time: 14.0700683594
epoch time: 104.2293624878s, input_fired_time: 1.4107514620, output_fired_time: 0.0000000000, propagate_forward_time: 100.4593811035, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3592300415
validation[          , genome  6397] predictions:    5738/  10000 (57.38%), best:    5848/10000 (58.48%), error:     12820.69629, best error:     12440.37402 on epoch:    39, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 1239.5734863281s, input_fired_time: 10.8032217026, output_fired_time: 9.8586034775, propagate_forward_time: 492.9287719727, propagate_backward_time: 712.1374511719, weight_update_time: 1.4219496250, other_time: 13.8453979492
epoch time: 101.5684814453s, input_fired_time: 1.3751591444, output_fired_time: 0.0000000000, propagate_forward_time: 97.7218475342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4714736938
validation[          , genome  6397] predictions:    5962/  10000 (59.62%), best:    5962/10000 (59.62%), error:     12300.85645, best error:     12300.85645 on epoch:    42, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 1249.2620849609s, input_fired_time: 10.8547935486, output_fired_time: 9.9191541672, propagate_forward_time: 497.1633911133, propagate_backward_time: 717.8027343750, weight_update_time: 1.4203798771, other_time: 13.5220336914
epoch time: 101.4496002197s, input_fired_time: 1.3519088030, output_fired_time: 0.0000000000, propagate_forward_time: 97.7445449829, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3531494141
validation[          , genome  6397] predictions:    5687/  10000 (56.87%), best:    5962/10000 (59.62%), error:     13391.43555, best error:     12300.85645 on epoch:    42, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 1237.8912353516s, input_fired_time: 10.7581415176, output_fired_time: 9.8354015350, propagate_forward_time: 491.9249267578, propagate_backward_time: 711.6222534180, weight_update_time: 1.4132466316, other_time: 13.7504272461
epoch time: 101.9020614624s, input_fired_time: 1.4224940538, output_fired_time: 0.0000000000, propagate_forward_time: 98.1731872559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3063812256
validation[          , genome  6397] predictions:    5911/  10000 (59.11%), best:    5962/10000 (59.62%), error:     12889.02051, best error:     12300.85645 on epoch:    42, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 1242.0270996094s, input_fired_time: 10.8857412338, output_fired_time: 9.8655176163, propagate_forward_time: 494.0617065430, propagate_backward_time: 713.9150390625, weight_update_time: 1.4173537493, other_time: 13.2991333008
epoch time: 103.2211608887s, input_fired_time: 1.4482712746, output_fired_time: 0.0000000000, propagate_forward_time: 99.4447860718, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3281021118
validation[          , genome  6397] predictions:    5846/  10000 (58.46%), best:    5962/10000 (59.62%), error:     12483.02539, best error:     12300.85645 on epoch:    42, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 1238.1147460938s, input_fired_time: 10.6644964218, output_fired_time: 9.8494024277, propagate_forward_time: 490.8113403320, propagate_backward_time: 712.4108886719, weight_update_time: 1.4108498096, other_time: 14.3786010742
epoch time: 102.3560028076s, input_fired_time: 1.3802025318, output_fired_time: 0.0000000000, propagate_forward_time: 98.7621231079, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2136764526
validation[          , genome  6397] predictions:    5922/  10000 (59.22%), best:    5962/10000 (59.62%), error:     12500.98047, best error:     12300.85645 on epoch:    42, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 1242.1623535156s, input_fired_time: 10.7560825348, output_fired_time: 9.8687868118, propagate_forward_time: 493.8791809082, propagate_backward_time: 714.0651855469, weight_update_time: 1.4127109051, other_time: 13.5931396484
epoch time: 102.2450027466s, input_fired_time: 1.3773410320, output_fired_time: 0.0000000000, propagate_forward_time: 98.4110183716, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4566421509
validation[          , genome  6397] predictions:    5822/  10000 (58.22%), best:    5962/10000 (59.62%), error:     12639.66309, best error:     12300.85645 on epoch:    42, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 1244.3923339844s, input_fired_time: 10.8106470108, output_fired_time: 9.8645057678, propagate_forward_time: 495.0274658203, propagate_backward_time: 715.5538940430, weight_update_time: 1.4134508371, other_time: 13.1358032227
epoch time: 102.5066070557s, input_fired_time: 1.4407154322, output_fired_time: 0.0000000000, propagate_forward_time: 98.7025146484, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3633804321
validation[          , genome  6397] predictions:    5686/  10000 (56.86%), best:    5962/10000 (59.62%), error:     13324.87598, best error:     12300.85645 on epoch:    42, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 1247.5401611328s, input_fired_time: 10.8130226135, output_fired_time: 9.9052019119, propagate_forward_time: 495.6083068848, propagate_backward_time: 717.3121948242, weight_update_time: 1.4179296494, other_time: 13.9015502930
epoch time: 104.1232910156s, input_fired_time: 1.4299802780, output_fired_time: 0.0000000000, propagate_forward_time: 100.3635253906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3297882080
validation[          , genome  6397] predictions:    5790/  10000 (57.90%), best:    5962/10000 (59.62%), error:     12841.19824, best error:     12300.85645 on epoch:    42, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 1240.2878417969s, input_fired_time: 10.7981224060, output_fired_time: 9.8535346985, propagate_forward_time: 492.1262207031, propagate_backward_time: 713.2969970703, weight_update_time: 1.4150222540, other_time: 14.2130126953
epoch time: 103.0391998291s, input_fired_time: 1.4182802439, output_fired_time: 0.0000000000, propagate_forward_time: 99.2640609741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3568572998
validation[          , genome  6397] predictions:    5812/  10000 (58.12%), best:    5962/10000 (59.62%), error:     12875.41309, best error:     12300.85645 on epoch:    42, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 1250.1142578125s, input_fired_time: 10.7937631607, output_fired_time: 9.9312820435, propagate_forward_time: 495.8521118164, propagate_backward_time: 719.5036621094, weight_update_time: 1.4180999994, other_time: 14.0333862305
epoch time: 102.0307464600s, input_fired_time: 1.4387505054, output_fired_time: 0.0000000000, propagate_forward_time: 98.2584762573, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3335189819
validation[          , genome  6397] predictions:    5925/  10000 (59.25%), best:    5962/10000 (59.62%), error:     12492.85938, best error:     12300.85645 on epoch:    42, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 1248.4681396484s, input_fired_time: 11.0205125809, output_fired_time: 9.9132156372, propagate_forward_time: 496.1930541992, propagate_backward_time: 717.7113037109, weight_update_time: 1.4296506643, other_time: 13.6300659180
epoch time: 103.3836669922s, input_fired_time: 1.4173830748, output_fired_time: 0.0000000000, propagate_forward_time: 99.5696640015, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3966217041
validation[          , genome  6397] predictions:    5896/  10000 (58.96%), best:    5962/10000 (59.62%), error:     12777.23633, best error:     12300.85645 on epoch:    42, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 1230.3831787109s, input_fired_time: 10.9092407227, output_fired_time: 9.8217926025, propagate_forward_time: 487.5955505371, propagate_backward_time: 707.5659179688, weight_update_time: 1.4184592962, other_time: 14.4906005859
epoch time: 101.9582061768s, input_fired_time: 1.4421080351, output_fired_time: 0.0000000000, propagate_forward_time: 98.1673583984, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3487396240
validation[          , genome  6397] predictions:    5968/  10000 (59.68%), best:    5968/10000 (59.68%), error:     12141.60059, best error:     12141.60059 on epoch:    53, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 1232.7036132813s, input_fired_time: 10.7173929214, output_fired_time: 9.8364448547, propagate_forward_time: 488.1887512207, propagate_backward_time: 708.9077148438, weight_update_time: 1.4114261866, other_time: 15.0533447266
epoch time: 102.0536422729s, input_fired_time: 1.4445500374, output_fired_time: 0.0000000000, propagate_forward_time: 98.2900772095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3190155029
validation[          , genome  6397] predictions:    5961/  10000 (59.61%), best:    5961/10000 (59.61%), error:     12080.27441, best error:     12080.27441 on epoch:    54, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 1236.1566162109s, input_fired_time: 10.7893056870, output_fired_time: 9.8348140717, propagate_forward_time: 490.7994995117, propagate_backward_time: 710.9136962891, weight_update_time: 1.4088451862, other_time: 13.8192749023
epoch time: 102.8702087402s, input_fired_time: 1.4006638527, output_fired_time: 0.0000000000, propagate_forward_time: 99.2570037842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2125396729
validation[          , genome  6397] predictions:    5917/  10000 (59.17%), best:    5961/10000 (59.61%), error:     12347.27734, best error:     12080.27441 on epoch:    54, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 1233.4283447266s, input_fired_time: 11.0068626404, output_fired_time: 9.8474416733, propagate_forward_time: 488.9289245605, propagate_backward_time: 709.0200805664, weight_update_time: 1.4226137400, other_time: 14.6250610352
epoch time: 103.1759338379s, input_fired_time: 1.4623039961, output_fired_time: 0.0000000000, propagate_forward_time: 99.3783874512, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3352432251
validation[          , genome  6397] predictions:    5843/  10000 (58.43%), best:    5961/10000 (59.61%), error:     12627.74609, best error:     12080.27441 on epoch:    54, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 1228.1131591797s, input_fired_time: 10.6855010986, output_fired_time: 9.8363552094, propagate_forward_time: 485.4571838379, propagate_backward_time: 706.9313964844, weight_update_time: 1.4115582705, other_time: 15.2027587891
epoch time: 102.1976623535s, input_fired_time: 1.4127764702, output_fired_time: 0.0000000000, propagate_forward_time: 98.4259796143, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3589096069
validation[          , genome  6397] predictions:    5937/  10000 (59.37%), best:    5961/10000 (59.61%), error:     12392.74707, best error:     12080.27441 on epoch:    54, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 1228.5653076172s, input_fired_time: 10.8486099243, output_fired_time: 9.8329076767, propagate_forward_time: 485.6501770020, propagate_backward_time: 706.0715332031, weight_update_time: 1.4218858480, other_time: 16.1621093750
epoch time: 100.9464035034s, input_fired_time: 1.3854240179, output_fired_time: 0.0000000000, propagate_forward_time: 95.9703369141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5906448364
validation[          , genome  6397] predictions:    6029/  10000 (60.29%), best:    6029/10000 (60.29%), error:     12013.44336, best error:     12013.44336 on epoch:    58, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 1239.1711425781s, input_fired_time: 11.0071315765, output_fired_time: 9.8822040558, propagate_forward_time: 491.0154724121, propagate_backward_time: 712.3475341797, weight_update_time: 1.4255106449, other_time: 14.9188232422
epoch time: 100.9966125488s, input_fired_time: 1.4174507856, output_fired_time: 0.0000000000, propagate_forward_time: 95.6991348267, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8800277710
validation[          , genome  6397] predictions:    6022/  10000 (60.22%), best:    6029/10000 (60.29%), error:     12089.91699, best error:     12013.44336 on epoch:    58, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 1243.3201904297s, input_fired_time: 11.0781831741, output_fired_time: 9.9007005692, propagate_forward_time: 493.9614257813, propagate_backward_time: 714.0734252930, weight_update_time: 1.4297690392, other_time: 14.3064575195
epoch time: 101.3728637695s, input_fired_time: 1.4187521935, output_fired_time: 0.0000000000, propagate_forward_time: 96.8385314941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1155776978
validation[          , genome  6397] predictions:    6051/  10000 (60.51%), best:    6051/10000 (60.51%), error:     11864.95410, best error:     11864.95410 on epoch:    60, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 1233.0783691406s, input_fired_time: 10.8087806702, output_fired_time: 9.8510780334, propagate_forward_time: 487.9285278320, propagate_backward_time: 709.3348999023, weight_update_time: 1.4161857367, other_time: 15.1550292969
epoch time: 102.8641586304s, input_fired_time: 1.3949058056, output_fired_time: 0.0000000000, propagate_forward_time: 99.2602386475, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2090148926
validation[          , genome  6397] predictions:    5979/  10000 (59.79%), best:    6051/10000 (60.51%), error:     12299.03223, best error:     11864.95410 on epoch:    60, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 1235.3940429688s, input_fired_time: 11.0346975327, output_fired_time: 9.8588047028, propagate_forward_time: 489.5880737305, propagate_backward_time: 709.8490600586, weight_update_time: 1.4269900322, other_time: 15.0634765625
epoch time: 100.5031051636s, input_fired_time: 1.4011005163, output_fired_time: 0.0000000000, propagate_forward_time: 95.1368484497, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9651565552
validation[          , genome  6397] predictions:    6125/  10000 (61.25%), best:    6125/10000 (61.25%), error:     11851.15234, best error:     11851.15234 on epoch:    62, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 1239.3282470703s, input_fired_time: 10.8807392120, output_fired_time: 9.8662195206, propagate_forward_time: 492.2074279785, propagate_backward_time: 712.4091186523, weight_update_time: 1.4154713154, other_time: 13.9647827148
epoch time: 102.0328063965s, input_fired_time: 1.4240937233, output_fired_time: 0.0000000000, propagate_forward_time: 98.1945190430, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4141921997
validation[          , genome  6397] predictions:    5889/  10000 (58.89%), best:    6125/10000 (61.25%), error:     12545.45215, best error:     11851.15234 on epoch:    62, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 1235.3846435547s, input_fired_time: 10.8757238388, output_fired_time: 9.8553714752, propagate_forward_time: 489.6273193359, propagate_backward_time: 710.4304199219, weight_update_time: 1.4164352417, other_time: 14.5958251953
epoch time: 102.0804138184s, input_fired_time: 1.4037669897, output_fired_time: 0.0000000000, propagate_forward_time: 98.3178329468, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3588104248
validation[          , genome  6397] predictions:    6053/  10000 (60.53%), best:    6125/10000 (61.25%), error:     12034.40918, best error:     11851.15234 on epoch:    62, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 1238.4910888672s, input_fired_time: 10.8150558472, output_fired_time: 9.8550500870, propagate_forward_time: 491.9260253906, propagate_backward_time: 712.4809570313, weight_update_time: 1.4073778391, other_time: 13.4139404297
epoch time: 101.8352203369s, input_fired_time: 1.4316083193, output_fired_time: 0.0000000000, propagate_forward_time: 97.8576126099, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5459976196
validation[          , genome  6397] predictions:    6142/  10000 (61.42%), best:    6142/10000 (61.42%), error:     11793.69238, best error:     11793.69238 on epoch:    65, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 1235.3862304688s, input_fired_time: 10.9384613037, output_fired_time: 9.8353033066, propagate_forward_time: 491.0218811035, propagate_backward_time: 710.3239135742, weight_update_time: 1.4128836393, other_time: 13.2666625977
epoch time: 102.0696411133s, input_fired_time: 1.4095311165, output_fired_time: 0.0000000000, propagate_forward_time: 97.7598800659, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9002304077
validation[          , genome  6397] predictions:    6040/  10000 (60.40%), best:    6142/10000 (61.42%), error:     12050.97852, best error:     11793.69238 on epoch:    65, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 1247.6604003906s, input_fired_time: 11.0704517365, output_fired_time: 9.9334135056, propagate_forward_time: 494.7931823730, propagate_backward_time: 716.7484741211, weight_update_time: 1.4312111139, other_time: 15.1148071289
epoch time: 102.3216247559s, input_fired_time: 1.4436155558, output_fired_time: 0.0000000000, propagate_forward_time: 98.3749465942, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5030593872
validation[          , genome  6397] predictions:    6053/  10000 (60.53%), best:    6142/10000 (61.42%), error:     12027.66992, best error:     11793.69238 on epoch:    65, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 1237.8840332031s, input_fired_time: 10.8321142197, output_fired_time: 9.8530416489, propagate_forward_time: 491.3927307129, propagate_backward_time: 712.2686157227, weight_update_time: 1.4080877304, other_time: 13.5375366211
epoch time: 101.3180236816s, input_fired_time: 1.4256801605, output_fired_time: 0.0000000000, propagate_forward_time: 97.4009628296, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4913787842
validation[          , genome  6397] predictions:    5995/  10000 (59.95%), best:    6142/10000 (61.42%), error:     12233.05762, best error:     11793.69238 on epoch:    65, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 1241.5679931641s, input_fired_time: 10.8407583237, output_fired_time: 9.8734493256, propagate_forward_time: 493.0484008789, propagate_backward_time: 713.7705078125, weight_update_time: 1.4099111557, other_time: 14.0349731445
epoch time: 101.1052474976s, input_fired_time: 1.3947335482, output_fired_time: 0.0000000000, propagate_forward_time: 97.3045120239, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4059982300
validation[          , genome  6397] predictions:    6024/  10000 (60.24%), best:    6142/10000 (61.42%), error:     11961.91602, best error:     11793.69238 on epoch:    65, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 1243.6466064453s, input_fired_time: 10.9052677155, output_fired_time: 9.8974514008, propagate_forward_time: 493.3858947754, propagate_backward_time: 714.8816528320, weight_update_time: 1.4192250967, other_time: 14.5763549805
epoch time: 103.0669708252s, input_fired_time: 1.4301440716, output_fired_time: 0.0000000000, propagate_forward_time: 99.3271942139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3096313477
validation[          , genome  6397] predictions:    6043/  10000 (60.43%), best:    6142/10000 (61.42%), error:     12051.84961, best error:     11793.69238 on epoch:    65, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 1243.4891357422s, input_fired_time: 11.0480546951, output_fired_time: 9.8922872543, propagate_forward_time: 493.2743835449, propagate_backward_time: 714.7033691406, weight_update_time: 1.4261516333, other_time: 14.5709228516
epoch time: 102.9757919312s, input_fired_time: 1.4143402576, output_fired_time: 0.0000000000, propagate_forward_time: 99.3610000610, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2004547119
validation[          , genome  6397] predictions:    6106/  10000 (61.06%), best:    6142/10000 (61.42%), error:     11859.84473, best error:     11793.69238 on epoch:    65, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 1248.4846191406s, input_fired_time: 11.0881948471, output_fired_time: 9.9483947754, propagate_forward_time: 494.3765563965, propagate_backward_time: 717.4114990234, weight_update_time: 1.4342162609, other_time: 15.6600341797
epoch time: 101.8502044678s, input_fired_time: 1.3877481222, output_fired_time: 0.0000000000, propagate_forward_time: 97.9425506592, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5199050903
validation[          , genome  6397] predictions:    6110/  10000 (61.10%), best:    6142/10000 (61.42%), error:     11936.64551, best error:     11793.69238 on epoch:    65, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 1239.3555908203s, input_fired_time: 10.8641815186, output_fired_time: 9.8429784775, propagate_forward_time: 492.5259094238, propagate_backward_time: 712.8789672852, weight_update_time: 1.4104365110, other_time: 13.2435913086
epoch time: 103.0379791260s, input_fired_time: 1.4596226215, output_fired_time: 0.0000000000, propagate_forward_time: 99.2361602783, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3421936035
validation[          , genome  6397] predictions:    6097/  10000 (60.97%), best:    6142/10000 (61.42%), error:     11958.41797, best error:     11793.69238 on epoch:    65, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 1239.0444335938s, input_fired_time: 10.8445501328, output_fired_time: 9.8693571091, propagate_forward_time: 492.1181030273, propagate_backward_time: 712.5576171875, weight_update_time: 1.4099574089, other_time: 13.6547241211
epoch time: 102.9104080200s, input_fired_time: 1.4102151394, output_fired_time: 0.0000000000, propagate_forward_time: 99.2954940796, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2046966553
validation[          , genome  6397] predictions:    6024/  10000 (60.24%), best:    6142/10000 (61.42%), error:     12222.11523, best error:     11793.69238 on epoch:    65, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 1238.5090332031s, input_fired_time: 10.8289661407, output_fired_time: 9.8489866257, propagate_forward_time: 491.7058410645, propagate_backward_time: 711.8344726563, weight_update_time: 1.4120870829, other_time: 14.2907714844
epoch time: 104.5859832764s, input_fired_time: 1.3836389780, output_fired_time: 0.0000000000, propagate_forward_time: 100.9513168335, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2510299683
validation[          , genome  6397] predictions:    6079/  10000 (60.79%), best:    6142/10000 (61.42%), error:     11829.54590, best error:     11793.69238 on epoch:    65, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 1242.2772216797s, input_fired_time: 10.8813047409, output_fired_time: 9.8853340149, propagate_forward_time: 492.5915222168, propagate_backward_time: 714.1776123047, weight_update_time: 1.4156318903, other_time: 14.7413330078
epoch time: 101.2996978760s, input_fired_time: 1.3966145515, output_fired_time: 0.0000000000, propagate_forward_time: 97.1093902588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7936935425
validation[          , genome  6397] predictions:    6097/  10000 (60.97%), best:    6142/10000 (61.42%), error:     11962.09668, best error:     11793.69238 on epoch:    65, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 1242.5358886719s, input_fired_time: 11.0349626541, output_fired_time: 9.9300346375, propagate_forward_time: 491.3479003906, propagate_backward_time: 714.4685058594, weight_update_time: 1.4321089983, other_time: 15.7545166016
epoch time: 101.2259597778s, input_fired_time: 1.4105539322, output_fired_time: 0.0000000000, propagate_forward_time: 96.6289443970, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1864624023
validation[          , genome  6397] predictions:    6067/  10000 (60.67%), best:    6142/10000 (61.42%), error:     11936.53711, best error:     11793.69238 on epoch:    65, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 1233.6812744141s, input_fired_time: 10.8612499237, output_fired_time: 9.8484497070, propagate_forward_time: 488.9894714355, propagate_backward_time: 709.5241088867, weight_update_time: 1.4100680351, other_time: 14.4580688477
epoch time: 101.5450668335s, input_fired_time: 1.4377663136, output_fired_time: 0.0000000000, propagate_forward_time: 97.3541793823, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7531204224
validation[          , genome  6397] predictions:    6048/  10000 (60.48%), best:    6142/10000 (61.42%), error:     11996.09668, best error:     11793.69238 on epoch:    65, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 1241.4172363281s, input_fired_time: 10.8527097702, output_fired_time: 9.8770170212, propagate_forward_time: 492.7658081055, propagate_backward_time: 713.5842895508, weight_update_time: 1.4119085073, other_time: 14.3374023438
epoch time: 105.1122283936s, input_fired_time: 1.4525955915, output_fired_time: 0.0000000000, propagate_forward_time: 101.0556640625, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6039657593
validation[          , genome  6397] predictions:    6114/  10000 (61.14%), best:    6114/10000 (61.14%), error:     11760.98633, best error:     11760.98633 on epoch:    79, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 1243.7746582031s, input_fired_time: 11.0652875900, output_fired_time: 9.8993215561, propagate_forward_time: 494.3807067871, propagate_backward_time: 714.0960083008, weight_update_time: 1.4228279591, other_time: 14.3333129883
epoch time: 102.1369857788s, input_fired_time: 1.4171746969, output_fired_time: 0.0000000000, propagate_forward_time: 98.1690673828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5507431030
validation[          , genome  6397] predictions:    6134/  10000 (61.34%), best:    6134/10000 (61.34%), error:     11666.91797, best error:     11666.91797 on epoch:    80, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 1241.4033203125s, input_fired_time: 10.9208250046, output_fired_time: 9.8822174072, propagate_forward_time: 493.0620117188, propagate_backward_time: 713.7122802734, weight_update_time: 1.4210103750, other_time: 13.8260498047
epoch time: 102.1480102539s, input_fired_time: 1.4521718025, output_fired_time: 0.0000000000, propagate_forward_time: 98.1332244873, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5626144409
validation[          , genome  6397] predictions:    6096/  10000 (60.96%), best:    6134/10000 (61.34%), error:     11750.35449, best error:     11666.91797 on epoch:    80, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 1235.9095458984s, input_fired_time: 11.0823173523, output_fired_time: 9.8660326004, propagate_forward_time: 490.1897277832, propagate_backward_time: 709.8428344727, weight_update_time: 1.4215824604, other_time: 14.9286499023
epoch time: 101.9222030640s, input_fired_time: 1.4654104710, output_fired_time: 0.0000000000, propagate_forward_time: 97.7350158691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7217788696
validation[          , genome  6397] predictions:    6077/  10000 (60.77%), best:    6134/10000 (61.34%), error:     11988.30371, best error:     11666.91797 on epoch:    80, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 1248.0977783203s, input_fired_time: 11.0557003021, output_fired_time: 9.9540910721, propagate_forward_time: 495.8954772949, propagate_backward_time: 716.6959228516, weight_update_time: 1.4291316271, other_time: 14.4965820313
epoch time: 103.4294357300s, input_fired_time: 1.4747562408, output_fired_time: 0.0000000000, propagate_forward_time: 99.6143112183, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3403701782
validation[          , genome  6397] predictions:    6073/  10000 (60.73%), best:    6134/10000 (61.34%), error:     11866.94629, best error:     11666.91797 on epoch:    80, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 1241.1387939453s, input_fired_time: 11.1205940247, output_fired_time: 9.9221916199, propagate_forward_time: 491.9735107422, propagate_backward_time: 712.8257446289, weight_update_time: 1.4290041924, other_time: 15.2966918945
epoch time: 102.8645553589s, input_fired_time: 1.4009613991, output_fired_time: 0.0000000000, propagate_forward_time: 99.2131805420, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2504119873
validation[          , genome  6397] predictions:    6076/  10000 (60.76%), best:    6134/10000 (61.34%), error:     11731.94824, best error:     11666.91797 on epoch:    80, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 1238.3634033203s, input_fired_time: 10.8205852509, output_fired_time: 9.8654928207, propagate_forward_time: 492.1423034668, propagate_backward_time: 711.5818481445, weight_update_time: 1.4068096876, other_time: 13.9531860352
epoch time: 104.3714523315s, input_fired_time: 1.4494508505, output_fired_time: 0.0000000000, propagate_forward_time: 100.5265502930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3954544067
validation[          , genome  6397] predictions:    6173/  10000 (61.73%), best:    6134/10000 (61.34%), error:     11691.47168, best error:     11666.91797 on epoch:    80, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 1241.3988037109s, input_fired_time: 11.1286344528, output_fired_time: 9.8928661346, propagate_forward_time: 492.7011108398, propagate_backward_time: 713.4920654297, weight_update_time: 1.4260275364, other_time: 14.1841430664
epoch time: 105.0568237305s, input_fired_time: 1.4509359598, output_fired_time: 0.0000000000, propagate_forward_time: 101.0319671631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5739212036
validation[          , genome  6397] predictions:    6080/  10000 (60.80%), best:    6134/10000 (61.34%), error:     11776.79492, best error:     11666.91797 on epoch:    80, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 1234.0428466797s, input_fired_time: 11.0812501907, output_fired_time: 9.8501062393, propagate_forward_time: 489.1031188965, propagate_backward_time: 709.3804931641, weight_update_time: 1.4193592072, other_time: 14.6278076172
epoch time: 101.6099853516s, input_fired_time: 1.3743195534, output_fired_time: 0.0000000000, propagate_forward_time: 97.3181381226, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9175262451
validation[          , genome  6397] predictions:    6136/  10000 (61.36%), best:    6134/10000 (61.34%), error:     11750.90527, best error:     11666.91797 on epoch:    80, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 1251.6591796875s, input_fired_time: 11.0369958878, output_fired_time: 9.9421510696, propagate_forward_time: 498.0726928711, propagate_backward_time: 719.0217285156, weight_update_time: 1.4224411249, other_time: 13.5856323242
epoch time: 105.3005142212s, input_fired_time: 1.3874702454, output_fired_time: 0.0000000000, propagate_forward_time: 101.6802902222, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2327499390
validation[          , genome  6397] predictions:    6146/  10000 (61.46%), best:    6134/10000 (61.34%), error:     11677.29785, best error:     11666.91797 on epoch:    80, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 1246.1561279297s, input_fired_time: 10.8921337128, output_fired_time: 9.9401826859, propagate_forward_time: 494.6501770020, propagate_backward_time: 715.7719116211, weight_update_time: 1.4186244011, other_time: 14.9017944336
epoch time: 104.1696319580s, input_fired_time: 1.4110513926, output_fired_time: 0.0000000000, propagate_forward_time: 100.4685287476, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2900543213
validation[          , genome  6397] predictions:    6118/  10000 (61.18%), best:    6134/10000 (61.34%), error:     11800.39063, best error:     11666.91797 on epoch:    80, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 1252.3812255859s, input_fired_time: 10.9916906357, output_fired_time: 9.9653396606, propagate_forward_time: 498.6084899902, propagate_backward_time: 718.8636474609, weight_update_time: 1.4238168001, other_time: 13.9520263672
epoch time: 104.7936172485s, input_fired_time: 1.3955895901, output_fired_time: 0.0000000000, propagate_forward_time: 101.1531448364, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2448806763
validation[          , genome  6397] predictions:    6110/  10000 (61.10%), best:    6134/10000 (61.34%), error:     11808.97168, best error:     11666.91797 on epoch:    80, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 1231.1132812500s, input_fired_time: 10.9686498642, output_fired_time: 9.8713150024, propagate_forward_time: 486.0502624512, propagate_backward_time: 707.1913452148, weight_update_time: 1.4263327122, other_time: 17.0316772461
epoch time: 101.7833251953s, input_fired_time: 1.4864070415, output_fired_time: 0.0000000000, propagate_forward_time: 97.9450225830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3518981934
validation[          , genome  6397] predictions:    6132/  10000 (61.32%), best:    6134/10000 (61.34%), error:     11707.73242, best error:     11666.91797 on epoch:    80, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 1242.2512207031s, input_fired_time: 10.8755798340, output_fired_time: 9.8719940186, propagate_forward_time: 493.0329589844, propagate_backward_time: 714.6610107422, weight_update_time: 1.4110867977, other_time: 13.8096923828
epoch time: 102.7645721436s, input_fired_time: 1.4828282595, output_fired_time: 0.0000000000, propagate_forward_time: 98.9548263550, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3269195557
validation[          , genome  6397] predictions:    6160/  10000 (61.60%), best:    6160/10000 (61.60%), error:     11642.71777, best error:     11642.71777 on epoch:    92, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 1243.5140380859s, input_fired_time: 11.0247182846, output_fired_time: 9.9457292557, propagate_forward_time: 492.8471374512, propagate_backward_time: 714.5469360352, weight_update_time: 1.4431641102, other_time: 15.1495971680
epoch time: 104.5266494751s, input_fired_time: 1.4045522213, output_fired_time: 0.0000000000, propagate_forward_time: 100.8071746826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3149261475
validation[          , genome  6397] predictions:    6096/  10000 (60.96%), best:    6160/10000 (61.60%), error:     11811.32617, best error:     11642.71777 on epoch:    92, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 1242.6639404297s, input_fired_time: 10.8353071213, output_fired_time: 9.8775558472, propagate_forward_time: 493.2572326660, propagate_backward_time: 714.9305419922, weight_update_time: 1.4094603062, other_time: 13.7633056641
epoch time: 103.0278015137s, input_fired_time: 1.4374449253, output_fired_time: 0.0000000000, propagate_forward_time: 98.1945114136, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3958435059
validation[          , genome  6397] predictions:    6165/  10000 (61.65%), best:    6165/10000 (61.65%), error:     11641.79102, best error:     11641.79102 on epoch:    94, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 1231.4626464844s, input_fired_time: 10.9789800644, output_fired_time: 9.8458194733, propagate_forward_time: 486.4690246582, propagate_backward_time: 707.6739501953, weight_update_time: 1.4209604263, other_time: 16.4948730469
epoch time: 105.2245941162s, input_fired_time: 1.4196242094, output_fired_time: 0.0000000000, propagate_forward_time: 101.4989700317, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3059997559
validation[          , genome  6397] predictions:    6136/  10000 (61.36%), best:    6165/10000 (61.65%), error:     11670.55566, best error:     11641.79102 on epoch:    94, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_182_6397.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_182_6397_4_r944138045_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.986438
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 9.08606e-05
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 3.63442e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 96
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 94
read number_validation_images: 10000
read best_validation_predictions: 6165
read best_validation_error: 11641.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6397
read normal distribution: '1 -0.641208350658417 2.29480671882629'
generator_str: '2076092847'
read generator: 2076092847
read generated_by_map:
1 split_node 1
reading 75 nodes.
reading 255 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1266.19067382813s, input_fired_time: 11.0361604690552, output_fired_time: 10.0867919921875, propagate_forward_time: 505.274108886719, propagate_backward_time: 725.927551269531, weight_update_time: 1.42737853527069, other_time: 13.8660888671875
epoch time: 103.305419921875s, input_fired_time: 1.43246579170227, output_fired_time: 0, propagate_forward_time: 99.6003036499023, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.27265167236328
validation[          , genome  6397] predictions:    6158/  10000 (61.58%), best:    6165/10000 (61.65%), error:     11771.42383, best error:     11641.79102 on epoch:    94, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 1269.6468505859s, input_fired_time: 11.1151952744, output_fired_time: 10.1383905411, propagate_forward_time: 506.8898925781, propagate_backward_time: 727.2866821289, weight_update_time: 1.4355463982, other_time: 14.2166137695
epoch time: 103.5260162354s, input_fired_time: 1.4218426943, output_fired_time: 0.0000000000, propagate_forward_time: 99.8490600586, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2551116943
validation[          , genome  6397] predictions:    6108/  10000 (61.08%), best:    6165/10000 (61.65%), error:     11862.90137, best error:     11641.79102 on epoch:    94, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 1252.9414062500s, input_fired_time: 11.1020612717, output_fired_time: 9.9920787811, propagate_forward_time: 498.2786560059, propagate_backward_time: 718.9807739258, weight_update_time: 1.4301705360, other_time: 14.5878295898
epoch time: 104.7726974487s, input_fired_time: 1.4338642359, output_fired_time: 0.0000000000, propagate_forward_time: 100.9616928101, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3771438599
validation[          , genome  6397] predictions:    6095/  10000 (60.95%), best:    6165/10000 (61.65%), error:     11961.01172, best error:     11641.79102 on epoch:    94, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 1255.3073730469s, input_fired_time: 11.2063188553, output_fired_time: 10.0014791489, propagate_forward_time: 498.9956054688, propagate_backward_time: 720.7263183594, weight_update_time: 1.4371250868, other_time: 14.3776855469
epoch time: 101.7918090820s, input_fired_time: 1.4374066591, output_fired_time: 0.0000000000, propagate_forward_time: 97.7683792114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5860214233
validation[          , genome  6397] predictions:    6099/  10000 (60.99%), best:    6165/10000 (61.65%), error:     12029.37598, best error:     11641.79102 on epoch:    94, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 1232.5931396484s, input_fired_time: 10.8280649185, output_fired_time: 9.8214797974, propagate_forward_time: 489.5693664551, propagate_backward_time: 709.3576660156, weight_update_time: 1.4000445604, other_time: 13.0164794922
epoch time: 102.4626541138s, input_fired_time: 1.4215627909, output_fired_time: 0.0000000000, propagate_forward_time: 98.8222961426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2187957764
validation[          , genome  6397] predictions:    6112/  10000 (61.12%), best:    6165/10000 (61.65%), error:     11748.71289, best error:     11641.79102 on epoch:    94, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 512.6795043945s, input_fired_time: 7.0435743332, output_fired_time: 0.0000000000, propagate_forward_time: 489.8652343750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 15.7706909180
best training[          , genome  6397] predictions:   39707/  50000 (79.41%), best:    6165/10000 (61.65%), error:     31520.02930, best error:     11641.79102 on epoch:    94, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 100.9934844971s, input_fired_time: 1.3996132612, output_fired_time: 0.0000000000, propagate_forward_time: 96.4872055054, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1066665649
   testing[          , genome  6397] predictions:    6165/  10000 (61.65%), best:    6165/10000 (61.65%), error:     11641.79102, best error:     11641.79102 on epoch:    94, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
13:02:03 (10200): called boinc_finish(0)

</stderr_txt>
]]>