Nameexact_genome_1540746903_182_5338_0
Workunit4075231
Created28 Oct 2018, 19:39:58 UTC
Sent28 Oct 2018, 20:13:03 UTC
Report deadline2 Jan 2019, 22:25:58 UTC
Received9 Dec 2018, 14:33:58 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64195
Run time1 days 5 hours 50 min 30 sec
CPU time1 days 4 hours 51 min 49 sec
Validate stateValid
Credit16,523.08
Device peak FLOPS3.80 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size520.91 MB
Peak swap size540.55 MB
Peak disk usage8.46 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
t error:     13045.87500 on epoch:    22, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 1323.8953857422s, input_fired_time: 11.0362100601, output_fired_time: 9.9816236496, propagate_forward_time: 497.5509948730, propagate_backward_time: 790.9328002930, weight_update_time: 1.3041359186, other_time: 14.3937377930
epoch time: 110.2527389526s, input_fired_time: 1.3992769718, output_fired_time: 0.0000000000, propagate_forward_time: 106.0704269409, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7830352783
validation[          , genome  5338] predictions:    5739/  10000 (57.39%), best:    5739/10000 (57.39%), error:     12715.21777, best error:     12715.21777 on epoch:    35, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 1361.6306152344s, input_fired_time: 11.4612846375, output_fired_time: 10.2771377563, propagate_forward_time: 510.6819458008, propagate_backward_time: 814.0419311523, weight_update_time: 1.4013966322, other_time: 15.1683349609
epoch time: 105.3265228271s, input_fired_time: 1.4419240952, output_fired_time: 0.0000000000, propagate_forward_time: 101.3472213745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5373764038
validation[          , genome  5338] predictions:    5860/  10000 (58.60%), best:    5860/10000 (58.60%), error:     12672.33789, best error:     12672.33789 on epoch:    36, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 1322.6010742188s, input_fired_time: 11.2288866043, output_fired_time: 9.8256444931, propagate_forward_time: 494.8906250000, propagate_backward_time: 792.2006225586, weight_update_time: 1.2593157291, other_time: 14.4552612305
epoch time: 107.4799575806s, input_fired_time: 1.4262146950, output_fired_time: 0.0000000000, propagate_forward_time: 103.4769287109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5768127441
validation[          , genome  5338] predictions:    5776/  10000 (57.76%), best:    5860/10000 (58.60%), error:     12915.41797, best error:     12672.33789 on epoch:    36, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 1315.6519775391s, input_fired_time: 10.7961483002, output_fired_time: 9.4689550400, propagate_forward_time: 492.9106445313, propagate_backward_time: 788.1016235352, weight_update_time: 1.2117795944, other_time: 14.3745727539
epoch time: 102.4308013916s, input_fired_time: 1.4417607784, output_fired_time: 0.0000000000, propagate_forward_time: 98.5920562744, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3969879150
validation[          , genome  5338] predictions:    5910/  10000 (59.10%), best:    5910/10000 (59.10%), error:     12316.54590, best error:     12316.54590 on epoch:    38, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 1266.7237548828s, input_fired_time: 10.5089445114, output_fired_time: 9.5050449371, propagate_forward_time: 475.7123413086, propagate_backward_time: 756.9180908203, weight_update_time: 1.2287983894, other_time: 14.0794067383
epoch time: 100.4693450928s, input_fired_time: 1.3961763382, output_fired_time: 0.0000000000, propagate_forward_time: 96.5514678955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5216979980
validation[          , genome  5338] predictions:    5777/  10000 (57.77%), best:    5910/10000 (59.10%), error:     12734.34863, best error:     12316.54590 on epoch:    38, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 1289.0087890625s, input_fired_time: 10.6164751053, output_fired_time: 9.6820182800, propagate_forward_time: 483.9809570313, propagate_backward_time: 770.6694335938, weight_update_time: 1.1982129812, other_time: 14.0599365234
epoch time: 103.1564331055s, input_fired_time: 1.3746381998, output_fired_time: 0.0000000000, propagate_forward_time: 99.2308502197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5509414673
validation[          , genome  5338] predictions:    5787/  10000 (57.87%), best:    5910/10000 (59.10%), error:     12796.80566, best error:     12316.54590 on epoch:    38, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 1300.0089111328s, input_fired_time: 11.0569238663, output_fired_time: 9.4391050339, propagate_forward_time: 487.0000915527, propagate_backward_time: 778.3972778320, weight_update_time: 1.2440801859, other_time: 14.1155395508
epoch time: 101.9353103638s, input_fired_time: 1.4501367807, output_fired_time: 0.0000000000, propagate_forward_time: 97.9282913208, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5568847656
validation[          , genome  5338] predictions:    5913/  10000 (59.13%), best:    5910/10000 (59.10%), error:     12465.36719, best error:     12316.54590 on epoch:    38, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 1285.6208496094s, input_fired_time: 10.7406501770, output_fired_time: 9.4496688843, propagate_forward_time: 481.3329467773, propagate_backward_time: 770.1621093750, weight_update_time: 1.3700400591, other_time: 13.9354858398
epoch time: 102.3058700562s, input_fired_time: 1.4079015255, output_fired_time: 0.0000000000, propagate_forward_time: 98.2715148926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6264572144
validation[          , genome  5338] predictions:    5812/  10000 (58.12%), best:    5910/10000 (59.10%), error:     12758.74121, best error:     12316.54590 on epoch:    38, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 1286.2789306641s, input_fired_time: 10.8423519135, output_fired_time: 9.4058618546, propagate_forward_time: 481.0278625488, propagate_backward_time: 770.5197143555, weight_update_time: 1.3222398758, other_time: 14.4830932617
epoch time: 97.4449691772s, input_fired_time: 1.4119329453, output_fired_time: 0.0000000000, propagate_forward_time: 93.5884323120, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4446029663
validation[          , genome  5338] predictions:    5701/  10000 (57.01%), best:    5910/10000 (59.10%), error:     12975.38086, best error:     12316.54590 on epoch:    38, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338_0_r2135223497_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.93871
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00130842
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 5.23369e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 44
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 38
read number_validation_images: 10000
read best_validation_predictions: 5910
read best_validation_error: 12316.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5338
read normal distribution: '1 0 0'
generator_str: '2121658122'
read generator: 2121658122
read generated_by_map:
1 disable_edge 1
reading 65 nodes.
reading 228 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338_0_r2135223497_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.93871
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00130842
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 5.23369e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 44
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 38
read number_validation_images: 10000
read best_validation_predictions: 5910
read best_validation_error: 12316.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5338
read normal distribution: '1 0 0'
generator_str: '2121658122'
read generator: 2121658122
read generated_by_map:
1 disable_edge 1
reading 65 nodes.
reading 228 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 734.783569335938s, input_fired_time: 7.68180751800537, output_fired_time: 5.8289361000061, propagate_forward_time: 259.630096435547, propagate_backward_time: 451.550140380859, weight_update_time: 0.891058683395386, other_time: 10.0925903320313
epoch time: 83.9439010620117s, input_fired_time: 1.21303772926331, output_fired_time: 0, propagate_forward_time: 80.4220962524414, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.30876922607422
validation[          , genome  5338] predictions:    5765/  10000 (57.65%), best:    5910/10000 (59.10%), error:     12983.62988, best error:     12316.54590 on epoch:    38, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 691.0270385742s, input_fired_time: 7.4598212242, output_fired_time: 5.5153102875, propagate_forward_time: 241.4418487549, propagate_backward_time: 426.9059143066, weight_update_time: 0.8645876646, other_time: 9.7041320801
epoch time: 66.5987701416s, input_fired_time: 1.0255032778, output_fired_time: 0.0000000000, propagate_forward_time: 63.6840476990, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8892173767
validation[          , genome  5338] predictions:    5847/  10000 (58.47%), best:    5910/10000 (59.10%), error:     12727.48730, best error:     12316.54590 on epoch:    38, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 734.4924926758s, input_fired_time: 7.6932849884, output_fired_time: 5.8255591393, propagate_forward_time: 259.6572265625, propagate_backward_time: 451.2819519043, weight_update_time: 0.8912287951, other_time: 10.0344543457
epoch time: 51.9719390869s, input_fired_time: 0.9062629938, output_fired_time: 0.0000000000, propagate_forward_time: 49.3111267090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7545509338
validation[          , genome  5338] predictions:    5651/  10000 (56.51%), best:    5910/10000 (59.10%), error:     13398.61816, best error:     12316.54590 on epoch:    38, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 740.1779174805s, input_fired_time: 7.7145242691, output_fired_time: 5.8190727234, propagate_forward_time: 261.8914489746, propagate_backward_time: 454.7527160645, weight_update_time: 0.8924036026, other_time: 10.0001220703
epoch time: 49.7057876587s, input_fired_time: 0.8743388057, output_fired_time: 0.0000000000, propagate_forward_time: 47.1038169861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7276306152
validation[          , genome  5338] predictions:    5827/  10000 (58.27%), best:    5910/10000 (59.10%), error:     12677.26074, best error:     12316.54590 on epoch:    38, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 675.2928466797s, input_fired_time: 7.4279303551, output_fired_time: 5.4290924072, propagate_forward_time: 234.9118957520, propagate_backward_time: 417.8571472168, weight_update_time: 0.8563175797, other_time: 9.6668090820
epoch time: 53.3079414368s, input_fired_time: 0.9162660241, output_fired_time: 0.0000000000, propagate_forward_time: 50.6222076416, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7694664001
validation[          , genome  5338] predictions:    5880/  10000 (58.80%), best:    5910/10000 (59.10%), error:     12571.26660, best error:     12316.54590 on epoch:    38, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 742.9807739258s, input_fired_time: 7.7569651604, output_fired_time: 5.8788366318, propagate_forward_time: 262.8794555664, propagate_backward_time: 456.3536376953, weight_update_time: 0.8997952342, other_time: 10.1118774414
epoch time: 55.4658203125s, input_fired_time: 0.9431735277, output_fired_time: 0.0000000000, propagate_forward_time: 52.7068977356, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8157501221
validation[          , genome  5338] predictions:    5833/  10000 (58.33%), best:    5910/10000 (59.10%), error:     12740.57422, best error:     12316.54590 on epoch:    38, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 702.6812744141s, input_fired_time: 7.5306086540, output_fired_time: 5.5589880943, propagate_forward_time: 246.6301879883, propagate_backward_time: 433.2427062988, weight_update_time: 0.8699383140, other_time: 9.7188415527
epoch time: 49.0851249695s, input_fired_time: 0.8808355927, output_fired_time: 0.0000000000, propagate_forward_time: 46.4996032715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7046852112
validation[          , genome  5338] predictions:    5948/  10000 (59.48%), best:    5948/10000 (59.48%), error:     12306.13379, best error:     12306.13379 on epoch:    50, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 742.7236328125s, input_fired_time: 7.8017644882, output_fired_time: 5.9218235016, propagate_forward_time: 262.3970336914, propagate_backward_time: 456.4284973145, weight_update_time: 0.8986420035, other_time: 10.1745300293
epoch time: 46.6639060974s, input_fired_time: 0.8532946706, output_fired_time: 0.0000000000, propagate_forward_time: 44.1396980286, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6709136963
validation[          , genome  5338] predictions:    5882/  10000 (58.82%), best:    5948/10000 (59.48%), error:     12432.62402, best error:     12306.13379 on epoch:    50, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 766.7964477539s, input_fired_time: 7.8775243759, output_fired_time: 6.0113058090, propagate_forward_time: 272.8056945801, propagate_backward_time: 469.8916625977, weight_update_time: 0.9107860327, other_time: 10.2102966309
epoch time: 55.6981849670s, input_fired_time: 0.9357592463, output_fired_time: 0.0000000000, propagate_forward_time: 52.9988136292, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7636108398
validation[          , genome  5338] predictions:    5892/  10000 (58.92%), best:    5892/10000 (58.92%), error:     12098.15430, best error:     12098.15430 on epoch:    52, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 719.8991699219s, input_fired_time: 7.6770467758, output_fired_time: 5.7269306183, propagate_forward_time: 253.2153930664, propagate_backward_time: 443.3244018555, weight_update_time: 0.8824334145, other_time: 9.9553833008
epoch time: 85.9573135376s, input_fired_time: 1.2342426777, output_fired_time: 0.0000000000, propagate_forward_time: 82.5076675415, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2154006958
validation[          , genome  5338] predictions:    5817/  10000 (58.17%), best:    5892/10000 (58.92%), error:     12943.04102, best error:     12098.15430 on epoch:    52, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 828.5406494141s, input_fired_time: 8.2464218140, output_fired_time: 6.5162997246, propagate_forward_time: 297.3533630371, propagate_backward_time: 505.6723632813, weight_update_time: 0.9524260759, other_time: 10.7521972656
epoch time: 56.3138427734s, input_fired_time: 0.9496281743, output_fired_time: 0.0000000000, propagate_forward_time: 53.5672760010, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7969398499
validation[          , genome  5338] predictions:    5970/  10000 (59.70%), best:    5892/10000 (58.92%), error:     12149.31543, best error:     12098.15430 on epoch:    52, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 840.8445434570s, input_fired_time: 8.3062276840, output_fired_time: 6.5835833549, propagate_forward_time: 302.9594726563, propagate_backward_time: 512.1244506836, weight_update_time: 0.9601241350, other_time: 10.8708496094
epoch time: 88.2361450195s, input_fired_time: 1.2581306696, output_fired_time: 0.0000000000, propagate_forward_time: 84.6819610596, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2960510254
validation[          , genome  5338] predictions:    5774/  10000 (57.74%), best:    5892/10000 (58.92%), error:     12809.06348, best error:     12098.15430 on epoch:    52, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 795.6444702148s, input_fired_time: 8.0227241516, output_fired_time: 6.1744565964, propagate_forward_time: 284.5787353516, propagate_backward_time: 486.5712280273, weight_update_time: 0.9260138869, other_time: 10.2973632813
epoch time: 57.1879920959s, input_fired_time: 0.9707254767, output_fired_time: 0.0000000000, propagate_forward_time: 54.3689689636, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8482971191
validation[          , genome  5338] predictions:    5969/  10000 (59.69%), best:    5892/10000 (58.92%), error:     12306.44336, best error:     12098.15430 on epoch:    52, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 788.8792724609s, input_fired_time: 8.0081520081, output_fired_time: 6.1562366486, propagate_forward_time: 281.8240356445, propagate_backward_time: 482.5634155273, weight_update_time: 0.9239003062, other_time: 10.3273925781
epoch time: 67.5353927612s, input_fired_time: 1.0621372461, output_fired_time: 0.0000000000, propagate_forward_time: 64.5117416382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9615173340
validation[          , genome  5338] predictions:    6094/  10000 (60.94%), best:    6094/10000 (60.94%), error:     11964.59180, best error:     11964.59180 on epoch:    57, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 720.8487548828s, input_fired_time: 7.6992774010, output_fired_time: 5.7204341888, propagate_forward_time: 253.9602508545, propagate_backward_time: 443.5389709473, weight_update_time: 0.8846163154, other_time: 9.9297790527
epoch time: 76.0724945068s, input_fired_time: 1.1385275126, output_fired_time: 0.0000000000, propagate_forward_time: 72.9069671631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0270004272
validation[          , genome  5338] predictions:    5936/  10000 (59.36%), best:    6094/10000 (60.94%), error:     12416.01172, best error:     11964.59180 on epoch:    57, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 780.6411743164s, input_fired_time: 7.9914646149, output_fired_time: 6.1140170097, propagate_forward_time: 278.5590515137, propagate_backward_time: 477.6581726074, weight_update_time: 0.9179587364, other_time: 10.3184814453
epoch time: 66.3484649658s, input_fired_time: 1.0588588715, output_fired_time: 0.0000000000, propagate_forward_time: 63.2920951843, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9975090027
validation[          , genome  5338] predictions:    5935/  10000 (59.35%), best:    6094/10000 (60.94%), error:     12496.46191, best error:     11964.59180 on epoch:    57, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 717.4278564453s, input_fired_time: 7.7123675346, output_fired_time: 5.7316622734, propagate_forward_time: 252.4605865479, propagate_backward_time: 441.5536193848, weight_update_time: 0.8828383684, other_time: 9.9696350098
epoch time: 46.3643341064s, input_fired_time: 0.8632823825, output_fired_time: 0.0000000000, propagate_forward_time: 43.8421478271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6589050293
validation[          , genome  5338] predictions:    5995/  10000 (59.95%), best:    6094/10000 (60.94%), error:     12026.80469, best error:     11964.59180 on epoch:    57, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 854.6730346680s, input_fired_time: 8.3485002518, output_fired_time: 6.5901975632, propagate_forward_time: 308.9673767090, propagate_backward_time: 519.9842529297, weight_update_time: 0.9640300274, other_time: 10.7827148438
epoch time: 69.4467544556s, input_fired_time: 1.0679820776, output_fired_time: 0.0000000000, propagate_forward_time: 66.4503936768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9283752441
validation[          , genome  5338] predictions:    5948/  10000 (59.48%), best:    6094/10000 (60.94%), error:     12291.81055, best error:     11964.59180 on epoch:    57, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 811.4686279297s, input_fired_time: 8.2063961029, output_fired_time: 6.3867964745, propagate_forward_time: 291.1225280762, propagate_backward_time: 495.0845336914, weight_update_time: 0.9435930252, other_time: 10.6683654785
epoch time: 48.7153930664s, input_fired_time: 0.8858653307, output_fired_time: 0.0000000000, propagate_forward_time: 46.1326942444, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6968345642
validation[          , genome  5338] predictions:    5909/  10000 (59.09%), best:    6094/10000 (60.94%), error:     12374.48145, best error:     11964.59180 on epoch:    57, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 773.1269531250s, input_fired_time: 7.9871950150, output_fired_time: 6.0674629211, propagate_forward_time: 275.2315063477, propagate_backward_time: 473.5401916504, weight_update_time: 0.9143878222, other_time: 10.3006286621
epoch time: 68.1322250366s, input_fired_time: 1.0794206858, output_fired_time: 0.0000000000, propagate_forward_time: 65.0257339478, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0270690918
validation[          , genome  5338] predictions:    5986/  10000 (59.86%), best:    6094/10000 (60.94%), error:     12183.57129, best error:     11964.59180 on epoch:    57, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 705.9102783203s, input_fired_time: 7.6514272690, output_fired_time: 5.6120829582, propagate_forward_time: 247.6581420898, propagate_backward_time: 435.1661682129, weight_update_time: 0.8765548468, other_time: 9.8224792480
epoch time: 57.5544738770s, input_fired_time: 0.9790491462, output_fired_time: 0.0000000000, propagate_forward_time: 54.7363319397, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8390922546
validation[          , genome  5338] predictions:    6070/  10000 (60.70%), best:    6070/10000 (60.70%), error:     11684.31738, best error:     11684.31738 on epoch:    64, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 792.2371826172s, input_fired_time: 8.1052560806, output_fired_time: 6.2002968788, propagate_forward_time: 283.1466064453, propagate_backward_time: 484.3298034668, weight_update_time: 0.9278018475, other_time: 10.4551696777
epoch time: 77.8604965210s, input_fired_time: 1.1532765627, output_fired_time: 0.0000000000, propagate_forward_time: 74.6864547729, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0207672119
validation[          , genome  5338] predictions:    5983/  10000 (59.83%), best:    6070/10000 (60.70%), error:     12192.50488, best error:     11684.31738 on epoch:    64, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 784.7921752930s, input_fired_time: 8.0405282974, output_fired_time: 6.1301012039, propagate_forward_time: 280.1520996094, propagate_backward_time: 480.1448059082, weight_update_time: 0.9216089845, other_time: 10.3246154785
epoch time: 46.5725936890s, input_fired_time: 0.8754168749, output_fired_time: 0.0000000000, propagate_forward_time: 44.0367622375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6604156494
validation[          , genome  5338] predictions:    6109/  10000 (61.09%), best:    6070/10000 (60.70%), error:     11810.40430, best error:     11684.31738 on epoch:    64, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 837.5217895508s, input_fired_time: 8.3341321945, output_fired_time: 6.5106449127, propagate_forward_time: 301.7617187500, propagate_backward_time: 510.1100158691, weight_update_time: 0.9563651085, other_time: 10.8053283691
epoch time: 70.4250030518s, input_fired_time: 1.1113030910, output_fired_time: 0.0000000000, propagate_forward_time: 67.2483978271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0652999878
validation[          , genome  5338] predictions:    5988/  10000 (59.88%), best:    6070/10000 (60.70%), error:     12165.14746, best error:     11684.31738 on epoch:    64, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 817.6239013672s, input_fired_time: 8.2596435547, output_fired_time: 6.3947453499, propagate_forward_time: 293.2694091797, propagate_backward_time: 499.0258178711, weight_update_time: 0.9441793561, other_time: 10.6742553711
epoch time: 85.8812789917s, input_fired_time: 1.2754585743, output_fired_time: 0.0000000000, propagate_forward_time: 82.2672119141, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3386077881
validation[          , genome  5338] predictions:    6126/  10000 (61.26%), best:    6126/10000 (61.26%), error:     11575.02344, best error:     11575.02344 on epoch:    68, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 746.4763183594s, input_fired_time: 7.8739657402, output_fired_time: 5.8865032196, propagate_forward_time: 264.2626647949, propagate_backward_time: 458.3544616699, weight_update_time: 0.8980836272, other_time: 10.0987548828
epoch time: 75.3285140991s, input_fired_time: 1.1540588140, output_fired_time: 0.0000000000, propagate_forward_time: 72.0657272339, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1087265015
validation[          , genome  5338] predictions:    6047/  10000 (60.47%), best:    6126/10000 (61.26%), error:     11853.14941, best error:     11575.02344 on epoch:    68, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 774.2236938477s, input_fired_time: 7.9938197136, output_fired_time: 6.0414533615, propagate_forward_time: 276.1183776855, propagate_backward_time: 473.8766784668, weight_update_time: 0.9137575626, other_time: 10.1933593750
epoch time: 46.6216888428s, input_fired_time: 0.8726629019, output_fired_time: 0.0000000000, propagate_forward_time: 44.0901832581, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6588439941
validation[          , genome  5338] predictions:    6007/  10000 (60.07%), best:    6126/10000 (61.26%), error:     12022.48828, best error:     11575.02344 on epoch:    68, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 808.4986572266s, input_fired_time: 8.2078571320, output_fired_time: 6.3199973106, propagate_forward_time: 289.9876403809, propagate_backward_time: 493.4085083008, weight_update_time: 0.9374150634, other_time: 10.5746154785
epoch time: 89.0483627319s, input_fired_time: 1.2975735664, output_fired_time: 0.0000000000, propagate_forward_time: 85.4334335327, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3173522949
validation[          , genome  5338] predictions:    6026/  10000 (60.26%), best:    6126/10000 (61.26%), error:     11942.39551, best error:     11575.02344 on epoch:    68, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 848.9024658203s, input_fired_time: 8.3878192902, output_fired_time: 6.5670843124, propagate_forward_time: 306.6301879883, propagate_backward_time: 516.5367431641, weight_update_time: 0.9618393779, other_time: 10.7806396484
epoch time: 84.9085006714s, input_fired_time: 1.2593828440, output_fired_time: 0.0000000000, propagate_forward_time: 81.3515014648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2976150513
validation[          , genome  5338] predictions:    6068/  10000 (60.68%), best:    6126/10000 (61.26%), error:     12093.74512, best error:     11575.02344 on epoch:    68, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 903.1116943359s, input_fired_time: 8.6374845505, output_fired_time: 6.9065589905, propagate_forward_time: 328.8094787598, propagate_backward_time: 547.7011108398, weight_update_time: 0.9929963350, other_time: 11.0570678711
epoch time: 63.8104553223s, input_fired_time: 1.0511567593, output_fired_time: 0.0000000000, propagate_forward_time: 60.8127098083, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9465904236
validation[          , genome  5338] predictions:    6099/  10000 (60.99%), best:    6126/10000 (61.26%), error:     11797.82813, best error:     11575.02344 on epoch:    68, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 925.7728271484s, input_fired_time: 8.7525081635, output_fired_time: 7.0691704750, propagate_forward_time: 338.7454528809, propagate_backward_time: 559.9805908203, weight_update_time: 1.0069490671, other_time: 11.2250976563
epoch time: 83.7002487183s, input_fired_time: 1.2260622978, output_fired_time: 0.0000000000, propagate_forward_time: 80.3148651123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1593246460
validation[          , genome  5338] predictions:    6067/  10000 (60.67%), best:    6126/10000 (61.26%), error:     11971.08887, best error:     11575.02344 on epoch:    68, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 911.2710571289s, input_fired_time: 8.7078533173, output_fired_time: 7.0036740303, propagate_forward_time: 332.6391906738, propagate_backward_time: 551.7347412109, weight_update_time: 1.0005044937, other_time: 11.1856689453
epoch time: 74.2204895020s, input_fired_time: 1.1383546591, output_fired_time: 0.0000000000, propagate_forward_time: 71.0662231445, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0159149170
validation[          , genome  5338] predictions:    6035/  10000 (60.35%), best:    6126/10000 (61.26%), error:     12256.89063, best error:     11575.02344 on epoch:    68, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 844.8109130859s, input_fired_time: 8.3681755066, output_fired_time: 6.5228157043, propagate_forward_time: 304.8524169922, propagate_backward_time: 514.3660278320, weight_update_time: 0.9571889043, other_time: 10.7014770508
epoch time: 86.8480911255s, input_fired_time: 1.2440611124, output_fired_time: 0.0000000000, propagate_forward_time: 83.4634170532, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1406097412
validation[          , genome  5338] predictions:    6131/  10000 (61.31%), best:    6126/10000 (61.26%), error:     11718.67480, best error:     11575.02344 on epoch:    68, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 934.3914184570s, input_fired_time: 8.8136854172, output_fired_time: 7.1218705177, propagate_forward_time: 341.7716064453, propagate_backward_time: 565.3894042969, weight_update_time: 1.0131686926, other_time: 11.2948608398
epoch time: 88.2563171387s, input_fired_time: 1.2707499266, output_fired_time: 0.0000000000, propagate_forward_time: 84.7818679810, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2036972046
validation[          , genome  5338] predictions:    6104/  10000 (61.04%), best:    6126/10000 (61.26%), error:     11661.10840, best error:     11575.02344 on epoch:    68, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 922.5167846680s, input_fired_time: 8.7623147964, output_fired_time: 7.0524644852, propagate_forward_time: 336.5787353516, propagate_backward_time: 558.9073486328, weight_update_time: 1.0061392784, other_time: 11.2158813477
epoch time: 57.3867492676s, input_fired_time: 0.9832453728, output_fired_time: 0.0000000000, propagate_forward_time: 54.6044540405, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7990493774
validation[          , genome  5338] predictions:    6108/  10000 (61.08%), best:    6126/10000 (61.26%), error:     11709.81836, best error:     11575.02344 on epoch:    68, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 949.2466430664s, input_fired_time: 8.8578910828, output_fired_time: 7.1873240471, propagate_forward_time: 347.8118896484, propagate_backward_time: 574.1083984375, weight_update_time: 1.0168159008, other_time: 11.2811279297
epoch time: 51.9797782898s, input_fired_time: 0.9386762977, output_fired_time: 0.0000000000, propagate_forward_time: 49.2985076904, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7425956726
validation[          , genome  5338] predictions:    6102/  10000 (61.02%), best:    6126/10000 (61.26%), error:     11806.47363, best error:     11575.02344 on epoch:    68, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 989.6498413086s, input_fired_time: 9.0704221725, output_fired_time: 7.4548625946, propagate_forward_time: 364.7137756348, propagate_backward_time: 596.8321533203, weight_update_time: 1.0432603359, other_time: 11.5786132813
epoch time: 68.8543243408s, input_fired_time: 1.0770304203, output_fired_time: 0.0000000000, propagate_forward_time: 65.8648834229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9124069214
validation[          , genome  5338] predictions:    5998/  10000 (59.98%), best:    6126/10000 (61.26%), error:     12256.60449, best error:     11575.02344 on epoch:    68, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 990.0217285156s, input_fired_time: 9.0572614670, output_fired_time: 7.4241471291, propagate_forward_time: 365.6098937988, propagate_backward_time: 596.4347534180, weight_update_time: 1.0421115160, other_time: 11.4956665039
epoch time: 90.6693649292s, input_fired_time: 1.2746601105, output_fired_time: 0.0000000000, propagate_forward_time: 87.2307052612, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1640014648
validation[          , genome  5338] predictions:    6061/  10000 (60.61%), best:    6126/10000 (61.26%), error:     11956.97559, best error:     11575.02344 on epoch:    68, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 993.2890625000s, input_fired_time: 9.1226453781, output_fired_time: 7.5305237770, propagate_forward_time: 365.8211059570, propagate_backward_time: 599.0983886719, weight_update_time: 1.0496824980, other_time: 11.7164306641
epoch time: 86.7054824829s, input_fired_time: 1.2863239050, output_fired_time: 0.0000000000, propagate_forward_time: 83.0763931274, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3427658081
validation[          , genome  5338] predictions:    6102/  10000 (61.02%), best:    6126/10000 (61.26%), error:     11710.15527, best error:     11575.02344 on epoch:    68, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 999.7359619141s, input_fired_time: 9.0959692001, output_fired_time: 7.4724879265, propagate_forward_time: 369.4390563965, propagate_backward_time: 602.1649169922, weight_update_time: 1.0481122732, other_time: 11.5635986328
epoch time: 82.8018875122s, input_fired_time: 1.2508367300, output_fired_time: 0.0000000000, propagate_forward_time: 79.2795639038, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2714843750
validation[          , genome  5338] predictions:    6107/  10000 (61.07%), best:    6126/10000 (61.26%), error:     11772.37988, best error:     11575.02344 on epoch:    68, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 1010.8576660156s, input_fired_time: 9.1648206711, output_fired_time: 7.5410623550, propagate_forward_time: 374.8706665039, propagate_backward_time: 607.6389770508, weight_update_time: 1.0573440790, other_time: 11.6421508789
epoch time: 80.2468338013s, input_fired_time: 1.1872295141, output_fired_time: 0.0000000000, propagate_forward_time: 77.0164794922, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0431213379
validation[          , genome  5338] predictions:    6144/  10000 (61.44%), best:    6126/10000 (61.26%), error:     11773.76465, best error:     11575.02344 on epoch:    68, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 1031.0571289063s, input_fired_time: 9.2182664871, output_fired_time: 7.6391310692, propagate_forward_time: 382.2049255371, propagate_backward_time: 620.3237915039, weight_update_time: 1.0655156374, other_time: 11.6709594727
epoch time: 65.5135955811s, input_fired_time: 1.0653072596, output_fired_time: 0.0000000000, propagate_forward_time: 62.5083656311, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9399223328
validation[          , genome  5338] predictions:    6029/  10000 (60.29%), best:    6126/10000 (61.26%), error:     12140.20605, best error:     11575.02344 on epoch:    68, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 922.9933471680s, input_fired_time: 8.7673625946, output_fired_time: 7.0153622627, propagate_forward_time: 337.5635070801, propagate_backward_time: 558.4635009766, weight_update_time: 1.0060950518, other_time: 11.1835937500
epoch time: 67.7725219727s, input_fired_time: 1.0785895586, output_fired_time: 0.0000000000, propagate_forward_time: 64.7977905273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8961410522
validation[          , genome  5338] predictions:    6071/  10000 (60.71%), best:    6126/10000 (61.26%), error:     11855.95020, best error:     11575.02344 on epoch:    68, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 967.3817749023s, input_fired_time: 8.9983415604, output_fired_time: 7.3232731819, propagate_forward_time: 356.8570861816, propagate_backward_time: 582.7460937500, weight_update_time: 1.0307193995, other_time: 11.4569091797
epoch time: 63.4135208130s, input_fired_time: 1.0436655283, output_fired_time: 0.0000000000, propagate_forward_time: 60.4855308533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8843231201
validation[          , genome  5338] predictions:    6108/  10000 (61.08%), best:    6126/10000 (61.26%), error:     11780.13672, best error:     11575.02344 on epoch:    68, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 969.2810058594s, input_fired_time: 8.9670820236, output_fired_time: 7.2628860474, propagate_forward_time: 357.0022583008, propagate_backward_time: 584.6004028320, weight_update_time: 1.0348167419, other_time: 11.4483642578
epoch time: 71.5807418823s, input_fired_time: 1.1217024326, output_fired_time: 0.0000000000, propagate_forward_time: 68.4692077637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9898300171
validation[          , genome  5338] predictions:    6118/  10000 (61.18%), best:    6126/10000 (61.26%), error:     11664.37598, best error:     11575.02344 on epoch:    68, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 976.4290161133s, input_fired_time: 9.0145845413, output_fired_time: 7.3698396683, propagate_forward_time: 359.1635742188, propagate_backward_time: 589.3750000000, weight_update_time: 1.0371351242, other_time: 11.5060424805
epoch time: 74.1669540405s, input_fired_time: 1.1460090876, output_fired_time: 0.0000000000, propagate_forward_time: 71.0047149658, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0162277222
validation[          , genome  5338] predictions:    6067/  10000 (60.67%), best:    6126/10000 (61.26%), error:     11929.37500, best error:     11575.02344 on epoch:    68, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 956.9508056641s, input_fired_time: 8.9268512726, output_fired_time: 7.2373147011, propagate_forward_time: 352.4320373535, propagate_backward_time: 577.0439453125, weight_update_time: 1.0227754116, other_time: 11.3106689453
epoch time: 69.3894958496s, input_fired_time: 1.0943925381, output_fired_time: 0.0000000000, propagate_forward_time: 66.3689880371, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9261169434
validation[          , genome  5338] predictions:    6137/  10000 (61.37%), best:    6126/10000 (61.26%), error:     11812.16992, best error:     11575.02344 on epoch:    68, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1540746903_182_5338_0_r2135223497_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.985397
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000117424
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 4.69697e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 91
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 68
read number_validation_images: 10000
read best_validation_predictions: 6126
read best_validation_error: 11575
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5338
read normal distribution: '1 0 0'
generator_str: '1040165942'
read generator: 1040165942
read generated_by_map:
1 disable_edge 1
reading 65 nodes.
reading 228 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1112.62219238281s, input_fired_time: 18.9974613189697, output_fired_time: 15.4507713317871, propagate_forward_time: 384.50634765625, propagate_backward_time: 646.804809570313, weight_update_time: 2.13532853126526, other_time: 46.86279296875
epoch time: 91.3238220214844s, input_fired_time: 2.87337803840637, output_fired_time: 0, propagate_forward_time: 78.9857635498047, propagate_backward_time: 0, weight_update_time: 0, other_time: 9.46468353271484
validation[          , genome  5338] predictions:    6073/  10000 (60.73%), best:    6126/10000 (61.26%), error:     11826.40723, best error:     11575.02344 on epoch:    68, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 1126.3137207031s, input_fired_time: 18.3857154846, output_fired_time: 15.0523138046, propagate_forward_time: 390.1838684082, propagate_backward_time: 657.2312011719, weight_update_time: 2.0918829441, other_time: 45.4605712891
epoch time: 82.2788314819s, input_fired_time: 2.5970551968, output_fired_time: 0.0000000000, propagate_forward_time: 71.9660568237, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.7157211304
validation[          , genome  5338] predictions:    6133/  10000 (61.33%), best:    6126/10000 (61.26%), error:     11686.43945, best error:     11575.02344 on epoch:    68, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 1110.5656738281s, input_fired_time: 18.2447681427, output_fired_time: 14.9039096832, propagate_forward_time: 385.4962768555, propagate_backward_time: 646.8876953125, weight_update_time: 2.0713119507, other_time: 45.0330200195
epoch time: 81.9527740479s, input_fired_time: 2.5991442204, output_fired_time: 0.0000000000, propagate_forward_time: 72.1407394409, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2128906250
validation[          , genome  5338] predictions:    6116/  10000 (61.16%), best:    6126/10000 (61.26%), error:     11778.89355, best error:     11575.02344 on epoch:    68, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 1113.1582031250s, input_fired_time: 17.9622020721, output_fired_time: 14.6565723419, propagate_forward_time: 385.7978210449, propagate_backward_time: 650.8831176758, weight_update_time: 2.0708494186, other_time: 43.8584594727
epoch time: 94.5125503540s, input_fired_time: 2.8229839802, output_fired_time: 0.0000000000, propagate_forward_time: 82.6032714844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.0862960815
validation[          , genome  5338] predictions:    6103/  10000 (61.03%), best:    6126/10000 (61.26%), error:     11806.89844, best error:     11575.02344 on epoch:    68, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 1159.7574462891s, input_fired_time: 18.5076694489, output_fired_time: 15.1191244125, propagate_forward_time: 404.1821899414, propagate_backward_time: 675.4441528320, weight_update_time: 2.1137270927, other_time: 46.5042724609
epoch time: 92.0129013062s, input_fired_time: 2.7797591686, output_fired_time: 0.0000000000, propagate_forward_time: 80.6955718994, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.5375671387
validation[          , genome  5338] predictions:    6181/  10000 (61.81%), best:    6126/10000 (61.26%), error:     11634.73535, best error:     11575.02344 on epoch:    68, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 1142.0817871094s, input_fired_time: 18.3878383636, output_fired_time: 14.9185409546, propagate_forward_time: 400.7979125977, propagate_backward_time: 662.3953247070, weight_update_time: 2.0847878456, other_time: 45.5821533203
epoch time: 91.0657730103s, input_fired_time: 2.6546638012, output_fired_time: 0.0000000000, propagate_forward_time: 79.7452774048, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.6658325195
validation[          , genome  5338] predictions:    6111/  10000 (61.11%), best:    6126/10000 (61.26%), error:     11717.52344, best error:     11575.02344 on epoch:    68, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 1144.2744140625s, input_fired_time: 18.5115528107, output_fired_time: 15.0636157990, propagate_forward_time: 397.3339843750, propagate_backward_time: 667.6848754883, weight_update_time: 2.0874006748, other_time: 45.6803588867
epoch time: 88.9366607666s, input_fired_time: 2.7867794037, output_fired_time: 0.0000000000, propagate_forward_time: 77.3864059448, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.7634735107
validation[          , genome  5338] predictions:    6143/  10000 (61.43%), best:    6126/10000 (61.26%), error:     11719.25684, best error:     11575.02344 on epoch:    68, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 1150.1614990234s, input_fired_time: 18.4847965240, output_fired_time: 15.1586990356, propagate_forward_time: 399.2823791504, propagate_backward_time: 672.4628906250, weight_update_time: 2.0955460072, other_time: 44.7728271484
epoch time: 91.5504684448s, input_fired_time: 2.6892893314, output_fired_time: 0.0000000000, propagate_forward_time: 80.9438400269, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.9173355103
validation[          , genome  5338] predictions:    6175/  10000 (61.75%), best:    6126/10000 (61.26%), error:     11695.88379, best error:     11575.02344 on epoch:    68, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 1200.6063232422s, input_fired_time: 18.9418087006, output_fired_time: 15.5401401520, propagate_forward_time: 418.2391357422, propagate_backward_time: 701.0787353516, weight_update_time: 2.1490180492, other_time: 46.8065185547
epoch time: 95.1872329712s, input_fired_time: 2.7919492722, output_fired_time: 0.0000000000, propagate_forward_time: 83.4999694824, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.8953170776
validation[          , genome  5338] predictions:    6126/  10000 (61.26%), best:    6126/10000 (61.26%), error:     11685.99707, best error:     11575.02344 on epoch:    68, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 1189.4342041016s, input_fired_time: 18.6948223114, output_fired_time: 15.4469223022, propagate_forward_time: 414.1773376465, propagate_backward_time: 693.9617309570, weight_update_time: 2.1381669044, other_time: 47.1533813477
epoch time: 93.7248077393s, input_fired_time: 2.7973756790, output_fired_time: 0.0000000000, propagate_forward_time: 82.7204208374, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.2070083618
validation[          , genome  5338] predictions:    6179/  10000 (61.79%), best:    6126/10000 (61.26%), error:     11699.43164, best error:     11575.02344 on epoch:    68, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 471.7055969238s, input_fired_time: 13.6400766373, output_fired_time: 0.0000000000, propagate_forward_time: 414.6649780273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 43.4005432129
best training[          , genome  5338] predictions:   37889/  50000 (75.78%), best:    6126/10000 (61.26%), error:     35954.32813, best error:     11575.02344 on epoch:    68, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 95.6371459961s, input_fired_time: 2.7175781727, output_fired_time: 0.0000000000, propagate_forward_time: 84.4446868896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4748840332
   testing[          , genome  5338] predictions:    6126/  10000 (61.26%), best:    6126/10000 (61.26%), error:     11575.02344, best error:     11575.02344 on epoch:    68, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
14:33:08 (45788): called boinc_finish(0)

</stderr_txt>
]]>