Nameexact_genome_1542649503_180_6313_1
Workunit4098979
Created19 Nov 2018, 23:41:59 UTC
Sent19 Nov 2018, 23:45:15 UTC
Report deadline21 Jan 2019, 8:46:22 UTC
Received16 Dec 2018, 23:36:31 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID79157
Run time15 hours 41 min 34 sec
CPU time15 hours 38 min 8 sec
Validate stateValid
Credit15,593.95
Device peak FLOPS4.43 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size829.55 MB
Peak swap size895.41 MB
Peak disk usage12.43 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6313.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6313_1_r715848063_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_180_6313.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.47534
read mu: 0.47534
read mu_delta: 0.927556
read initial_learning_rate: 0.00355545
read learning_rate: 0.00355545
read learning_rate_delta: 0.91357
read initial_weight_decay: 0.00044626
read weight_decay: 0.00044626
read weight_decay_delta: 0.942931
read batch_size: 114
read epsilon: 1e-07
read alpha: 0.104303
read input_dropout_probability: 0.00143857
read hidden_dropout_probability: 0.0846578
read velocity_reset: 1385
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6313
read normal distribution: '1 -1.35876524448395 1.9418625831604'
generator_str: '745542842'
read generator: 745542842
read generated_by_map:
1 add_edge 1
reading 95 nodes.
reading 389 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1918948896
post shuffle 1: 1050722575
epoch time: 1036.89868164063s, input_fired_time: 17.5971584320068, output_fired_time: 6.09624338150024, propagate_forward_time: 292.453430175781, propagate_backward_time: 712.087646484375, weight_update_time: 0.18905571103096, other_time: 8.6641845703125
epoch time: 61.4102630615234s, input_fired_time: 1.04369187355042, output_fired_time: 0, propagate_forward_time: 58.876651763916, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.48991775512695
validation[          , genome  6313] predictions:    5581/  10000 (55.81%), best:    5581/10000 (55.81%), error:     12923.41895, best error:     12923.41895 on epoch:     0, epoch:    0/50, mu: 0.4753397107, learning_rate: 0.0035554452, weight_decay: 0.0004462599

epoch time: 1037.2445068359s, input_fired_time: 17.8118629456, output_fired_time: 6.1000542641, propagate_forward_time: 292.5581970215, propagate_backward_time: 711.9135131836, weight_update_time: 0.1884977967, other_time: 8.8609008789
epoch time: 60.9314193726s, input_fired_time: 1.0614663363, output_fired_time: 0.0000000000, propagate_forward_time: 58.5032081604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3667449951
validation[          , genome  6313] predictions:    5737/  10000 (57.37%), best:    5737/10000 (57.37%), error:     12322.58398, best error:     12322.58398 on epoch:     1, epoch:    1/50, mu: 0.5126237869, learning_rate: 0.0032481474, weight_decay: 0.0004207921

epoch time: 1036.1151123047s, input_fired_time: 17.8369846344, output_fired_time: 6.0863714218, propagate_forward_time: 292.1578979492, propagate_backward_time: 711.4160766602, weight_update_time: 0.1879158616, other_time: 8.6177978516
epoch time: 60.9380645752s, input_fired_time: 1.0752091408, output_fired_time: 0.0000000000, propagate_forward_time: 58.4978599548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3649940491
validation[          , genome  6313] predictions:    5874/  10000 (58.74%), best:    5874/10000 (58.74%), error:     11906.26367, best error:     11906.26367 on epoch:     2, epoch:    2/50, mu: 0.5472068787, learning_rate: 0.0029674093, weight_decay: 0.0003967778

epoch time: 1038.8132324219s, input_fired_time: 17.9404659271, output_fired_time: 6.1442618370, propagate_forward_time: 293.0678405762, propagate_backward_time: 713.0305786133, weight_update_time: 0.1883877367, other_time: 8.6300659180
epoch time: 61.1415061951s, input_fired_time: 1.0700966120, output_fired_time: 0.0000000000, propagate_forward_time: 58.7091255188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3622856140
validation[          , genome  6313] predictions:    5776/  10000 (57.76%), best:    5874/10000 (58.74%), error:     12265.23730, best error:     11906.26367 on epoch:     2, epoch:    3/50, mu: 0.5792846084, learning_rate: 0.0027109357, weight_decay: 0.0003741339

epoch time: 1038.2882080078s, input_fired_time: 17.9074363708, output_fired_time: 6.1093544960, propagate_forward_time: 292.8041076660, propagate_backward_time: 712.8245239258, weight_update_time: 0.1908470392, other_time: 8.6427612305
epoch time: 60.9789428711s, input_fired_time: 1.0614056587, output_fired_time: 0.0000000000, propagate_forward_time: 58.5470199585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3705177307
validation[          , genome  6313] predictions:    5624/  10000 (56.24%), best:    5874/10000 (58.74%), error:     12429.08691, best error:     11906.26367 on epoch:     2, epoch:    4/50, mu: 0.6090385318, learning_rate: 0.0024766289, weight_decay: 0.0003527823

epoch time: 1040.6892089844s, input_fired_time: 18.0030193329, output_fired_time: 6.1030025482, propagate_forward_time: 293.4722595215, propagate_backward_time: 714.3866577148, weight_update_time: 0.1900615394, other_time: 8.7243041992
epoch time: 61.1986465454s, input_fired_time: 1.0866049528, output_fired_time: 0.0000000000, propagate_forward_time: 58.7327194214, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3793220520
validation[          , genome  6313] predictions:    6016/  10000 (60.16%), best:    6016/10000 (60.16%), error:     11469.48535, best error:     11469.48535 on epoch:     5, epoch:    5/50, mu: 0.6366369128, learning_rate: 0.0022625735, weight_decay: 0.0003326492

epoch time: 1038.6005859375s, input_fired_time: 17.9670581818, output_fired_time: 6.1103105545, propagate_forward_time: 293.0581665039, propagate_backward_time: 712.7872924805, weight_update_time: 0.1880907714, other_time: 8.6777954102
epoch time: 60.9317855835s, input_fired_time: 1.0817762613, output_fired_time: 0.0000000000, propagate_forward_time: 58.4626312256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3873786926
validation[          , genome  6313] predictions:    5916/  10000 (59.16%), best:    6016/10000 (60.16%), error:     11730.04980, best error:     11469.48535 on epoch:     5, epoch:    6/50, mu: 0.6622359753, learning_rate: 0.0020670188, weight_decay: 0.0003136651

epoch time: 1038.3863525391s, input_fired_time: 17.9484710693, output_fired_time: 6.1069116592, propagate_forward_time: 292.9812622070, propagate_backward_time: 712.7145385742, weight_update_time: 0.1881472915, other_time: 8.6351318359
epoch time: 61.0316772461s, input_fired_time: 1.0778365135, output_fired_time: 0.0000000000, propagate_forward_time: 58.5907402039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3631019592
validation[          , genome  6313] predictions:    6035/  10000 (60.35%), best:    6035/10000 (60.35%), error:     11420.69434, best error:     11420.69434 on epoch:     7, epoch:    7/50, mu: 0.6859805584, learning_rate: 0.0018883660, weight_decay: 0.0002957644

epoch time: 1038.6209716797s, input_fired_time: 17.9844360352, output_fired_time: 6.0953488350, propagate_forward_time: 292.8623046875, propagate_backward_time: 713.0233154297, weight_update_time: 0.1888536215, other_time: 8.6555786133
epoch time: 60.9615020752s, input_fired_time: 1.0819624662, output_fired_time: 0.0000000000, propagate_forward_time: 58.5003852844, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3791542053
validation[          , genome  6313] predictions:    6030/  10000 (60.30%), best:    6035/10000 (60.35%), error:     11683.97266, best error:     11420.69434 on epoch:     7, epoch:    8/50, mu: 0.7080049515, learning_rate: 0.0017251541, weight_decay: 0.0002788853

epoch time: 1038.1496582031s, input_fired_time: 17.9806766510, output_fired_time: 6.0997681618, propagate_forward_time: 292.5929565430, propagate_backward_time: 712.8497924805, weight_update_time: 0.1885678470, other_time: 8.6264648438
epoch time: 60.9195480347s, input_fired_time: 1.0755501986, output_fired_time: 0.0000000000, propagate_forward_time: 58.4820823669, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3619155884
validation[          , genome  6313] predictions:    6110/  10000 (61.10%), best:    6110/10000 (61.10%), error:     11310.77148, best error:     11310.77148 on epoch:     9, epoch:    9/50, mu: 0.7284338474, learning_rate: 0.0015760487, weight_decay: 0.0002629695

epoch time: 1037.3161621094s, input_fired_time: 17.9608306885, output_fired_time: 6.0985450745, propagate_forward_time: 292.3716125488, propagate_backward_time: 712.2443847656, weight_update_time: 0.1887900531, other_time: 8.6407470703
epoch time: 61.0451736450s, input_fired_time: 1.0880749226, output_fired_time: 0.0000000000, propagate_forward_time: 58.5915603638, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3655395508
validation[          , genome  6313] predictions:    6158/  10000 (61.58%), best:    6158/10000 (61.58%), error:     11159.27051, best error:     11159.27051 on epoch:    10, epoch:   10/50, mu: 0.7473827600, learning_rate: 0.0014398305, weight_decay: 0.0002479620

epoch time: 1037.4720458984s, input_fired_time: 18.0149059296, output_fired_time: 6.1167578697, propagate_forward_time: 292.5260314941, propagate_backward_time: 712.2062377930, weight_update_time: 0.1875282526, other_time: 8.6080932617
epoch time: 61.0657196045s, input_fired_time: 1.0909703970, output_fired_time: 0.0000000000, propagate_forward_time: 58.5911903381, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3835601807
validation[          , genome  6313] predictions:    6163/  10000 (61.63%), best:    6163/10000 (61.63%), error:     10998.40723, best error:     10998.40723 on epoch:    11, epoch:   11/50, mu: 0.7649589181, learning_rate: 0.0013153857, weight_decay: 0.0002338109

epoch time: 1036.5267333984s, input_fired_time: 17.9921913147, output_fired_time: 6.0986890793, propagate_forward_time: 292.3139343262, propagate_backward_time: 711.2678222656, weight_update_time: 0.1888326854, other_time: 8.8541259766
epoch time: 61.1021690369s, input_fired_time: 1.0918862820, output_fired_time: 0.0000000000, propagate_forward_time: 58.4993286133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5109558105
validation[          , genome  6313] predictions:    6241/  10000 (62.41%), best:    6163/10000 (61.63%), error:     10998.71875, best error:     10998.40723 on epoch:    11, epoch:   12/50, mu: 0.7812618017, learning_rate: 0.0012016967, weight_decay: 0.0002204675

epoch time: 1036.6241455078s, input_fired_time: 18.0055847168, output_fired_time: 6.0985503197, propagate_forward_time: 292.2808532715, propagate_backward_time: 711.3563232422, weight_update_time: 0.1883801520, other_time: 8.8828125000
epoch time: 61.0599555969s, input_fired_time: 1.0910614729, output_fired_time: 0.0000000000, propagate_forward_time: 58.6047859192, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3641090393
validation[          , genome  6313] predictions:    6190/  10000 (61.90%), best:    6163/10000 (61.63%), error:     11030.52539, best error:     10998.40723 on epoch:    11, epoch:   13/50, mu: 0.7963836193, learning_rate: 0.0010978339, weight_decay: 0.0002078855

epoch time: 1037.2252197266s, input_fired_time: 18.0282936096, output_fired_time: 6.0986247063, propagate_forward_time: 292.7443237305, propagate_backward_time: 711.6312866211, weight_update_time: 0.1889072657, other_time: 8.7226562500
epoch time: 61.0677413940s, input_fired_time: 1.1061053276, output_fired_time: 0.0000000000, propagate_forward_time: 58.5888671875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3727684021
validation[          , genome  6313] predictions:    6281/  10000 (62.81%), best:    6281/10000 (62.81%), error:     10754.96973, best error:     10754.96973 on epoch:    14, epoch:   14/50, mu: 0.8104099631, learning_rate: 0.0010029479, weight_decay: 0.0001960216

epoch time: 1036.5344238281s, input_fired_time: 18.0302772522, output_fired_time: 6.0935254097, propagate_forward_time: 292.3423461914, propagate_backward_time: 711.4519653320, weight_update_time: 0.1875659376, other_time: 8.6163330078
epoch time: 60.9792518616s, input_fired_time: 1.1014896631, output_fired_time: 0.0000000000, propagate_forward_time: 58.5146408081, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3631210327
validation[          , genome  6313] predictions:    6369/  10000 (63.69%), best:    6369/10000 (63.69%), error:     10462.20410, best error:     10462.20410 on epoch:    15, epoch:   15/50, mu: 0.8234202266, learning_rate: 0.0009162629, weight_decay: 0.0001848348

epoch time: 1037.0272216797s, input_fired_time: 18.0417194366, output_fired_time: 6.0942459106, propagate_forward_time: 292.3037109375, propagate_backward_time: 711.9104003906, weight_update_time: 0.1882086098, other_time: 8.6771240234
epoch time: 61.0297431946s, input_fired_time: 1.0959585905, output_fired_time: 0.0000000000, propagate_forward_time: 58.5676727295, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3661117554
validation[          , genome  6313] predictions:    6201/  10000 (62.01%), best:    6369/10000 (63.69%), error:     10919.05176, best error:     10462.20410 on epoch:    15, epoch:   16/50, mu: 0.8354879618, learning_rate: 0.0008370701, weight_decay: 0.0001742864

epoch time: 1037.1156005859s, input_fired_time: 18.0770626068, output_fired_time: 6.0971593857, propagate_forward_time: 292.4991455078, propagate_backward_time: 711.8039550781, weight_update_time: 0.1887204051, other_time: 8.6382446289
epoch time: 61.0200653076s, input_fired_time: 1.1010828018, output_fired_time: 0.0000000000, propagate_forward_time: 58.5431137085, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3758697510
validation[          , genome  6313] predictions:    6336/  10000 (63.36%), best:    6369/10000 (63.69%), error:     10635.90723, best error:     10462.20410 on epoch:    15, epoch:   17/50, mu: 0.8466814160, learning_rate: 0.0007647220, weight_decay: 0.0001643399

epoch time: 1038.2661132813s, input_fired_time: 18.1191959381, output_fired_time: 6.1086177826, propagate_forward_time: 292.8753356934, propagate_backward_time: 712.5502319336, weight_update_time: 0.1884124726, other_time: 8.6127319336
epoch time: 61.1276626587s, input_fired_time: 1.1030486822, output_fired_time: 0.0000000000, propagate_forward_time: 58.6484031677, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3762092590
validation[          , genome  6313] predictions:    6156/  10000 (61.56%), best:    6369/10000 (63.69%), error:     11013.37988, best error:     10462.20410 on epoch:    15, epoch:   18/50, mu: 0.8570640087, learning_rate: 0.0006986269, weight_decay: 0.0001549612

epoch time: 1036.6356201172s, input_fired_time: 18.0929641724, output_fired_time: 6.0881805420, propagate_forward_time: 292.2267150879, propagate_backward_time: 711.6054687500, weight_update_time: 0.1873723119, other_time: 8.6223144531
epoch time: 61.0069427490s, input_fired_time: 1.1040667295, output_fired_time: 0.0000000000, propagate_forward_time: 58.5365638733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3663139343
validation[          , genome  6313] predictions:    6363/  10000 (63.63%), best:    6369/10000 (63.69%), error:     10465.92578, best error:     10462.20410 on epoch:    15, epoch:   19/50, mu: 0.8666944504, learning_rate: 0.0006382444, weight_decay: 0.0001461176

epoch time: 1038.1149902344s, input_fired_time: 18.0970115662, output_fired_time: 6.1042962074, propagate_forward_time: 292.6292419434, propagate_backward_time: 712.7092285156, weight_update_time: 0.1875524968, other_time: 8.5751953125
epoch time: 60.9583320618s, input_fired_time: 1.1052117348, output_fired_time: 0.0000000000, propagate_forward_time: 58.4893608093, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3637580872
validation[          , genome  6313] predictions:    6345/  10000 (63.45%), best:    6369/10000 (63.69%), error:     10546.52051, best error:     10462.20410 on epoch:    15, epoch:   20/50, mu: 0.8756272197, learning_rate: 0.0005830809, weight_decay: 0.0001377788

epoch time: 1037.7589111328s, input_fired_time: 18.1307315826, output_fired_time: 6.1006140709, propagate_forward_time: 292.6690063477, propagate_backward_time: 712.2561035156, weight_update_time: 0.1880494803, other_time: 8.6024780273
epoch time: 61.0006866455s, input_fired_time: 1.1071645021, output_fired_time: 0.0000000000, propagate_forward_time: 58.5282287598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3652915955
validation[          , genome  6313] predictions:    6325/  10000 (63.25%), best:    6369/10000 (63.69%), error:     10647.51758, best error:     10462.20410 on epoch:    15, epoch:   21/50, mu: 0.8839128613, learning_rate: 0.0005326851, weight_decay: 0.0001299158

epoch time: 1037.1206054688s, input_fired_time: 18.1210575104, output_fired_time: 6.0926094055, propagate_forward_time: 292.3859252930, propagate_backward_time: 711.8389892578, weight_update_time: 0.1885053664, other_time: 8.6820678711
epoch time: 61.0247421265s, input_fired_time: 1.1117231846, output_fired_time: 0.0000000000, propagate_forward_time: 58.5421371460, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3708801270
validation[          , genome  6313] predictions:    6390/  10000 (63.90%), best:    6390/10000 (63.90%), error:     10459.97754, best error:     10459.97754 on epoch:    22, epoch:   22/50, mu: 0.8915982246, learning_rate: 0.0004866450, weight_decay: 0.0001225016

epoch time: 1040.5576171875s, input_fired_time: 18.1646251678, output_fired_time: 6.0987353325, propagate_forward_time: 293.2140808105, propagate_backward_time: 714.2490844727, weight_update_time: 0.1888056397, other_time: 8.8311157227
epoch time: 61.1954612732s, input_fired_time: 1.1263761520, output_fired_time: 0.0000000000, propagate_forward_time: 58.6432571411, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4258270264
validation[          , genome  6313] predictions:    6268/  10000 (62.68%), best:    6390/10000 (63.90%), error:     10769.96289, best error:     10459.97754 on epoch:    22, epoch:   23/50, mu: 0.8987268806, learning_rate: 0.0004445842, weight_decay: 0.0001155105

epoch time: 1038.0504150391s, input_fired_time: 18.1553287506, output_fired_time: 6.0900626183, propagate_forward_time: 292.9655761719, propagate_backward_time: 712.1049804688, weight_update_time: 0.1885175258, other_time: 8.7344360352
epoch time: 61.0644416809s, input_fired_time: 1.1210225821, output_fired_time: 0.0000000000, propagate_forward_time: 58.5646858215, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3787345886
validation[          , genome  6313] predictions:    6381/  10000 (63.81%), best:    6390/10000 (63.90%), error:     10469.70117, best error:     10459.97754 on epoch:    22, epoch:   24/50, mu: 0.9053390622, learning_rate: 0.0004061587, weight_decay: 0.0001089184

epoch time: 1037.0135498047s, input_fired_time: 18.1393413544, output_fired_time: 6.0841712952, propagate_forward_time: 292.5054016113, propagate_backward_time: 711.5537719727, weight_update_time: 0.1896777004, other_time: 8.7308959961
epoch time: 60.9940376282s, input_fired_time: 1.1147871017, output_fired_time: 0.0000000000, propagate_forward_time: 58.5128250122, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3664245605
validation[          , genome  6313] predictions:    6425/  10000 (64.25%), best:    6425/10000 (64.25%), error:     10390.50293, best error:     10390.50293 on epoch:    25, epoch:   25/50, mu: 0.9114722610, learning_rate: 0.0003710543, weight_decay: 0.0001027025

epoch time: 1036.4600830078s, input_fired_time: 18.1434097290, output_fired_time: 6.1091198921, propagate_forward_time: 292.2367553711, propagate_backward_time: 711.3231811523, weight_update_time: 0.1875775456, other_time: 8.6475830078
epoch time: 60.9153823853s, input_fired_time: 1.1144433022, output_fired_time: 0.0000000000, propagate_forward_time: 58.4354553223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3654823303
validation[          , genome  6313] predictions:    6344/  10000 (63.44%), best:    6425/10000 (64.25%), error:     10440.09375, best error:     10390.50293 on epoch:    25, epoch:   26/50, mu: 0.9171611071, learning_rate: 0.0003389840, weight_decay: 0.0000968413

epoch time: 1037.1143798828s, input_fired_time: 18.1525993347, output_fired_time: 6.0896530151, propagate_forward_time: 292.4441528320, propagate_backward_time: 711.5783081055, weight_update_time: 0.1889898032, other_time: 8.8496704102
epoch time: 61.0755233765s, input_fired_time: 1.1263740063, output_fired_time: 0.0000000000, propagate_forward_time: 58.5706558228, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3784942627
validation[          , genome  6313] predictions:    6376/  10000 (63.76%), best:    6425/10000 (64.25%), error:     10528.60742, best error:     10390.50293 on epoch:    25, epoch:   27/50, mu: 0.9224378467, learning_rate: 0.0003096856, weight_decay: 0.0000913146

epoch time: 1037.1435546875s, input_fired_time: 18.1871013641, output_fired_time: 6.0914583206, propagate_forward_time: 292.5422058105, propagate_backward_time: 711.6470336914, weight_update_time: 0.1884295344, other_time: 8.6758422852
epoch time: 60.9814796448s, input_fired_time: 1.1270655394, output_fired_time: 0.0000000000, propagate_forward_time: 58.4895248413, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3648910522
validation[          , genome  6313] predictions:    6519/  10000 (65.19%), best:    6519/10000 (65.19%), error:     10154.14063, best error:     10154.14063 on epoch:    28, epoch:   28/50, mu: 0.9273323417, learning_rate: 0.0002829194, weight_decay: 0.0000861034

epoch time: 1038.1202392578s, input_fired_time: 18.1923294067, output_fired_time: 6.1096906662, propagate_forward_time: 292.7055053711, propagate_backward_time: 712.4666748047, weight_update_time: 0.1884041131, other_time: 8.6460571289
epoch time: 61.0809097290s, input_fired_time: 1.1259418726, output_fired_time: 0.0000000000, propagate_forward_time: 58.5822105408, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3727569580
validation[          , genome  6313] predictions:    6423/  10000 (64.23%), best:    6519/10000 (65.19%), error:     10287.33887, best error:     10154.14063 on epoch:    28, epoch:   29/50, mu: 0.9318722486, learning_rate: 0.0002584666, weight_decay: 0.0000811895

epoch time: 1038.6358642578s, input_fired_time: 18.1993980408, output_fired_time: 6.0962305069, propagate_forward_time: 292.7194824219, propagate_backward_time: 712.9430541992, weight_update_time: 0.1875412911, other_time: 8.6776733398
epoch time: 61.1534690857s, input_fired_time: 1.1284410954, output_fired_time: 0.0000000000, propagate_forward_time: 58.6513481140, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3736801147
validation[          , genome  6313] predictions:    6410/  10000 (64.10%), best:    6519/10000 (65.19%), error:     10314.58887, best error:     10154.14063 on epoch:    28, epoch:   30/50, mu: 0.9360832572, learning_rate: 0.0002361273, weight_decay: 0.0000765560

epoch time: 1038.6447753906s, input_fired_time: 18.2178993225, output_fired_time: 6.0937929153, propagate_forward_time: 292.6333007813, propagate_backward_time: 712.6615600586, weight_update_time: 0.1881503016, other_time: 9.0382080078
epoch time: 61.2684326172s, input_fired_time: 1.1286509037, output_fired_time: 0.0000000000, propagate_forward_time: 58.6651306152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4746513367
validation[          , genome  6313] predictions:    6483/  10000 (64.83%), best:    6483/10000 (64.83%), error:     10126.56738, best error:     10126.56738 on epoch:    31, epoch:   31/50, mu: 0.9399892092, learning_rate: 0.0002157188, weight_decay: 0.0000721870

epoch time: 1037.2795410156s, input_fired_time: 18.1915225983, output_fired_time: 6.0979771614, propagate_forward_time: 292.3185424805, propagate_backward_time: 711.6174316406, weight_update_time: 0.1885517687, other_time: 9.0540771484
epoch time: 61.2508049011s, input_fired_time: 1.1348320246, output_fired_time: 0.0000000000, propagate_forward_time: 58.6446418762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4713325500
validation[          , genome  6313] predictions:    6521/  10000 (65.21%), best:    6521/10000 (65.21%), error:     10019.04883, best error:     10019.04883 on epoch:    32, epoch:   32/50, mu: 0.9436122179, learning_rate: 0.0001970742, weight_decay: 0.0000680674

epoch time: 1037.6918945313s, input_fired_time: 18.2148513794, output_fired_time: 6.0864195824, propagate_forward_time: 292.3996582031, propagate_backward_time: 712.1624145508, weight_update_time: 0.1888305694, other_time: 8.8285522461
epoch time: 60.9430427551s, input_fired_time: 1.1304403543, output_fired_time: 0.0000000000, propagate_forward_time: 58.4418869019, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3707160950
validation[          , genome  6313] predictions:    6491/  10000 (64.91%), best:    6521/10000 (65.21%), error:     10129.86816, best error:     10019.04883 on epoch:    32, epoch:   33/50, mu: 0.9469727278, learning_rate: 0.0001800410, weight_decay: 0.0000641828

epoch time: 1037.3903808594s, input_fired_time: 18.2479610443, output_fired_time: 6.0911164284, propagate_forward_time: 292.4787597656, propagate_backward_time: 711.8926391602, weight_update_time: 0.1887892038, other_time: 8.6798706055
epoch time: 60.9749870300s, input_fired_time: 1.1419152021, output_fired_time: 0.0000000000, propagate_forward_time: 58.4667167664, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3663558960
validation[          , genome  6313] predictions:    6419/  10000 (64.19%), best:    6521/10000 (65.21%), error:     10170.34277, best error:     10019.04883 on epoch:    32, epoch:   34/50, mu: 0.9500898123, learning_rate: 0.0001644800, weight_decay: 0.0000605199

epoch time: 1037.0648193359s, input_fired_time: 18.2566757202, output_fired_time: 6.0946497917, propagate_forward_time: 292.3553466797, propagate_backward_time: 711.7217407227, weight_update_time: 0.1899848878, other_time: 8.6364135742
epoch time: 60.9857292175s, input_fired_time: 1.1367135048, output_fired_time: 0.0000000000, propagate_forward_time: 58.4779891968, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3710250854
validation[          , genome  6313] predictions:    6486/  10000 (64.86%), best:    6521/10000 (65.21%), error:     10124.74414, best error:     10019.04883 on epoch:    32, epoch:   35/50, mu: 0.9529810548, learning_rate: 0.0001502640, weight_decay: 0.0000570661

epoch time: 1036.7381591797s, input_fired_time: 18.2450981140, output_fired_time: 6.0946569443, propagate_forward_time: 292.3291625977, propagate_backward_time: 711.2481689453, weight_update_time: 0.1879402399, other_time: 8.8210449219
epoch time: 61.3228302002s, input_fired_time: 1.1361851692, output_fired_time: 0.0000000000, propagate_forward_time: 58.8137969971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3728485107
validation[          , genome  6313] predictions:    6519/  10000 (65.19%), best:    6521/10000 (65.21%), error:     10073.31250, best error:     10019.04883 on epoch:    32, epoch:   36/50, mu: 0.9556628466, learning_rate: 0.0001372766, weight_decay: 0.0000538094

epoch time: 1038.0197753906s, input_fired_time: 18.2508888245, output_fired_time: 6.0919160843, propagate_forward_time: 292.4035034180, propagate_backward_time: 711.8840332031, weight_update_time: 0.1895527840, other_time: 9.3894042969
epoch time: 61.1905860901s, input_fired_time: 1.1472274065, output_fired_time: 0.0000000000, propagate_forward_time: 58.6600646973, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3832931519
validation[          , genome  6313] predictions:    6546/  10000 (65.46%), best:    6546/10000 (65.46%), error:      9935.22754, best error:      9935.22754 on epoch:    37, epoch:   37/50, mu: 0.9581503868, learning_rate: 0.0001254118, weight_decay: 0.0000507385

epoch time: 1038.3175048828s, input_fired_time: 18.2801494598, output_fired_time: 6.0887045860, propagate_forward_time: 292.8335876465, propagate_backward_time: 712.1782836914, weight_update_time: 0.1878983974, other_time: 8.9368286133
epoch time: 61.1577529907s, input_fired_time: 1.1400779486, output_fired_time: 0.0000000000, propagate_forward_time: 58.6448783875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3727951050
validation[          , genome  6313] predictions:    6414/  10000 (64.14%), best:    6546/10000 (65.46%), error:     10192.05859, best error:      9935.22754 on epoch:    37, epoch:   38/50, mu: 0.9604576826, learning_rate: 0.0001145724, weight_decay: 0.0000478429

epoch time: 1037.4047851563s, input_fired_time: 18.2597103119, output_fired_time: 6.0942926407, propagate_forward_time: 292.4800109863, propagate_backward_time: 711.6226196289, weight_update_time: 0.1888648421, other_time: 8.9481811523
epoch time: 61.0047607422s, input_fired_time: 1.1435073614, output_fired_time: 0.0000000000, propagate_forward_time: 58.4939422607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3673095703
validation[          , genome  6313] predictions:    6573/  10000 (65.73%), best:    6546/10000 (65.46%), error:      9951.16699, best error:      9935.22754 on epoch:    37, epoch:   39/50, mu: 0.9625978470, learning_rate: 0.0001046699, weight_decay: 0.0000451125

epoch time: 1038.4550781250s, input_fired_time: 18.3001251221, output_fired_time: 6.0991053581, propagate_forward_time: 292.9011230469, propagate_backward_time: 712.1819458008, weight_update_time: 0.1892594546, other_time: 8.9727783203
epoch time: 61.0314292908s, input_fired_time: 1.1426175833, output_fired_time: 0.0000000000, propagate_forward_time: 58.4769630432, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4118499756
validation[          , genome  6313] predictions:    6539/  10000 (65.39%), best:    6546/10000 (65.46%), error:      9996.26172, best error:      9935.22754 on epoch:    37, epoch:   40/50, mu: 0.9645829797, learning_rate: 0.0000956233, weight_decay: 0.0000425380

epoch time: 1036.5402832031s, input_fired_time: 18.2342014313, output_fired_time: 6.1009459496, propagate_forward_time: 292.0188598633, propagate_backward_time: 710.9888305664, weight_update_time: 0.1876562685, other_time: 9.1974487305
epoch time: 61.2487335205s, input_fired_time: 1.1498910189, output_fired_time: 0.0000000000, propagate_forward_time: 58.6124877930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4863548279
validation[          , genome  6313] predictions:    6551/  10000 (65.51%), best:    6551/10000 (65.51%), error:      9918.90332, best error:      9918.90332 on epoch:    41, epoch:   41/50, mu: 0.9664242864, learning_rate: 0.0000873585, weight_decay: 0.0000401103

epoch time: 1036.5592041016s, input_fired_time: 18.2667083740, output_fired_time: 6.0856647491, propagate_forward_time: 292.2080078125, propagate_backward_time: 711.1091308594, weight_update_time: 0.1889024526, other_time: 8.8896484375
epoch time: 61.0420074463s, input_fired_time: 1.1479039192, output_fired_time: 0.0000000000, propagate_forward_time: 58.4922370911, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4018669128
validation[          , genome  6313] predictions:    6590/  10000 (65.90%), best:    6590/10000 (65.90%), error:      9857.50586, best error:      9857.50586 on epoch:    42, epoch:   42/50, mu: 0.9681321979, learning_rate: 0.0000798081, weight_decay: 0.0000378213

epoch time: 1037.2023925781s, input_fired_time: 18.2750625610, output_fired_time: 6.0888247490, propagate_forward_time: 292.2671813965, propagate_backward_time: 711.7340698242, weight_update_time: 0.1880177706, other_time: 8.8372192383
epoch time: 60.9219017029s, input_fired_time: 1.1440967321, output_fired_time: 0.0000000000, propagate_forward_time: 58.3787765503, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3990287781
validation[          , genome  6313] predictions:    6549/  10000 (65.49%), best:    6590/10000 (65.90%), error:      9882.06836, best error:      9857.50586 on epoch:    42, epoch:   43/50, mu: 0.9697163701, learning_rate: 0.0000729103, weight_decay: 0.0000356628

epoch time: 1036.6064453125s, input_fired_time: 18.2703895569, output_fired_time: 6.0979037285, propagate_forward_time: 292.0989685059, propagate_backward_time: 711.2949218750, weight_update_time: 0.1877786368, other_time: 8.8442382813
epoch time: 61.1231575012s, input_fired_time: 1.1533125639, output_fired_time: 0.0000000000, propagate_forward_time: 58.5583229065, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4115219116
validation[          , genome  6313] predictions:    6607/  10000 (66.07%), best:    6607/10000 (66.07%), error:      9800.95703, best error:      9800.95703 on epoch:    44, epoch:   44/50, mu: 0.9711858034, learning_rate: 0.0000666086, weight_decay: 0.0000336276

epoch time: 1037.5576171875s, input_fired_time: 18.3068180084, output_fired_time: 6.1030764580, propagate_forward_time: 292.4013366699, propagate_backward_time: 711.9259033203, weight_update_time: 0.1884561926, other_time: 8.8204345703
epoch time: 61.0455818176s, input_fired_time: 1.1490249634, output_fired_time: 0.0000000000, propagate_forward_time: 58.4995689392, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3969879150
validation[          , genome  6313] predictions:    6571/  10000 (65.71%), best:    6607/10000 (66.07%), error:      9859.67188, best error:      9800.95703 on epoch:    44, epoch:   45/50, mu: 0.9725487828, learning_rate: 0.0000608516, weight_decay: 0.0000317085

epoch time: 1037.5638427734s, input_fired_time: 18.2818870544, output_fired_time: 6.0997796059, propagate_forward_time: 292.2995300293, propagate_backward_time: 711.8499145508, weight_update_time: 0.1879393160, other_time: 9.0327758789
epoch time: 61.1770782471s, input_fired_time: 1.1467204094, output_fired_time: 0.0000000000, propagate_forward_time: 58.5405349731, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4898223877
validation[          , genome  6313] predictions:    6504/  10000 (65.04%), best:    6607/10000 (66.07%), error:     10009.86035, best error:      9800.95703 on epoch:    44, epoch:   46/50, mu: 0.9738129973, learning_rate: 0.0000555922, weight_decay: 0.0000298989

epoch time: 1037.4239501953s, input_fired_time: 18.2837562561, output_fired_time: 6.0991868973, propagate_forward_time: 292.3255004883, propagate_backward_time: 711.5335693359, weight_update_time: 0.1878591031, other_time: 9.1819458008
epoch time: 61.1554374695s, input_fired_time: 1.1515765190, output_fired_time: 0.0000000000, propagate_forward_time: 58.4940071106, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5098533630
validation[          , genome  6313] predictions:    6644/  10000 (66.44%), best:    6644/10000 (66.44%), error:      9753.92773, best error:      9753.92773 on epoch:    47, epoch:   47/50, mu: 0.9749856591, learning_rate: 0.0000507874, weight_decay: 0.0000281926

epoch time: 1036.4638671875s, input_fired_time: 18.2825737000, output_fired_time: 6.0873889923, propagate_forward_time: 292.1294250488, propagate_backward_time: 710.9076538086, weight_update_time: 0.1879199594, other_time: 9.0568237305
epoch time: 61.3837814331s, input_fired_time: 1.1574150324, output_fired_time: 0.0000000000, propagate_forward_time: 58.8161773682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4101905823
validation[          , genome  6313] predictions:    6650/  10000 (66.50%), best:    6650/10000 (66.50%), error:      9688.90430, best error:      9688.90430 on epoch:    48, epoch:   48/50, mu: 0.9760733843, learning_rate: 0.0000463978, weight_decay: 0.0000265836

epoch time: 1037.8499755859s, input_fired_time: 18.3217449188, output_fired_time: 6.0991468430, propagate_forward_time: 292.5895080566, propagate_backward_time: 711.8740234375, weight_update_time: 0.1902169287, other_time: 8.9655761719
epoch time: 61.2756958008s, input_fired_time: 1.1595789194, output_fired_time: 0.0000000000, propagate_forward_time: 58.6306610107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4854545593
validation[          , genome  6313] predictions:    6668/  10000 (66.68%), best:    6668/10000 (66.68%), error:      9680.31641, best error:      9680.31641 on epoch:    49, epoch:   49/50, mu: 0.9770823121, learning_rate: 0.0000423876, weight_decay: 0.0000250665

epoch time: 1038.1857910156s, input_fired_time: 18.3264369965, output_fired_time: 6.0917372704, propagate_forward_time: 292.5741271973, propagate_backward_time: 711.9469604492, weight_update_time: 0.1879535764, other_time: 9.2465209961
epoch time: 61.1258811951s, input_fired_time: 1.1533247232, output_fired_time: 0.0000000000, propagate_forward_time: 58.4695816040, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5029754639
validation[          , genome  6313] predictions:    6658/  10000 (66.58%), best:    6668/10000 (66.68%), error:      9729.84863, best error:      9680.31641 on epoch:    49, epoch:   50/50, mu: 0.9780181050, learning_rate: 0.0000387241, weight_decay: 0.0000236360

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 304.3763122559s, input_fired_time: 5.7706594467, output_fired_time: 0.0000000000, propagate_forward_time: 291.5606079102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.0450439453
best training[          , genome  6313] predictions:   37235/  50000 (74.47%), best:    6668/10000 (66.68%), error:     39133.79297, best error:      9680.31641 on epoch:    49, epoch:   51/50, mu: 0.9788861275, learning_rate: 0.0000353771, weight_decay: 0.0000222871
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 61.0617065430s, input_fired_time: 1.1561824083, output_fired_time: 0.0000000000, propagate_forward_time: 58.4986457825, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4068794250
   testing[          , genome  6313] predictions:    6668/  10000 (66.68%), best:    6668/10000 (66.68%), error:      9680.31641, best error:      9680.31641 on epoch:    49, epoch:   51/50, mu: 0.9788861275, learning_rate: 0.0000353771, weight_decay: 0.0000222871
backpropagation finished successfully!
17:36:00 (7008): called boinc_finish(0)

</stderr_txt>
]]>