Nameexact_genome_1542692703_179_6390_0
Workunit4100693
Created20 Nov 2018, 9:10:58 UTC
Sent20 Nov 2018, 9:45:14 UTC
Report deadline23 Jan 2019, 14:58:51 UTC
Received17 Dec 2018, 14:49:12 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID79157
Run time16 hours 12 min 52 sec
CPU time16 hours 9 min 26 sec
Validate stateValid
Credit16,054.45
Device peak FLOPS4.43 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size572.48 MB
Peak swap size592.30 MB
Peak disk usage15.69 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_179_6390.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_179_6390_0_r898349946_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542692703_179_6390.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.501019
read mu: 0.501019
read mu_delta: 0.955775
read initial_learning_rate: 0.00253121
read learning_rate: 0.00253121
read learning_rate_delta: 0.915257
read initial_weight_decay: 0.000442962
read weight_decay: 0.000442962
read weight_decay_delta: 0.936415
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.1236
read input_dropout_probability: 0.00134577
read hidden_dropout_probability: 0.0928644
read velocity_reset: 1790
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6390
read normal distribution: '0 -1.78885757923126 0.238026544451714'
generator_str: '1028529543'
read generator: 1028529543
read generated_by_map:
1 add_node 1
reading 108 nodes.
reading 573 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1400154498
post shuffle 1: 1088502111
epoch time: 1068.30322265625s, input_fired_time: 15.771861076355, output_fired_time: 5.4768180847168, propagate_forward_time: 320.281677246094, propagate_backward_time: 718.624267578125, weight_update_time: 0.274027228355408, other_time: 8.14862060546875
epoch time: 66.7521438598633s, input_fired_time: 0.939038157463074, output_fired_time: 0, propagate_forward_time: 64.5356826782227, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.27742004394531
validation[          , genome  6390] predictions:    5550/  10000 (55.50%), best:    5550/10000 (55.50%), error:     12721.62109, best error:     12721.62109 on epoch:     0, epoch:    0/50, mu: 0.5010185838, learning_rate: 0.0025312102, weight_decay: 0.0004429620

epoch time: 1067.7019042969s, input_fired_time: 15.9123945236, output_fired_time: 5.4738383293, propagate_forward_time: 320.0906066895, propagate_backward_time: 717.8577880859, weight_update_time: 0.2721367776, other_time: 8.3673095703
epoch time: 67.1457901001s, input_fired_time: 0.9516742229, output_fired_time: 0.0000000000, propagate_forward_time: 64.9124069214, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2817077637
validation[          , genome  6390] predictions:    5863/  10000 (58.63%), best:    5863/10000 (58.63%), error:     11824.13477, best error:     11824.13477 on epoch:     1, epoch:    1/50, mu: 0.5226438046, learning_rate: 0.0023167068, weight_decay: 0.0004147963

epoch time: 1067.6427001953s, input_fired_time: 15.9454345703, output_fired_time: 5.4709639549, propagate_forward_time: 320.1442260742, propagate_backward_time: 717.7186279297, weight_update_time: 0.2737368345, other_time: 8.3634643555
epoch time: 66.5541381836s, input_fired_time: 0.9368403554, output_fired_time: 0.0000000000, propagate_forward_time: 64.3375473022, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2797470093
validation[          , genome  6390] predictions:    6005/  10000 (60.05%), best:    6005/10000 (60.05%), error:     11671.09473, best error:     11671.09473 on epoch:     2, epoch:    2/50, mu: 0.5433126688, learning_rate: 0.0021203812, weight_decay: 0.0003884216

epoch time: 1067.5452880859s, input_fired_time: 15.9415197372, output_fired_time: 5.4724097252, propagate_forward_time: 319.8443298340, propagate_backward_time: 717.9984130859, weight_update_time: 0.2735200226, other_time: 8.2885742188
epoch time: 66.6931228638s, input_fired_time: 0.9464362860, output_fired_time: 0.0000000000, propagate_forward_time: 64.4599914551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2866973877
validation[          , genome  6390] predictions:    5932/  10000 (59.32%), best:    6005/10000 (60.05%), error:     11831.25098, best error:     11671.09473 on epoch:     2, epoch:    3/50, mu: 0.5630674362, learning_rate: 0.0019406929, weight_decay: 0.0003637238

epoch time: 1067.6923828125s, input_fired_time: 15.9796600342, output_fired_time: 5.4695329666, propagate_forward_time: 319.9685668945, propagate_backward_time: 717.9789428711, weight_update_time: 0.2728137672, other_time: 8.2957763672
epoch time: 66.8760452271s, input_fired_time: 0.9592579007, output_fired_time: 0.0000000000, propagate_forward_time: 64.6299133301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2868728638
validation[          , genome  6390] predictions:    6182/  10000 (61.82%), best:    6182/10000 (61.82%), error:     10968.63184, best error:     10968.63184 on epoch:     4, epoch:    4/50, mu: 0.5819485188, learning_rate: 0.0017762320, weight_decay: 0.0003405965

epoch time: 1069.3939208984s, input_fired_time: 16.0143756866, output_fired_time: 5.4758329391, propagate_forward_time: 320.6152038574, propagate_backward_time: 718.9922485352, weight_update_time: 0.2735635042, other_time: 8.2962036133
epoch time: 66.7974014282s, input_fired_time: 0.9533087015, output_fired_time: 0.0000000000, propagate_forward_time: 64.5510559082, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2930374146
validation[          , genome  6390] predictions:    6135/  10000 (61.35%), best:    6182/10000 (61.82%), error:     11212.78906, best error:     10968.63184 on epoch:     4, epoch:    5/50, mu: 0.5999945998, learning_rate: 0.0016257081, weight_decay: 0.0003189397

epoch time: 1071.4862060547s, input_fired_time: 16.0825462341, output_fired_time: 5.4911890030, propagate_forward_time: 321.5610046387, propagate_backward_time: 720.0145263672, weight_update_time: 0.2746396065, other_time: 8.3369140625
epoch time: 66.7037429810s, input_fired_time: 0.9507381320, output_fired_time: 0.0000000000, propagate_forward_time: 64.4668045044, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2862014771
validation[          , genome  6390] predictions:    6232/  10000 (62.32%), best:    6232/10000 (62.32%), error:     10867.82715, best error:     10867.82715 on epoch:     6, epoch:    6/50, mu: 0.6172425747, learning_rate: 0.0014879401, weight_decay: 0.0002986599

epoch time: 1069.6096191406s, input_fired_time: 16.0390605927, output_fired_time: 5.4854135513, propagate_forward_time: 320.6561889648, propagate_backward_time: 719.0515747070, weight_update_time: 0.2744249403, other_time: 8.3773193359
epoch time: 66.8684005737s, input_fired_time: 0.9590878487, output_fired_time: 0.0000000000, propagate_forward_time: 64.6072387695, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3020706177
validation[          , genome  6390] predictions:    6171/  10000 (61.71%), best:    6171/10000 (61.71%), error:     10836.12402, best error:     10836.12402 on epoch:     7, epoch:    7/50, mu: 0.6337277889, learning_rate: 0.0013618470, weight_decay: 0.0002796696

epoch time: 1069.7839355469s, input_fired_time: 16.0474948883, output_fired_time: 5.4794054031, propagate_forward_time: 320.7135314941, propagate_backward_time: 719.1522827148, weight_update_time: 0.2741580009, other_time: 8.3912963867
epoch time: 66.9126205444s, input_fired_time: 0.9595760703, output_fired_time: 0.0000000000, propagate_forward_time: 64.6698379517, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2832031250
validation[          , genome  6390] predictions:    6257/  10000 (62.57%), best:    6171/10000 (61.71%), error:     10880.99609, best error:     10836.12402 on epoch:     7, epoch:    8/50, mu: 0.6494839191, learning_rate: 0.0012464395, weight_decay: 0.0002618869

epoch time: 1069.0401611328s, input_fired_time: 16.0634403229, output_fired_time: 5.4922013283, propagate_forward_time: 320.2987976074, propagate_backward_time: 718.8813476563, weight_update_time: 0.2729758918, other_time: 8.3043212891
epoch time: 67.0393905640s, input_fired_time: 0.9740895629, output_fired_time: 0.0000000000, propagate_forward_time: 64.7782516479, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2870483398
validation[          , genome  6390] predictions:    6144/  10000 (61.44%), best:    6171/10000 (61.71%), error:     11065.11621, best error:     10836.12402 on epoch:     7, epoch:    9/50, mu: 0.6645432711, learning_rate: 0.0011408121, weight_decay: 0.0002452348

epoch time: 1068.8039550781s, input_fired_time: 16.0675048828, output_fired_time: 5.4780659676, propagate_forward_time: 320.3193359375, propagate_backward_time: 718.5936889648, weight_update_time: 0.2755044103, other_time: 8.3453979492
epoch time: 66.6554107666s, input_fired_time: 0.9605239034, output_fired_time: 0.0000000000, propagate_forward_time: 64.4116821289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2832031250
validation[          , genome  6390] predictions:    6367/  10000 (63.67%), best:    6367/10000 (63.67%), error:     10581.45801, best error:     10581.45801 on epoch:    10, epoch:   10/50, mu: 0.6789366007, learning_rate: 0.0010441358, weight_decay: 0.0002296416

epoch time: 1069.9449462891s, input_fired_time: 16.0861091614, output_fired_time: 5.4811806679, propagate_forward_time: 320.7662963867, propagate_backward_time: 719.2736206055, weight_update_time: 0.2728417516, other_time: 8.3377685547
epoch time: 66.6607742310s, input_fired_time: 0.9720738530, output_fired_time: 0.0000000000, propagate_forward_time: 64.4056320190, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2830657959
validation[          , genome  6390] predictions:    6416/  10000 (64.16%), best:    6416/10000 (64.16%), error:     10382.53809, best error:     10382.53809 on epoch:    11, epoch:   11/50, mu: 0.6926934123, learning_rate: 0.0009556522, weight_decay: 0.0002150398

epoch time: 1068.8083496094s, input_fired_time: 16.0950393677, output_fired_time: 5.4743151665, propagate_forward_time: 320.3053588867, propagate_backward_time: 718.4534912109, weight_update_time: 0.2760934532, other_time: 8.4800415039
epoch time: 66.9421768188s, input_fired_time: 0.9687938094, output_fired_time: 0.0000000000, propagate_forward_time: 64.6168365479, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3565444946
validation[          , genome  6390] predictions:    6310/  10000 (63.10%), best:    6416/10000 (64.16%), error:     10539.62207, best error:     10382.53809 on epoch:    11, epoch:   12/50, mu: 0.7058417797, learning_rate: 0.0008746670, weight_decay: 0.0002013666

epoch time: 1069.2913818359s, input_fired_time: 16.0967159271, output_fired_time: 5.4821829796, propagate_forward_time: 320.4726562500, propagate_backward_time: 718.5938720703, weight_update_time: 0.2737087607, other_time: 8.6459960938
epoch time: 66.7969131470s, input_fired_time: 0.9775741100, output_fired_time: 0.0000000000, propagate_forward_time: 64.4545288086, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3648071289
validation[          , genome  6390] predictions:    6445/  10000 (64.45%), best:    6445/10000 (64.45%), error:     10246.85547, best error:     10246.85547 on epoch:    13, epoch:   13/50, mu: 0.7184087038, learning_rate: 0.0008005447, weight_decay: 0.0001885627

epoch time: 1069.1179199219s, input_fired_time: 16.0995330811, output_fired_time: 5.4812660217, propagate_forward_time: 320.2950744629, propagate_backward_time: 718.6433715820, weight_update_time: 0.2744391859, other_time: 8.5986938477
epoch time: 66.7069320679s, input_fired_time: 0.9707632065, output_fired_time: 0.0000000000, propagate_forward_time: 64.4565887451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2795791626
validation[          , genome  6390] predictions:    6391/  10000 (63.91%), best:    6445/10000 (64.45%), error:     10336.85938, best error:     10246.85547 on epoch:    13, epoch:   14/50, mu: 0.7304198146, learning_rate: 0.0007327039, weight_decay: 0.0001765729

epoch time: 1067.9058837891s, input_fired_time: 16.0809803009, output_fired_time: 5.4755439758, propagate_forward_time: 320.0538940430, propagate_backward_time: 718.0018920898, weight_update_time: 0.2732535899, other_time: 8.2935791016
epoch time: 66.7548141479s, input_fired_time: 0.9850442410, output_fired_time: 0.0000000000, propagate_forward_time: 64.4828109741, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2869567871
validation[          , genome  6390] predictions:    6363/  10000 (63.63%), best:    6445/10000 (64.45%), error:     10298.93555, best error:     10246.85547 on epoch:    13, epoch:   15/50, mu: 0.7418997288, learning_rate: 0.0006706121, weight_decay: 0.0001653456

epoch time: 1068.7242431641s, input_fired_time: 16.1288375854, output_fired_time: 5.4853334427, propagate_forward_time: 320.5141601563, propagate_backward_time: 718.3038330078, weight_update_time: 0.2743853033, other_time: 8.2921142578
epoch time: 66.8581848145s, input_fired_time: 0.9790593386, output_fired_time: 0.0000000000, propagate_forward_time: 64.5962753296, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2828521729
validation[          , genome  6390] predictions:    6180/  10000 (61.80%), best:    6445/10000 (64.45%), error:     11237.22656, best error:     10246.85547 on epoch:    13, epoch:   16/50, mu: 0.7528719902, learning_rate: 0.0006137821, weight_decay: 0.0001548321

epoch time: 1069.8403320313s, input_fired_time: 16.1491394043, output_fired_time: 5.4922819138, propagate_forward_time: 320.7457885742, propagate_backward_time: 719.1881713867, weight_update_time: 0.2750520706, other_time: 8.2648925781
epoch time: 66.6908569336s, input_fired_time: 0.9725679159, output_fired_time: 0.0000000000, propagate_forward_time: 64.4415130615, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2767791748
validation[          , genome  6390] predictions:    6356/  10000 (63.56%), best:    6445/10000 (64.45%), error:     10436.70508, best error:     10246.85547 on epoch:    13, epoch:   17/50, mu: 0.7633590102, learning_rate: 0.0005617682, weight_decay: 0.0001449871

epoch time: 1068.7797851563s, input_fired_time: 16.1421966553, output_fired_time: 5.4738721848, propagate_forward_time: 320.6807861328, propagate_backward_time: 718.1912231445, weight_update_time: 0.2730425894, other_time: 8.2916870117
epoch time: 66.7571105957s, input_fired_time: 0.9778320789, output_fired_time: 0.0000000000, propagate_forward_time: 64.4847183228, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2945632935
validation[          , genome  6390] predictions:    6345/  10000 (63.45%), best:    6445/10000 (64.45%), error:     10433.00684, best error:     10246.85547 on epoch:    13, epoch:   18/50, mu: 0.7733821869, learning_rate: 0.0005141621, weight_decay: 0.0001357681

epoch time: 1061.3804931641s, input_fired_time: 16.0218219757, output_fired_time: 5.4437131882, propagate_forward_time: 318.0383300781, propagate_backward_time: 713.6899414063, weight_update_time: 0.2713451684, other_time: 8.1866455078
epoch time: 65.9545364380s, input_fired_time: 0.9702883363, output_fired_time: 0.0000000000, propagate_forward_time: 63.7264060974, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2578392029
validation[          , genome  6390] predictions:    6511/  10000 (65.11%), best:    6511/10000 (65.11%), error:     10005.13184, best error:     10005.13184 on epoch:    19, epoch:   19/50, mu: 0.7829621434, learning_rate: 0.0004705902, weight_decay: 0.0001271353

epoch time: 1057.3572998047s, input_fired_time: 15.9724731445, output_fired_time: 5.4122381210, propagate_forward_time: 316.8930664063, propagate_backward_time: 710.9527587891, weight_update_time: 0.2711987495, other_time: 8.1267089844
epoch time: 66.0854263306s, input_fired_time: 0.9734556079, output_fired_time: 0.0000000000, propagate_forward_time: 63.8566474915, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2553215027
validation[          , genome  6390] predictions:    6519/  10000 (65.19%), best:    6511/10000 (65.11%), error:     10073.75879, best error:     10005.13184 on epoch:    19, epoch:   20/50, mu: 0.7921183705, learning_rate: 0.0004307108, weight_decay: 0.0001190514

epoch time: 1048.9500732422s, input_fired_time: 15.8544044495, output_fired_time: 5.3764514923, propagate_forward_time: 314.3631896973, propagate_backward_time: 705.3257446289, weight_update_time: 0.2672252059, other_time: 8.0303344727
epoch time: 64.6538619995s, input_fired_time: 0.9485524893, output_fired_time: 0.0000000000, propagate_forward_time: 62.4646568298, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2406539917
validation[          , genome  6390] predictions:    6493/  10000 (64.93%), best:    6511/10000 (65.11%), error:     10067.11133, best error:     10005.13184 on epoch:    19, epoch:   21/50, mu: 0.8008697033, learning_rate: 0.0003942109, weight_decay: 0.0001114815

epoch time: 1035.5998535156s, input_fired_time: 15.6427202225, output_fired_time: 5.2998919487, propagate_forward_time: 310.2604064941, propagate_backward_time: 696.4630126953, weight_update_time: 0.2633104920, other_time: 7.9338378906
epoch time: 64.6473312378s, input_fired_time: 0.9457089305, output_fired_time: 0.0000000000, propagate_forward_time: 62.4635620117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2380599976
validation[          , genome  6390] predictions:    6195/  10000 (61.95%), best:    6511/10000 (65.11%), error:     10859.57813, best error:     10005.13184 on epoch:    19, epoch:   22/50, mu: 0.8092340231, learning_rate: 0.0003608041, weight_decay: 0.0001043930

epoch time: 1038.8516845703s, input_fired_time: 15.6995906830, output_fired_time: 5.3218460083, propagate_forward_time: 311.2256164551, propagate_backward_time: 698.6140136719, weight_update_time: 0.2654778659, other_time: 7.9906005859
epoch time: 66.0888214111s, input_fired_time: 0.9751934409, output_fired_time: 0.0000000000, propagate_forward_time: 63.8628730774, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2507514954
validation[          , genome  6390] predictions:    6548/  10000 (65.48%), best:    6548/10000 (65.48%), error:      9906.09082, best error:      9906.09082 on epoch:    23, epoch:   23/50, mu: 0.8172283769, learning_rate: 0.0003302284, weight_decay: 0.0000977552

epoch time: 1054.1361083984s, input_fired_time: 15.9347295761, output_fired_time: 5.3967704773, propagate_forward_time: 315.8648071289, propagate_backward_time: 708.7968750000, weight_update_time: 0.2698282003, other_time: 8.1430053711
epoch time: 65.8043365479s, input_fired_time: 0.9793563485, output_fired_time: 0.0000000000, propagate_forward_time: 63.5745239258, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2504577637
validation[          , genome  6390] predictions:    6432/  10000 (64.32%), best:    6548/10000 (65.48%), error:     10189.27637, best error:      9906.09082 on epoch:    23, epoch:   24/50, mu: 0.8248692155, learning_rate: 0.0003022437, weight_decay: 0.0000915394

epoch time: 1055.7385253906s, input_fired_time: 15.9797668457, output_fired_time: 5.4135761261, propagate_forward_time: 316.3545227051, propagate_backward_time: 709.8167724609, weight_update_time: 0.2702293694, other_time: 8.1739501953
epoch time: 66.9683837891s, input_fired_time: 0.9897676706, output_fired_time: 0.0000000000, propagate_forward_time: 64.7032318115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2753829956
validation[          , genome  6390] predictions:    6587/  10000 (65.87%), best:    6587/10000 (65.87%), error:      9849.67383, best error:      9849.67383 on epoch:    25, epoch:   25/50, mu: 0.8321721554, learning_rate: 0.0002766306, weight_decay: 0.0000857189

epoch time: 1071.5996093750s, input_fired_time: 16.2261199951, output_fired_time: 5.4998326302, propagate_forward_time: 321.0922241211, propagate_backward_time: 720.2120971680, weight_update_time: 0.2735819817, other_time: 8.5693359375
epoch time: 66.9959716797s, input_fired_time: 0.9865096211, output_fired_time: 0.0000000000, propagate_forward_time: 64.7325057983, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2769546509
validation[          , genome  6390] predictions:    6439/  10000 (64.39%), best:    6587/10000 (65.87%), error:     10085.61426, best error:      9849.67383 on epoch:    25, epoch:   26/50, mu: 0.8391520977, learning_rate: 0.0002531880, weight_decay: 0.0000802685

epoch time: 1072.9483642578s, input_fired_time: 16.2535133362, output_fired_time: 5.5003647804, propagate_forward_time: 321.3788146973, propagate_backward_time: 721.2799072266, weight_update_time: 0.2730055749, other_time: 8.5357666016
epoch time: 66.9850540161s, input_fired_time: 0.9996322393, output_fired_time: 0.0000000000, propagate_forward_time: 64.7065200806, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2789001465
validation[          , genome  6390] predictions:    6550/  10000 (65.50%), best:    6587/10000 (65.87%), error:      9904.89551, best error:      9849.67383 on epoch:    25, epoch:   27/50, mu: 0.8458233476, learning_rate: 0.0002317320, weight_decay: 0.0000751646

epoch time: 1072.7469482422s, input_fired_time: 16.2554473877, output_fired_time: 5.5066437721, propagate_forward_time: 321.3605957031, propagate_backward_time: 721.2922363281, weight_update_time: 0.2753556669, other_time: 8.3320312500
epoch time: 66.9966888428s, input_fired_time: 0.9950393438, output_fired_time: 0.0000000000, propagate_forward_time: 64.7147750854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2868728638
validation[          , genome  6390] predictions:    6636/  10000 (66.36%), best:    6636/10000 (66.36%), error:      9707.61133, best error:      9707.61133 on epoch:    28, epoch:   28/50, mu: 0.8521995544, learning_rate: 0.0002120942, weight_decay: 0.0000703853

epoch time: 1072.4276123047s, input_fired_time: 16.2778987885, output_fired_time: 5.5007205009, propagate_forward_time: 321.5243225098, propagate_backward_time: 720.7804565430, weight_update_time: 0.2737920582, other_time: 8.3441772461
epoch time: 67.0897293091s, input_fired_time: 1.0015147924, output_fired_time: 0.0000000000, propagate_forward_time: 64.7998657227, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2883453369
validation[          , genome  6390] predictions:    6646/  10000 (66.46%), best:    6636/10000 (66.36%), error:      9715.31641, best error:      9707.61133 on epoch:    28, epoch:   29/50, mu: 0.8582937717, learning_rate: 0.0001941206, weight_decay: 0.0000659098

epoch time: 1072.1613769531s, input_fired_time: 16.2752571106, output_fired_time: 5.5193605423, propagate_forward_time: 321.3232116699, propagate_backward_time: 720.7636108398, weight_update_time: 0.2741245031, other_time: 8.2798461914
epoch time: 66.9341430664s, input_fired_time: 0.9992340803, output_fired_time: 0.0000000000, propagate_forward_time: 64.6561737061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2787322998
validation[          , genome  6390] predictions:    6704/  10000 (67.04%), best:    6704/10000 (67.04%), error:      9575.04004, best error:      9575.04004 on epoch:    30, epoch:   30/50, mu: 0.8641184568, learning_rate: 0.0001776702, weight_decay: 0.0000617189

epoch time: 1073.0625000000s, input_fired_time: 16.2806739807, output_fired_time: 5.4963631630, propagate_forward_time: 321.6588134766, propagate_backward_time: 721.2001953125, weight_update_time: 0.2737692297, other_time: 8.4265136719
epoch time: 66.8067245483s, input_fired_time: 1.0050048828, output_fired_time: 0.0000000000, propagate_forward_time: 64.5240631104, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2776565552
validation[          , genome  6390] predictions:    6650/  10000 (66.50%), best:    6704/10000 (67.04%), error:      9770.86035, best error:      9575.04004 on epoch:    30, epoch:   31/50, mu: 0.8696855903, learning_rate: 0.0001626138, weight_decay: 0.0000577946

epoch time: 1070.4465332031s, input_fired_time: 16.2779579163, output_fired_time: 5.4948124886, propagate_forward_time: 320.8655395508, propagate_backward_time: 719.3887939453, weight_update_time: 0.2739958465, other_time: 8.4193725586
epoch time: 67.0903396606s, input_fired_time: 1.0102089643, output_fired_time: 0.0000000000, propagate_forward_time: 64.7543029785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3258285522
validation[          , genome  6390] predictions:    6577/  10000 (65.77%), best:    6704/10000 (67.04%), error:      9934.87207, best error:      9575.04004 on epoch:    30, epoch:   32/50, mu: 0.8750064969, learning_rate: 0.0001488334, weight_decay: 0.0000541197

epoch time: 1071.9243164063s, input_fired_time: 16.3065338135, output_fired_time: 5.4967479706, propagate_forward_time: 321.2634582520, propagate_backward_time: 720.5174560547, weight_update_time: 0.2740001976, other_time: 8.3402099609
epoch time: 67.0204162598s, input_fired_time: 1.0008543730, output_fired_time: 0.0000000000, propagate_forward_time: 64.7419815063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2775802612
validation[          , genome  6390] predictions:    6643/  10000 (66.43%), best:    6704/10000 (67.04%), error:      9745.08203, best error:      9575.04004 on epoch:    30, epoch:   33/50, mu: 0.8800920844, learning_rate: 0.0001362207, weight_decay: 0.0000506785

epoch time: 1075.5013427734s, input_fired_time: 16.3322582245, output_fired_time: 5.5013775826, propagate_forward_time: 323.4206848145, propagate_backward_time: 721.9210205078, weight_update_time: 0.2739591002, other_time: 8.3260498047
epoch time: 67.0333557129s, input_fired_time: 1.0074375868, output_fired_time: 0.0000000000, propagate_forward_time: 64.7581481934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2677688599
validation[          , genome  6390] predictions:    6742/  10000 (67.42%), best:    6742/10000 (67.42%), error:      9506.52051, best error:      9506.52051 on epoch:    34, epoch:   34/50, mu: 0.8849527836, learning_rate: 0.0001246769, weight_decay: 0.0000474561

epoch time: 1072.3795166016s, input_fired_time: 16.2972812653, output_fired_time: 5.5016031265, propagate_forward_time: 321.3730468750, propagate_backward_time: 720.8052368164, weight_update_time: 0.2751047909, other_time: 8.4024047852
epoch time: 67.0535278320s, input_fired_time: 1.0106446743, output_fired_time: 0.0000000000, propagate_forward_time: 64.7685317993, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2743530273
validation[          , genome  6390] predictions:    6716/  10000 (67.16%), best:    6742/10000 (67.42%), error:      9510.06250, best error:      9506.52051 on epoch:    34, epoch:   35/50, mu: 0.8895984888, learning_rate: 0.0001141114, weight_decay: 0.0000444386

epoch time: 1072.6181640625s, input_fired_time: 16.2954463959, output_fired_time: 5.4982857704, propagate_forward_time: 321.6538085938, propagate_backward_time: 720.8505859375, weight_update_time: 0.2729111314, other_time: 8.3200683594
epoch time: 67.1236648560s, input_fired_time: 1.0270948410, output_fired_time: 0.0000000000, propagate_forward_time: 64.8100891113, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2864837646
validation[          , genome  6390] predictions:    6724/  10000 (67.24%), best:    6724/10000 (67.24%), error:      9504.45605, best error:      9504.45605 on epoch:    36, epoch:   36/50, mu: 0.8940387368, learning_rate: 0.0001044412, weight_decay: 0.0000416130

epoch time: 1072.1151123047s, input_fired_time: 16.3042545319, output_fired_time: 5.5097150803, propagate_forward_time: 321.2832031250, propagate_backward_time: 720.7203979492, weight_update_time: 0.2747690976, other_time: 8.2975463867
epoch time: 67.0286865234s, input_fired_time: 1.0126513243, output_fired_time: 0.0000000000, propagate_forward_time: 64.7427291870, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2733078003
validation[          , genome  6390] predictions:    6699/  10000 (66.99%), best:    6724/10000 (67.24%), error:      9558.66699, best error:      9504.45605 on epoch:    36, epoch:   37/50, mu: 0.8982826471, learning_rate: 0.0000955905, weight_decay: 0.0000389670

epoch time: 1068.7580566406s, input_fired_time: 16.2530555725, output_fired_time: 5.4858803749, propagate_forward_time: 320.4476623535, propagate_backward_time: 718.2410278320, weight_update_time: 0.2748591304, other_time: 8.3305053711
epoch time: 66.7155075073s, input_fired_time: 1.0033096075, output_fired_time: 0.0000000000, propagate_forward_time: 64.3768463135, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3353500366
validation[          , genome  6390] predictions:    6526/  10000 (65.26%), best:    6724/10000 (67.24%), error:      9882.05078, best error:      9504.45605 on epoch:    36, epoch:   38/50, mu: 0.9023388624, learning_rate: 0.0000874898, weight_decay: 0.0000364893

epoch time: 1070.0198974609s, input_fired_time: 16.2893638611, output_fired_time: 5.4884552956, propagate_forward_time: 320.8068237305, propagate_backward_time: 719.0838012695, weight_update_time: 0.2738283277, other_time: 8.3515625000
epoch time: 67.0627670288s, input_fired_time: 1.0221074820, output_fired_time: 0.0000000000, propagate_forward_time: 64.7605056763, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2801513672
validation[          , genome  6390] predictions:    6617/  10000 (66.17%), best:    6724/10000 (67.24%), error:      9673.31543, best error:      9504.45605 on epoch:    36, epoch:   39/50, mu: 0.9062156677, learning_rate: 0.0000800756, weight_decay: 0.0000341691

epoch time: 1072.7951660156s, input_fired_time: 16.3590946198, output_fired_time: 5.5069952011, propagate_forward_time: 321.6667785645, propagate_backward_time: 720.8685302734, weight_update_time: 0.2744009495, other_time: 8.3937988281
epoch time: 66.9972457886s, input_fired_time: 1.0134447813, output_fired_time: 0.0000000000, propagate_forward_time: 64.7098236084, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2739791870
validation[          , genome  6390] predictions:    6768/  10000 (67.68%), best:    6768/10000 (67.68%), error:      9434.15332, best error:      9434.15332 on epoch:    40, epoch:   40/50, mu: 0.9099210501, learning_rate: 0.0000732898, weight_decay: 0.0000319965

epoch time: 1072.4918212891s, input_fired_time: 16.3419227600, output_fired_time: 5.4953322411, propagate_forward_time: 321.6469421387, propagate_backward_time: 720.7420654297, weight_update_time: 0.2739549577, other_time: 8.2655029297
epoch time: 67.0900650024s, input_fired_time: 1.0113971233, output_fired_time: 0.0000000000, propagate_forward_time: 64.7918090820, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2868576050
validation[          , genome  6390] predictions:    6716/  10000 (67.16%), best:    6768/10000 (67.68%), error:      9560.31543, best error:      9434.15332 on epoch:    40, epoch:   41/50, mu: 0.9134625196, learning_rate: 0.0000670789, weight_decay: 0.0000299620

epoch time: 1072.0889892578s, input_fired_time: 16.3346042633, output_fired_time: 5.4984154701, propagate_forward_time: 321.2348937988, propagate_backward_time: 720.7180175781, weight_update_time: 0.2741176486, other_time: 8.3031005859
epoch time: 67.0892257690s, input_fired_time: 1.0205546618, output_fired_time: 0.0000000000, propagate_forward_time: 64.7967987061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2718734741
validation[          , genome  6390] predictions:    6752/  10000 (67.52%), best:    6768/10000 (67.68%), error:      9446.56641, best error:      9434.15332 on epoch:    40, epoch:   42/50, mu: 0.9168474078, learning_rate: 0.0000613945, weight_decay: 0.0000280569

epoch time: 1072.4639892578s, input_fired_time: 16.3350925446, output_fired_time: 5.4944243431, propagate_forward_time: 321.6847229004, propagate_backward_time: 720.6438598633, weight_update_time: 0.2742023766, other_time: 8.3059692383
epoch time: 67.0275802612s, input_fired_time: 1.0157042742, output_fired_time: 0.0000000000, propagate_forward_time: 64.7309494019, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2809295654
validation[          , genome  6390] predictions:    6767/  10000 (67.67%), best:    6767/10000 (67.67%), error:      9415.92773, best error:      9415.92773 on epoch:    43, epoch:   43/50, mu: 0.9200825691, learning_rate: 0.0000561917, weight_decay: 0.0000262729

epoch time: 1072.5311279297s, input_fired_time: 16.3605995178, output_fired_time: 5.4936237335, propagate_forward_time: 321.7300415039, propagate_backward_time: 720.6118774414, weight_update_time: 0.2739669681, other_time: 8.3349609375
epoch time: 67.0673675537s, input_fired_time: 1.0186562538, output_fired_time: 0.0000000000, propagate_forward_time: 64.7764816284, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2722320557
validation[          , genome  6390] predictions:    6721/  10000 (67.21%), best:    6767/10000 (67.67%), error:      9576.83105, best error:      9415.92773 on epoch:    43, epoch:   44/50, mu: 0.9231746793, learning_rate: 0.0000514298, weight_decay: 0.0000246023

epoch time: 1072.2359619141s, input_fired_time: 16.3623580933, output_fired_time: 5.4979920387, propagate_forward_time: 321.3450317383, propagate_backward_time: 720.7286987305, weight_update_time: 0.2743522227, other_time: 8.3018798828
epoch time: 67.0277252197s, input_fired_time: 1.0149413347, output_fired_time: 0.0000000000, propagate_forward_time: 64.7417907715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2709960938
validation[          , genome  6390] predictions:    6739/  10000 (67.39%), best:    6739/10000 (67.39%), error:      9407.69043, best error:      9407.69043 on epoch:    45, epoch:   45/50, mu: 0.9261300564, learning_rate: 0.0000470715, weight_decay: 0.0000230380

epoch time: 1072.9398193359s, input_fired_time: 16.3982067108, output_fired_time: 5.4958925247, propagate_forward_time: 321.7844848633, propagate_backward_time: 720.9249877930, weight_update_time: 0.2741481066, other_time: 8.3363037109
epoch time: 67.1210403442s, input_fired_time: 1.0178711414, output_fired_time: 0.0000000000, propagate_forward_time: 64.8306121826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2725601196
validation[          , genome  6390] predictions:    6719/  10000 (67.19%), best:    6739/10000 (67.39%), error:      9487.15820, best error:      9407.69043 on epoch:    45, epoch:   46/50, mu: 0.9289547205, learning_rate: 0.0000430825, weight_decay: 0.0000215731

epoch time: 1071.4042968750s, input_fired_time: 16.3529510498, output_fired_time: 5.5027437210, propagate_forward_time: 321.1512451172, propagate_backward_time: 720.1051025391, weight_update_time: 0.2746778727, other_time: 8.2923583984
epoch time: 67.0162048340s, input_fired_time: 1.0202944279, output_fired_time: 0.0000000000, propagate_forward_time: 64.7255096436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2704010010
validation[          , genome  6390] predictions:    6753/  10000 (67.53%), best:    6753/10000 (67.53%), error:      9338.93555, best error:      9338.93555 on epoch:    47, epoch:   47/50, mu: 0.9316544533, learning_rate: 0.0000394315, weight_decay: 0.0000202014

epoch time: 1072.3111572266s, input_fired_time: 16.3713073730, output_fired_time: 5.5016613007, propagate_forward_time: 321.3330383301, propagate_backward_time: 720.7745361328, weight_update_time: 0.2734140754, other_time: 8.3305664063
epoch time: 67.0902557373s, input_fired_time: 1.0176974535, output_fired_time: 0.0000000000, propagate_forward_time: 64.8005447388, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2720108032
validation[          , genome  6390] predictions:    6768/  10000 (67.68%), best:    6753/10000 (67.53%), error:      9353.13965, best error:      9338.93555 on epoch:    47, epoch:   48/50, mu: 0.9342347980, learning_rate: 0.0000360900, weight_decay: 0.0000189169

epoch time: 1072.6765136719s, input_fired_time: 16.3782310486, output_fired_time: 5.5024619102, propagate_forward_time: 321.4958801270, propagate_backward_time: 721.0030517578, weight_update_time: 0.2747633755, other_time: 8.2969970703
epoch time: 67.0095291138s, input_fired_time: 1.0223605633, output_fired_time: 0.0000000000, propagate_forward_time: 64.7156982422, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2714691162
validation[          , genome  6390] predictions:    6762/  10000 (67.62%), best:    6753/10000 (67.53%), error:      9341.82422, best error:      9338.93555 on epoch:    47, epoch:   49/50, mu: 0.9367009997, learning_rate: 0.0000330316, weight_decay: 0.0000177141

epoch time: 1072.3605957031s, input_fired_time: 16.3672943115, output_fired_time: 5.4944510460, propagate_forward_time: 321.2413330078, propagate_backward_time: 720.9410400391, weight_update_time: 0.2749647200, other_time: 8.3164062500
epoch time: 67.0072937012s, input_fired_time: 1.0184817314, output_fired_time: 0.0000000000, propagate_forward_time: 64.7070236206, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2817916870
validation[          , genome  6390] predictions:    6782/  10000 (67.82%), best:    6782/10000 (67.82%), error:      9298.50586, best error:      9298.50586 on epoch:    50, epoch:   50/50, mu: 0.9390581250, learning_rate: 0.0000302324, weight_decay: 0.0000165877

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 332.7044372559s, input_fired_time: 5.0745010376, output_fired_time: 0.0000000000, propagate_forward_time: 321.1992797852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4306640625
best training[          , genome  6390] predictions:   38269/  50000 (76.54%), best:    6782/10000 (67.82%), error:     36154.51172, best error:      9298.50586 on epoch:    50, epoch:   51/50, mu: 0.9413110018, learning_rate: 0.0000276704, weight_decay: 0.0000155330
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 67.8242645264s, input_fired_time: 1.0183080435, output_fired_time: 0.0000000000, propagate_forward_time: 64.6854553223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1204986572
   testing[          , genome  6390] predictions:    6782/  10000 (67.82%), best:    6782/10000 (67.82%), error:      9298.50586, best error:      9298.50586 on epoch:    50, epoch:   51/50, mu: 0.9413110018, learning_rate: 0.0000276704, weight_decay: 0.0000155330
backpropagation finished successfully!
09:48:54 (12732): called boinc_finish(0)

</stderr_txt>
]]>