Nameexact_genome_1542649503_174_6320_0
Workunit4099005
Created19 Nov 2018, 23:44:30 UTC
Sent19 Nov 2018, 23:45:29 UTC
Report deadline3 Jan 2019, 14:45:09 UTC
Received21 Nov 2018, 1:13:41 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60660
Run time13 hours 43 min 4 sec
CPU time13 hours 27 min 27 sec
Validate stateValid
Credit11,156.19
Device peak FLOPS3.44 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size526.18 MB
Peak swap size546.09 MB
Peak disk usage9.66 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_174_6320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_174_6320_0_r1350268184_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_174_6320.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.5
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0125
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.0005
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6320
read normal distribution: '1 -1.24922525882721 -0.337624698877335'
generator_str: '1163959074'
read generator: 1163959074
read generated_by_map:
1 merge_node 1
reading 92 nodes.
reading 322 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1231616195
post shuffle 1: 1031769412
epoch time: 893.69970703125s, input_fired_time: 9.30097770690918, output_fired_time: 6.92389917373657, propagate_forward_time: 321.522277832031, propagate_backward_time: 544.124389648438, weight_update_time: 1.04800546169281, other_time: 11.8281860351563
epoch time: 67.3659133911133s, input_fired_time: 1.06786227226257, output_fired_time: 0, propagate_forward_time: 64.2003936767578, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.09765625
validation[          , genome  6320] predictions:    4827/  10000 (48.27%), best:    4827/10000 (48.27%), error:     18260.79688, best error:     18260.79688 on epoch:     0, epoch:    0/50, mu: 0.5000000000, learning_rate: 0.0125000002, weight_decay: 0.0005000000

epoch time: 892.1481933594s, input_fired_time: 9.0955924988, output_fired_time: 7.0378599167, propagate_forward_time: 320.5466613770, propagate_backward_time: 543.8163452148, weight_update_time: 1.0351369381, other_time: 11.6517944336
epoch time: 66.3110733032s, input_fired_time: 0.9883089662, output_fired_time: 0.0000000000, propagate_forward_time: 63.3537826538, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9689788818
validation[          , genome  6320] predictions:    4667/  10000 (46.67%), best:    4827/10000 (48.27%), error:     18736.46094, best error:     18260.79688 on epoch:     0, epoch:    1/50, mu: 0.5245000124, learning_rate: 0.0118749999, weight_decay: 0.0004750000

epoch time: 894.1824951172s, input_fired_time: 9.2687454224, output_fired_time: 6.9278964996, propagate_forward_time: 321.5335083008, propagate_backward_time: 544.4706420898, weight_update_time: 1.0354384184, other_time: 11.9816894531
epoch time: 67.1033477783s, input_fired_time: 0.9910781384, output_fired_time: 0.0000000000, propagate_forward_time: 64.0724487305, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0398178101
validation[          , genome  6320] predictions:    4929/  10000 (49.29%), best:    4929/10000 (49.29%), error:     17770.28711, best error:     17770.28711 on epoch:     2, epoch:    2/50, mu: 0.5477750301, learning_rate: 0.0112812500, weight_decay: 0.0004512500

epoch time: 1089.1409912109s, input_fired_time: 9.0543107986, output_fired_time: 6.9712538719, propagate_forward_time: 514.8499145508, propagate_backward_time: 546.2208251953, weight_update_time: 1.0488853455, other_time: 12.0446166992
epoch time: 67.2874526978s, input_fired_time: 1.0398824215, output_fired_time: 0.0000000000, propagate_forward_time: 64.1892776489, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0582962036
validation[          , genome  6320] predictions:    5078/  10000 (50.78%), best:    5078/10000 (50.78%), error:     15787.17676, best error:     15787.17676 on epoch:     3, epoch:    3/50, mu: 0.5698862672, learning_rate: 0.0107171871, weight_decay: 0.0004286875

epoch time: 894.7333984375s, input_fired_time: 9.1301088333, output_fired_time: 6.9321117401, propagate_forward_time: 321.3786010742, propagate_backward_time: 545.3829956055, weight_update_time: 1.0263439417, other_time: 11.9095458984
epoch time: 66.6901321411s, input_fired_time: 1.0121784210, output_fired_time: 0.0000000000, propagate_forward_time: 63.5774536133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1005020142
validation[          , genome  6320] predictions:    5160/  10000 (51.60%), best:    5078/10000 (50.78%), error:     16431.81250, best error:     15787.17676 on epoch:     3, epoch:    4/50, mu: 0.5908919573, learning_rate: 0.0101813274, weight_decay: 0.0004072531

epoch time: 903.4418945313s, input_fired_time: 9.0444240570, output_fired_time: 7.0033898354, propagate_forward_time: 325.1235656738, propagate_backward_time: 550.4851074219, weight_update_time: 1.0591167212, other_time: 11.7854003906
epoch time: 67.5960388184s, input_fired_time: 1.0740679502, output_fired_time: 0.0000000000, propagate_forward_time: 64.4775009155, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0444717407
validation[          , genome  6320] predictions:    5266/  10000 (52.66%), best:    5266/10000 (52.66%), error:     15223.17383, best error:     15223.17383 on epoch:     5, epoch:    5/50, mu: 0.6108473539, learning_rate: 0.0096722608, weight_decay: 0.0003868904

epoch time: 898.1972045898s, input_fired_time: 9.2832918167, output_fired_time: 6.9840250015, propagate_forward_time: 323.3338012695, propagate_backward_time: 546.6206054688, weight_update_time: 1.0395995378, other_time: 11.9755249023
epoch time: 67.2509918213s, input_fired_time: 1.0086321831, output_fired_time: 0.0000000000, propagate_forward_time: 64.0759658813, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1663970947
validation[          , genome  6320] predictions:    5340/  10000 (53.40%), best:    5340/10000 (53.40%), error:     14745.98047, best error:     14745.98047 on epoch:     6, epoch:    6/50, mu: 0.6298049688, learning_rate: 0.0091886474, weight_decay: 0.0003675459

epoch time: 892.8024902344s, input_fired_time: 9.0381755829, output_fired_time: 6.9939370155, propagate_forward_time: 320.9303588867, propagate_backward_time: 543.8316040039, weight_update_time: 1.0387102365, other_time: 12.0084228516
epoch time: 67.5485076904s, input_fired_time: 1.0237352848, output_fired_time: 0.0000000000, propagate_forward_time: 64.3692932129, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1554794312
validation[          , genome  6320] predictions:    4648/  10000 (46.48%), best:    5340/10000 (53.40%), error:     18034.02344, best error:     14745.98047 on epoch:     6, epoch:    7/50, mu: 0.6478147507, learning_rate: 0.0087292148, weight_decay: 0.0003491686

epoch time: 896.8977661133s, input_fired_time: 9.0967054367, output_fired_time: 7.0055646896, propagate_forward_time: 322.5861816406, propagate_backward_time: 546.1105957031, weight_update_time: 1.0423463583, other_time: 12.0987548828
epoch time: 66.8214950562s, input_fired_time: 1.0172232389, output_fired_time: 0.0000000000, propagate_forward_time: 63.7631530762, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0411224365
validation[          , genome  6320] predictions:    5141/  10000 (51.41%), best:    5340/10000 (53.40%), error:     15603.15723, best error:     14745.98047 on epoch:     6, epoch:    8/50, mu: 0.6649240255, learning_rate: 0.0082927542, weight_decay: 0.0003317102

epoch time: 888.4534912109s, input_fired_time: 8.9018383026, output_fired_time: 6.9262199402, propagate_forward_time: 319.8116455078, propagate_backward_time: 541.1080322266, weight_update_time: 1.0344135761, other_time: 11.7057495117
epoch time: 67.5786209106s, input_fired_time: 0.9970318675, output_fired_time: 0.0000000000, propagate_forward_time: 64.4906082153, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0909805298
validation[          , genome  6320] predictions:    5296/  10000 (52.96%), best:    5340/10000 (53.40%), error:     14865.19336, best error:     14745.98047 on epoch:     6, epoch:    9/50, mu: 0.6811778545, learning_rate: 0.0078781163, weight_decay: 0.0003151247

epoch time: 892.9652709961s, input_fired_time: 9.0464630127, output_fired_time: 7.0234608650, propagate_forward_time: 321.1425781250, propagate_backward_time: 543.8115844727, weight_update_time: 1.0459679365, other_time: 11.9412231445
epoch time: 67.3777847290s, input_fired_time: 1.0325359106, output_fired_time: 0.0000000000, propagate_forward_time: 64.2099151611, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1353302002
validation[          , genome  6320] predictions:    5345/  10000 (53.45%), best:    5345/10000 (53.45%), error:     14189.95801, best error:     14189.95801 on epoch:    10, epoch:   10/50, mu: 0.6966189742, learning_rate: 0.0074842102, weight_decay: 0.0002993684

epoch time: 893.6364135742s, input_fired_time: 9.0709819794, output_fired_time: 6.9581794739, propagate_forward_time: 321.7635803223, propagate_backward_time: 544.0124511719, weight_update_time: 1.0273258686, other_time: 11.8311767578
epoch time: 68.0452804565s, input_fired_time: 1.0522584915, output_fired_time: 0.0000000000, propagate_forward_time: 64.7426071167, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2504119873
validation[          , genome  6320] predictions:    4985/  10000 (49.85%), best:    5345/10000 (53.45%), error:     16029.59375, best error:     14189.95801 on epoch:    10, epoch:   11/50, mu: 0.7112880349, learning_rate: 0.0071099997, weight_decay: 0.0002844000

epoch time: 2877.4062500000s, input_fired_time: 9.0607166290, output_fired_time: 6.9411292076, propagate_forward_time: 320.1866149902, propagate_backward_time: 2529.4306640625, weight_update_time: 1.0389022827, other_time: 11.7871093750
epoch time: 67.0979080200s, input_fired_time: 1.0157711506, output_fired_time: 0.0000000000, propagate_forward_time: 64.0505905151, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0315475464
validation[          , genome  6320] predictions:    5250/  10000 (52.50%), best:    5345/10000 (53.45%), error:     14795.89551, best error:     14189.95801 on epoch:    10, epoch:   12/50, mu: 0.7252236605, learning_rate: 0.0067544994, weight_decay: 0.0002701800

epoch time: 929.9591674805s, input_fired_time: 8.9540767670, output_fired_time: 6.8875298500, propagate_forward_time: 320.1864624023, propagate_backward_time: 582.3347167969, weight_update_time: 1.0510165691, other_time: 11.5963745117
epoch time: 66.8430328369s, input_fired_time: 0.9900871515, output_fired_time: 0.0000000000, propagate_forward_time: 63.8338088989, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0191345215
validation[          , genome  6320] predictions:    5194/  10000 (51.94%), best:    5345/10000 (53.45%), error:     15238.10547, best error:     14189.95801 on epoch:    10, epoch:   13/50, mu: 0.7384625077, learning_rate: 0.0064167744, weight_decay: 0.0002566710

epoch time: 892.4697875977s, input_fired_time: 9.0947856903, output_fired_time: 6.9541716576, propagate_forward_time: 320.9782714844, propagate_backward_time: 543.6360473633, weight_update_time: 1.0368623734, other_time: 11.8065185547
epoch time: 67.4053268433s, input_fired_time: 1.0387066603, output_fired_time: 0.0000000000, propagate_forward_time: 64.2384567261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1281661987
validation[          , genome  6320] predictions:    5487/  10000 (54.87%), best:    5487/10000 (54.87%), error:     13586.58301, best error:     13586.58301 on epoch:    14, epoch:   14/50, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 892.9262695313s, input_fired_time: 9.0222826004, output_fired_time: 6.9863724709, propagate_forward_time: 321.7086486816, propagate_backward_time: 543.4716186523, weight_update_time: 1.0348802805, other_time: 11.7373657227
epoch time: 67.1638183594s, input_fired_time: 1.0342094898, output_fired_time: 0.0000000000, propagate_forward_time: 64.0867843628, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0428237915
validation[          , genome  6320] predictions:    5459/  10000 (54.59%), best:    5487/10000 (54.87%), error:     13878.09863, best error:     13586.58301 on epoch:    14, epoch:   15/50, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 889.1009521484s, input_fired_time: 8.9993753433, output_fired_time: 6.8652939796, propagate_forward_time: 320.0239257813, propagate_backward_time: 541.7145996094, weight_update_time: 1.0258740187, other_time: 11.4977416992
epoch time: 68.2713775635s, input_fired_time: 1.0584113598, output_fired_time: 0.0000000000, propagate_forward_time: 65.0478820801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1650848389
validation[          , genome  6320] predictions:    5309/  10000 (53.09%), best:    5487/10000 (54.87%), error:     14272.79004, best error:     13586.58301 on epoch:    14, epoch:   16/50, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 893.6007080078s, input_fired_time: 8.9864511490, output_fired_time: 7.0789318085, propagate_forward_time: 321.3191528320, propagate_backward_time: 544.5648193359, weight_update_time: 1.0258224010, other_time: 11.6513671875
epoch time: 67.6734008789s, input_fired_time: 1.0398555994, output_fired_time: 0.0000000000, propagate_forward_time: 64.5239639282, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1095809937
validation[          , genome  6320] predictions:    5533/  10000 (55.33%), best:    5487/10000 (54.87%), error:     13969.32617, best error:     13586.58301 on epoch:    14, epoch:   17/50, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 892.6279296875s, input_fired_time: 9.0097284317, output_fired_time: 6.9074535370, propagate_forward_time: 321.0675048828, propagate_backward_time: 543.8104248047, weight_update_time: 1.0250067711, other_time: 11.8328247070
epoch time: 67.0582046509s, input_fired_time: 1.0221379995, output_fired_time: 0.0000000000, propagate_forward_time: 63.9033012390, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1327629089
validation[          , genome  6320] predictions:    5478/  10000 (54.78%), best:    5487/10000 (54.87%), error:     14241.04590, best error:     13586.58301 on epoch:    14, epoch:   18/50, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 890.7853393555s, input_fired_time: 9.0662841797, output_fired_time: 6.9823317528, propagate_forward_time: 320.2726745605, propagate_backward_time: 542.7260131836, weight_update_time: 1.0327132940, other_time: 11.7379760742
epoch time: 67.1873779297s, input_fired_time: 0.9676094055, output_fired_time: 0.0000000000, propagate_forward_time: 64.1573715210, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0624008179
validation[          , genome  6320] predictions:    4862/  10000 (48.62%), best:    5487/10000 (54.87%), error:     17769.91602, best error:     13586.58301 on epoch:    14, epoch:   19/50, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

epoch time: 1036.2298583984s, input_fired_time: 8.9392490387, output_fired_time: 6.8731937408, propagate_forward_time: 319.2614746094, propagate_backward_time: 689.6235961914, weight_update_time: 1.0242859125, other_time: 11.5324096680
epoch time: 66.9618225098s, input_fired_time: 0.9946915507, output_fired_time: 0.0000000000, propagate_forward_time: 63.8904075623, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0767250061
validation[          , genome  6320] predictions:    5412/  10000 (54.12%), best:    5487/10000 (54.87%), error:     13975.01074, best error:     13586.58301 on epoch:    14, epoch:   20/50, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 892.9801025391s, input_fired_time: 9.0884389877, output_fired_time: 6.9234395027, propagate_forward_time: 321.6148071289, propagate_backward_time: 543.6104125977, weight_update_time: 1.0389726162, other_time: 11.7429809570
epoch time: 67.8547897339s, input_fired_time: 1.0350095034, output_fired_time: 0.0000000000, propagate_forward_time: 64.7251739502, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0946044922
validation[          , genome  6320] predictions:    5654/  10000 (56.54%), best:    5654/10000 (56.54%), error:     13181.61621, best error:     13181.61621 on epoch:    21, epoch:   21/50, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 1715.2384033203s, input_fired_time: 9.1659107208, output_fired_time: 6.9809246063, propagate_forward_time: 320.5260314941, propagate_backward_time: 1366.4853515625, weight_update_time: 1.0299737453, other_time: 12.0802001953
epoch time: 67.7224731445s, input_fired_time: 1.0300548077, output_fired_time: 0.0000000000, propagate_forward_time: 64.5751190186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1173019409
validation[          , genome  6320] predictions:    5589/  10000 (55.89%), best:    5654/10000 (56.54%), error:     13609.05371, best error:     13181.61621 on epoch:    21, epoch:   22/50, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 887.8672485352s, input_fired_time: 9.0795803070, output_fired_time: 6.9410772324, propagate_forward_time: 319.2438964844, propagate_backward_time: 540.7404174805, weight_update_time: 1.0259718895, other_time: 11.8622436523
epoch time: 67.1720123291s, input_fired_time: 1.0196144581, output_fired_time: 0.0000000000, propagate_forward_time: 64.1245040894, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0278930664
validation[          , genome  6320] predictions:    5682/  10000 (56.82%), best:    5682/10000 (56.82%), error:     13154.25293, best error:     13154.25293 on epoch:    23, epoch:   23/50, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 920.1937255859s, input_fired_time: 9.5453233719, output_fired_time: 7.0689558983, propagate_forward_time: 330.1915588379, propagate_backward_time: 560.9597167969, weight_update_time: 1.0735888481, other_time: 12.4281005859
epoch time: 77.6321868896s, input_fired_time: 1.2839255333, output_fired_time: 0.0000000000, propagate_forward_time: 73.9437408447, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4045181274
validation[          , genome  6320] predictions:    5575/  10000 (55.75%), best:    5682/10000 (56.82%), error:     13415.08984, best error:     13154.25293 on epoch:    23, epoch:   24/50, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 886.1029052734s, input_fired_time: 9.1218452454, output_fired_time: 6.8491563797, propagate_forward_time: 318.6849365234, propagate_backward_time: 539.3228759766, weight_update_time: 1.0355861187, other_time: 12.1240844727
epoch time: 67.0750503540s, input_fired_time: 1.0288100243, output_fired_time: 0.0000000000, propagate_forward_time: 63.9354896545, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1107521057
validation[          , genome  6320] predictions:    5564/  10000 (55.64%), best:    5682/10000 (56.82%), error:     13818.94727, best error:     13154.25293 on epoch:    23, epoch:   25/50, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 891.7160034180s, input_fired_time: 9.2533931732, output_fired_time: 6.9302196503, propagate_forward_time: 320.7506103516, propagate_backward_time: 542.7626342773, weight_update_time: 1.0632517338, other_time: 12.0191040039
epoch time: 66.4416503906s, input_fired_time: 1.0225756168, output_fired_time: 0.0000000000, propagate_forward_time: 63.2358474731, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1832275391
validation[          , genome  6320] predictions:    5504/  10000 (55.04%), best:    5682/10000 (56.82%), error:     13823.83203, best error:     13154.25293 on epoch:    23, epoch:   26/50, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 887.9119262695s, input_fired_time: 9.0852937698, output_fired_time: 6.8697719574, propagate_forward_time: 319.4360656738, propagate_backward_time: 540.5140991211, weight_update_time: 1.0291389227, other_time: 12.0067749023
epoch time: 67.1964263916s, input_fired_time: 1.0453107357, output_fired_time: 0.0000000000, propagate_forward_time: 64.0699157715, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0811996460
validation[          , genome  6320] predictions:    5610/  10000 (56.10%), best:    5682/10000 (56.82%), error:     13303.96680, best error:     13154.25293 on epoch:    23, epoch:   27/50, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 885.3519897461s, input_fired_time: 9.0266294479, output_fired_time: 6.8592410088, propagate_forward_time: 318.5762329102, propagate_backward_time: 538.9818725586, weight_update_time: 1.0267301798, other_time: 11.9080200195
epoch time: 67.1151809692s, input_fired_time: 1.0179208517, output_fired_time: 0.0000000000, propagate_forward_time: 64.0153350830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0819244385
validation[          , genome  6320] predictions:    5601/  10000 (56.01%), best:    5682/10000 (56.82%), error:     13517.72559, best error:     13154.25293 on epoch:    23, epoch:   28/50, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1527.2316894531s, input_fired_time: 9.1680793762, output_fired_time: 6.9317784309, propagate_forward_time: 320.8927307129, propagate_backward_time: 1178.2707519531, weight_update_time: 1.0312589407, other_time: 11.9683837891
epoch time: 67.4060668945s, input_fired_time: 1.0423408747, output_fired_time: 0.0000000000, propagate_forward_time: 64.3171310425, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0465927124
validation[          , genome  6320] predictions:    5684/  10000 (56.84%), best:    5682/10000 (56.82%), error:     13456.04980, best error:     13154.25293 on epoch:    23, epoch:   29/50, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 892.7852172852s, input_fired_time: 9.2302494049, output_fired_time: 6.9652667046, propagate_forward_time: 321.2406005859, propagate_backward_time: 543.5296020508, weight_update_time: 1.0280363560, other_time: 11.8195190430
epoch time: 67.2142105103s, input_fired_time: 1.0450476408, output_fired_time: 0.0000000000, propagate_forward_time: 64.0804672241, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0886993408
validation[          , genome  6320] predictions:    5643/  10000 (56.43%), best:    5682/10000 (56.82%), error:     13521.72852, best error:     13154.25293 on epoch:    23, epoch:   30/50, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 888.9887084961s, input_fired_time: 9.1195096970, output_fired_time: 6.8946871758, propagate_forward_time: 320.0223388672, propagate_backward_time: 541.2709350586, weight_update_time: 1.0191782713, other_time: 11.6812133789
epoch time: 67.2827682495s, input_fired_time: 1.0333406925, output_fired_time: 0.0000000000, propagate_forward_time: 64.1998748779, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0495529175
validation[          , genome  6320] predictions:    5683/  10000 (56.83%), best:    5682/10000 (56.82%), error:     13303.61133, best error:     13154.25293 on epoch:    23, epoch:   31/50, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 890.3643798828s, input_fired_time: 9.2074422836, output_fired_time: 6.8655552864, propagate_forward_time: 320.1976928711, propagate_backward_time: 542.3075561523, weight_update_time: 1.0178728104, other_time: 11.7861328125
epoch time: 67.0788497925s, input_fired_time: 1.0351974964, output_fired_time: 0.0000000000, propagate_forward_time: 63.9778594971, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0657958984
validation[          , genome  6320] predictions:    5583/  10000 (55.83%), best:    5682/10000 (56.82%), error:     13466.19727, best error:     13154.25293 on epoch:    23, epoch:   32/50, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 1577.9083251953s, input_fired_time: 9.1202688217, output_fired_time: 6.8531055450, propagate_forward_time: 508.7100524902, propagate_backward_time: 1041.4433593750, weight_update_time: 1.0259587765, other_time: 11.7814941406
epoch time: 66.4308166504s, input_fired_time: 1.0120525360, output_fired_time: 0.0000000000, propagate_forward_time: 63.3163795471, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1023826599
validation[          , genome  6320] predictions:    5840/  10000 (58.40%), best:    5840/10000 (58.40%), error:     12668.91406, best error:     12668.91406 on epoch:    33, epoch:   33/50, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 2990.6245117188s, input_fired_time: 9.1250839233, output_fired_time: 2111.4289550781, propagate_forward_time: 319.0643005371, propagate_backward_time: 539.3265991211, weight_update_time: 1.0181610584, other_time: 11.6796264648
epoch time: 66.9426040649s, input_fired_time: 1.0283948183, output_fired_time: 0.0000000000, propagate_forward_time: 63.8029556274, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1112518311
validation[          , genome  6320] predictions:    5605/  10000 (56.05%), best:    5840/10000 (58.40%), error:     13131.13672, best error:     12668.91406 on epoch:    33, epoch:   34/50, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 886.2663574219s, input_fired_time: 9.0473871231, output_fired_time: 6.8026475906, propagate_forward_time: 318.9942016602, propagate_backward_time: 539.9425048828, weight_update_time: 1.0106166601, other_time: 11.4796142578
epoch time: 67.1585464478s, input_fired_time: 1.0303667784, output_fired_time: 0.0000000000, propagate_forward_time: 64.0908279419, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0373535156
validation[          , genome  6320] predictions:    5718/  10000 (57.18%), best:    5840/10000 (58.40%), error:     12996.36035, best error:     12668.91406 on epoch:    33, epoch:   35/50, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 1369.7899169922s, input_fired_time: 9.1155967712, output_fired_time: 6.9418129921, propagate_forward_time: 320.0620422363, propagate_backward_time: 1022.1038208008, weight_update_time: 1.0246213675, other_time: 11.5667114258
epoch time: 66.9106140137s, input_fired_time: 1.0539212227, output_fired_time: 0.0000000000, propagate_forward_time: 63.8443450928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0123443604
validation[          , genome  6320] predictions:    5736/  10000 (57.36%), best:    5840/10000 (58.40%), error:     12906.54590, best error:     12668.91406 on epoch:    33, epoch:   36/50, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 30451.2753906250s, input_fired_time: 9.2151842117, output_fired_time: 6.8658423424, propagate_forward_time: 29843.8476562500, propagate_backward_time: 544.6492309570, weight_update_time: 1.0116373301, other_time: 46.6984252930
epoch time: 67.3762359619s, input_fired_time: 1.0635662079, output_fired_time: 0.0000000000, propagate_forward_time: 64.2241134644, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0885543823
validation[          , genome  6320] predictions:    5634/  10000 (56.34%), best:    5840/10000 (58.40%), error:     13509.35254, best error:     12668.91406 on epoch:    33, epoch:   37/50, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 894.0013427734s, input_fired_time: 9.2495021820, output_fired_time: 6.9129309654, propagate_forward_time: 321.8631286621, propagate_backward_time: 544.1613159180, weight_update_time: 1.0249075890, other_time: 11.8145141602
epoch time: 67.4895019531s, input_fired_time: 1.0423086882, output_fired_time: 0.0000000000, propagate_forward_time: 64.3486022949, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0985946655
validation[          , genome  6320] predictions:    5857/  10000 (58.57%), best:    5840/10000 (58.40%), error:     12679.89551, best error:     12668.91406 on epoch:    33, epoch:   38/50, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 894.6870117188s, input_fired_time: 9.3297176361, output_fired_time: 6.8967137337, propagate_forward_time: 322.0291748047, propagate_backward_time: 544.6181640625, weight_update_time: 1.0417267084, other_time: 11.8132324219
epoch time: 67.5916442871s, input_fired_time: 1.0675520897, output_fired_time: 0.0000000000, propagate_forward_time: 64.4076004028, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1164932251
validation[          , genome  6320] predictions:    5782/  10000 (57.82%), best:    5840/10000 (58.40%), error:     12894.28613, best error:     12668.91406 on epoch:    33, epoch:   39/50, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 892.9977416992s, input_fired_time: 9.3018341064, output_fired_time: 6.9045410156, propagate_forward_time: 321.3895263672, propagate_backward_time: 543.6196289063, weight_update_time: 1.0393799543, other_time: 11.7822265625
epoch time: 67.2200012207s, input_fired_time: 1.0553847551, output_fired_time: 0.0000000000, propagate_forward_time: 64.0592193604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1054000854
validation[          , genome  6320] predictions:    5628/  10000 (56.28%), best:    5840/10000 (58.40%), error:     13701.95215, best error:     12668.91406 on epoch:    33, epoch:   40/50, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 892.3115234375s, input_fired_time: 9.2485561371, output_fired_time: 6.9629626274, propagate_forward_time: 320.9109191895, propagate_backward_time: 543.4040527344, weight_update_time: 1.0218003988, other_time: 11.7850341797
epoch time: 67.5472488403s, input_fired_time: 1.0671339035, output_fired_time: 0.0000000000, propagate_forward_time: 64.3768920898, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1032257080
validation[          , genome  6320] predictions:    5885/  10000 (58.85%), best:    5885/10000 (58.85%), error:     12647.53027, best error:     12647.53027 on epoch:    41, epoch:   41/50, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 892.7567138672s, input_fired_time: 9.3273220062, output_fired_time: 6.9074521065, propagate_forward_time: 321.3528442383, propagate_backward_time: 543.4526367188, weight_update_time: 1.0230759382, other_time: 11.7164306641
epoch time: 68.2412109375s, input_fired_time: 1.0624972582, output_fired_time: 0.0000000000, propagate_forward_time: 65.0640563965, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1146545410
validation[          , genome  6320] predictions:    5797/  10000 (57.97%), best:    5885/10000 (58.85%), error:     12921.26270, best error:     12647.53027 on epoch:    41, epoch:   42/50, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 891.6884155273s, input_fired_time: 9.2935113907, output_fired_time: 6.8545227051, propagate_forward_time: 321.0845642090, propagate_backward_time: 542.6048583984, weight_update_time: 1.0707930326, other_time: 11.8509521484
epoch time: 67.6034240723s, input_fired_time: 1.0900340080, output_fired_time: 0.0000000000, propagate_forward_time: 64.3755645752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1378250122
validation[          , genome  6320] predictions:    5909/  10000 (59.09%), best:    5909/10000 (59.09%), error:     12560.65527, best error:     12560.65527 on epoch:    43, epoch:   43/50, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 1118.7604980469s, input_fired_time: 9.3484058380, output_fired_time: 6.9030008316, propagate_forward_time: 321.8209228516, propagate_backward_time: 768.8770141602, weight_update_time: 1.0438791513, other_time: 11.8112182617
epoch time: 67.8069915771s, input_fired_time: 1.0887466669, output_fired_time: 0.0000000000, propagate_forward_time: 64.6114578247, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1067886353
validation[          , genome  6320] predictions:    5960/  10000 (59.60%), best:    5960/10000 (59.60%), error:     12509.43945, best error:     12509.43945 on epoch:    44, epoch:   44/50, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 1067.2622070313s, input_fired_time: 9.2973890305, output_fired_time: 6.8966550827, propagate_forward_time: 496.1686401367, propagate_backward_time: 543.0677490234, weight_update_time: 1.0365731716, other_time: 11.8318481445
epoch time: 67.3246154785s, input_fired_time: 1.0841023922, output_fired_time: 0.0000000000, propagate_forward_time: 64.1078643799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1326522827
validation[          , genome  6320] predictions:    5892/  10000 (58.92%), best:    5960/10000 (59.60%), error:     12545.57617, best error:     12509.43945 on epoch:    44, epoch:   45/50, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 891.1228637695s, input_fired_time: 9.3161821365, output_fired_time: 6.9231119156, propagate_forward_time: 320.6415405273, propagate_backward_time: 542.5545654297, weight_update_time: 1.0409029722, other_time: 11.6875000000
epoch time: 67.0515594482s, input_fired_time: 1.0648884773, output_fired_time: 0.0000000000, propagate_forward_time: 63.9389114380, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0477600098
validation[          , genome  6320] predictions:    5906/  10000 (59.06%), best:    5906/10000 (59.06%), error:     12410.23145, best error:     12410.23145 on epoch:    46, epoch:   46/50, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 895.6001586914s, input_fired_time: 9.3984308243, output_fired_time: 6.9632134438, propagate_forward_time: 322.3092651367, propagate_backward_time: 545.2585449219, weight_update_time: 1.0592887402, other_time: 11.6707153320
epoch time: 67.2941207886s, input_fired_time: 1.1074185371, output_fired_time: 0.0000000000, propagate_forward_time: 64.0489578247, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1377410889
validation[          , genome  6320] predictions:    5785/  10000 (57.85%), best:    5906/10000 (59.06%), error:     12945.36621, best error:     12410.23145 on epoch:    46, epoch:   47/50, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 894.9082031250s, input_fired_time: 9.3166561127, output_fired_time: 6.9039082527, propagate_forward_time: 321.7863159180, propagate_backward_time: 545.1301269531, weight_update_time: 1.0356692076, other_time: 11.7711791992
epoch time: 69.6948852539s, input_fired_time: 1.0759856701, output_fired_time: 0.0000000000, propagate_forward_time: 66.4715499878, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1473464966
validation[          , genome  6320] predictions:    5864/  10000 (58.64%), best:    5906/10000 (59.06%), error:     12551.76172, best error:     12410.23145 on epoch:    46, epoch:   48/50, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 890.5969238281s, input_fired_time: 9.3428068161, output_fired_time: 6.9066519737, propagate_forward_time: 320.4924926758, propagate_backward_time: 542.0371704102, weight_update_time: 1.0406217575, other_time: 11.8177490234
epoch time: 66.7368240356s, input_fired_time: 1.0674401522, output_fired_time: 0.0000000000, propagate_forward_time: 63.6150741577, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0543060303
validation[          , genome  6320] predictions:    5996/  10000 (59.96%), best:    5996/10000 (59.96%), error:     12205.83984, best error:     12205.83984 on epoch:    49, epoch:   49/50, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 1550.9870605469s, input_fired_time: 9.3219633102, output_fired_time: 6.8936696053, propagate_forward_time: 538.6318969727, propagate_backward_time: 984.4633789063, weight_update_time: 1.0275018215, other_time: 11.6760864258
epoch time: 67.4778213501s, input_fired_time: 1.0691871643, output_fired_time: 0.0000000000, propagate_forward_time: 64.3166275024, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0920028687
validation[          , genome  6320] predictions:    5932/  10000 (59.32%), best:    5996/10000 (59.96%), error:     12463.92285, best error:     12205.83984 on epoch:    49, epoch:   50/50, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 336.9911499023s, input_fired_time: 5.4571309090, output_fired_time: 0.0000000000, propagate_forward_time: 320.7698669434, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.7641601563
best training[          , genome  6320] predictions:   36352/  50000 (72.70%), best:    5996/10000 (59.96%), error:     39666.49609, best error:     12205.83984 on epoch:    49, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 67.8183517456s, input_fired_time: 1.0817763805, output_fired_time: 0.0000000000, propagate_forward_time: 64.6143798828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1221923828
   testing[          , genome  6320] predictions:    5996/  10000 (59.96%), best:    5996/10000 (59.96%), error:     12205.83984, best error:     12205.83984 on epoch:    49, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
backpropagation finished successfully!
19:13:13 (12556): called boinc_finish(0)

</stderr_txt>
]]>