Nameexact_genome_1542152103_178_5847_0
Workunit4087660
Created14 Nov 2018, 2:17:13 UTC
Sent14 Nov 2018, 2:32:47 UTC
Report deadline17 Jan 2019, 15:21:19 UTC
Received17 Nov 2018, 23:01:18 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID71624
Run time1 days 0 hours 18 min 51 sec
CPU time17 hours 38 min 40 sec
Validate stateValid
Credit16,133.43
Device peak FLOPS3.39 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34

Stderr output

<core_client_version>7.0.80</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_178_5847.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542152103_178_5847_0_r1832836124_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542152103_178_5847.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.490773
read mu: 0.490773
read mu_delta: 0.954328
read initial_learning_rate: 1e-05
read learning_rate: 1e-05
read learning_rate_delta: 0.92218
read initial_weight_decay: 0.000430203
read weight_decay: 0.000430203
read weight_decay_delta: 0.927714
read batch_size: 97
read epsilon: 1e-07
read alpha: 0.0999227
read input_dropout_probability: 0.00109963
read hidden_dropout_probability: 0.105804
read velocity_reset: 2018
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5847
read normal distribution: '1 0 0'
generator_str: '1115157143'
read generator: 1115157143
read generated_by_map:
1 enable_node 1
reading 105 nodes.
reading 569 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1356315032
post shuffle 1: 1639771712
epoch time: 2813.16381835938s, input_fired_time: 42.4356842041016, output_fired_time: 10.7245864868164, propagate_forward_time: 839.483276367188, propagate_backward_time: 1896.79907226563, weight_update_time: 0.28964900970459, other_time: 23.7210693359375
epoch time: 187.086547851563s, input_fired_time: 2.82518887519836, output_fired_time: 0, propagate_forward_time: 179.707702636719, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.55364990234375
validation[          , genome  5847] predictions:    6489/  10000 (64.89%), best:    6489/10000 (64.89%), error:     10733.92480, best error:     10733.92480 on epoch:     0, epoch:    0/50, mu: 0.4907729328, learning_rate: 0.0000100000, weight_decay: 0.0004302033

epoch time: 2850.8911132813s, input_fired_time: 30.3150062561, output_fired_time: 13.9281463623, propagate_forward_time: 841.3638916016, propagate_backward_time: 1949.3356933594, weight_update_time: 0.3011705577, other_time: 15.9483642578
epoch time: 175.1749114990s, input_fired_time: 1.8873379230, output_fired_time: 0.0000000000, propagate_forward_time: 159.5804901123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 13.7070770264
validation[          , genome  5847] predictions:    6430/  10000 (64.30%), best:    6489/10000 (64.89%), error:     11301.76270, best error:     10733.92480 on epoch:     0, epoch:    1/50, mu: 0.5135738254, learning_rate: 0.0000092218, weight_decay: 0.0003991057

epoch time: 2865.8410644531s, input_fired_time: 35.7828102112, output_fired_time: 8.0247116089, propagate_forward_time: 873.7468872070, propagate_backward_time: 1929.8120117188, weight_update_time: 0.3266021609, other_time: 18.4748535156
epoch time: 185.2114868164s, input_fired_time: 1.9874982834, output_fired_time: 0.0000000000, propagate_forward_time: 181.2427825928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9812011719
validation[          , genome  5847] predictions:    6333/  10000 (63.33%), best:    6489/10000 (64.89%), error:     11919.31836, best error:     10733.92480 on epoch:     0, epoch:    2/50, mu: 0.5353333354, learning_rate: 0.0000085042, weight_decay: 0.0003702559

epoch time: 4071.8352050781s, input_fired_time: 58.7672348022, output_fired_time: 19.6862335205, propagate_forward_time: 1256.1563720703, propagate_backward_time: 2703.8669433594, weight_update_time: 0.3662857115, other_time: 33.3581542969
epoch time: 180.5919952393s, input_fired_time: 0.9151601195, output_fired_time: 0.0000000000, propagate_forward_time: 175.9931335449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6837005615
validation[          , genome  5847] predictions:    6183/  10000 (61.83%), best:    6489/10000 (64.89%), error:     12609.24707, best error:     10733.92480 on epoch:     0, epoch:    3/50, mu: 0.5560990572, learning_rate: 0.0000078424, weight_decay: 0.0003434916

epoch time: 3063.5764160156s, input_fired_time: 37.8062744141, output_fired_time: 13.2763586044, propagate_forward_time: 923.8039550781, propagate_backward_time: 2062.4914550781, weight_update_time: 0.3270640373, other_time: 26.1982421875
epoch time: 181.3670959473s, input_fired_time: 0.8971004486, output_fired_time: 0.0000000000, propagate_forward_time: 175.9805450439, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4894561768
validation[          , genome  5847] predictions:    6055/  10000 (60.55%), best:    6489/10000 (64.89%), error:     13110.37305, best error:     10733.92480 on epoch:     0, epoch:    4/50, mu: 0.5759163499, learning_rate: 0.0000072321, weight_decay: 0.0003186619

epoch time: 2925.2517089844s, input_fired_time: 40.1382026672, output_fired_time: 10.0120353699, propagate_forward_time: 875.7932128906, propagate_backward_time: 1982.0561523438, weight_update_time: 0.3061430156, other_time: 17.2521972656
epoch time: 180.8859710693s, input_fired_time: 1.9068541527, output_fired_time: 0.0000000000, propagate_forward_time: 174.4178924561, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5612182617
validation[          , genome  5847] predictions:    6035/  10000 (60.35%), best:    6489/10000 (64.89%), error:     13528.32520, best error:     10733.92480 on epoch:     0, epoch:    5/50, mu: 0.5948285460, learning_rate: 0.0000066693, weight_decay: 0.0002956271

epoch time: 4201.0585937500s, input_fired_time: 68.8557968140, output_fired_time: 21.5108089447, propagate_forward_time: 1256.8486328125, propagate_backward_time: 2821.8691406250, weight_update_time: 1.4688066244, other_time: 31.9741210938
epoch time: 292.5651245117s, input_fired_time: 4.6173419952, output_fired_time: 0.0000000000, propagate_forward_time: 277.7705383301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.1772460938
validation[          , genome  5847] predictions:    5850/  10000 (58.50%), best:    6489/10000 (64.89%), error:     13929.33301, best error:     10733.92480 on epoch:     0, epoch:    6/50, mu: 0.6128769517, learning_rate: 0.0000061503, weight_decay: 0.0002742574

epoch time: 4778.7773437500s, input_fired_time: 82.7818145752, output_fired_time: 18.2378005981, propagate_forward_time: 1411.6530761719, propagate_backward_time: 3231.5061035156, weight_update_time: 0.5101912618, other_time: 34.5986328125
epoch time: 298.3717956543s, input_fired_time: 4.4180669785, output_fired_time: 0.0000000000, propagate_forward_time: 290.7444152832, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2093200684
validation[          , genome  5847] predictions:    5757/  10000 (57.57%), best:    6489/10000 (64.89%), error:     14337.44727, best error:     10733.92480 on epoch:     0, epoch:    7/50, mu: 0.6301010251, learning_rate: 0.0000056717, weight_decay: 0.0002544324

epoch time: 4883.0024414063s, input_fired_time: 69.1628494263, output_fired_time: 16.5008773804, propagate_forward_time: 1468.6904296875, propagate_backward_time: 3293.7333984375, weight_update_time: 0.5333642364, other_time: 34.9145507813
epoch time: 300.4184570313s, input_fired_time: 6.4170203209, output_fired_time: 0.0000000000, propagate_forward_time: 288.5813903809, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4200439453
validation[          , genome  5847] predictions:    5756/  10000 (57.56%), best:    6489/10000 (64.89%), error:     14594.07031, best error:     10733.92480 on epoch:     0, epoch:    8/50, mu: 0.6465384364, learning_rate: 0.0000052303, weight_decay: 0.0002360405

epoch time: 4801.9140625000s, input_fired_time: 56.6445083618, output_fired_time: 26.9884891510, propagate_forward_time: 1463.9188232422, propagate_backward_time: 3222.1557617188, weight_update_time: 0.4693752825, other_time: 32.2065429688
epoch time: 297.8380126953s, input_fired_time: 5.2835450172, output_fired_time: 0.0000000000, propagate_forward_time: 285.1167602539, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4377136230
validation[          , genome  5847] predictions:    5544/  10000 (55.44%), best:    6489/10000 (64.89%), error:     14689.31543, best error:     10733.92480 on epoch:     0, epoch:    9/50, mu: 0.6622251272, learning_rate: 0.0000048233, weight_decay: 0.0002189781

epoch time: 4737.7080078125s, input_fired_time: 50.3557891846, output_fired_time: 16.8729476929, propagate_forward_time: 1429.5291748047, propagate_backward_time: 3200.5280761719, weight_update_time: 0.4789570570, other_time: 40.4216308594
epoch time: 300.3468933105s, input_fired_time: 5.6268873215, output_fired_time: 0.0000000000, propagate_forward_time: 291.3509216309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3690795898
validation[          , genome  5847] predictions:    5467/  10000 (54.67%), best:    6489/10000 (64.89%), error:     14957.85547, best error:     10733.92480 on epoch:     0, epoch:   10/50, mu: 0.6771953702, learning_rate: 0.0000044479, weight_decay: 0.0002031491

epoch time: 4712.3203125000s, input_fired_time: 54.1708908081, output_fired_time: 17.7814750671, propagate_forward_time: 1408.7670898438, propagate_backward_time: 3199.5690917969, weight_update_time: 0.4648118913, other_time: 32.0319824219
epoch time: 300.0380249023s, input_fired_time: 3.2969913483, output_fired_time: 0.0000000000, propagate_forward_time: 293.2728576660, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4681701660
validation[          , genome  5847] predictions:    5340/  10000 (53.40%), best:    6489/10000 (64.89%), error:     14995.29590, best error:     10733.92480 on epoch:     0, epoch:   11/50, mu: 0.6914818883, learning_rate: 0.0000041018, weight_decay: 0.0001884642

epoch time: 4701.7607421875s, input_fired_time: 60.6439132690, output_fired_time: 13.6348743439, propagate_forward_time: 1401.6997070313, propagate_backward_time: 3191.0991210938, weight_update_time: 0.4999057055, other_time: 34.6831054688
epoch time: 292.9132690430s, input_fired_time: 1.4797476530, output_fired_time: 0.0000000000, propagate_forward_time: 282.9667053223, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4668273926
validation[          , genome  5847] predictions:    5409/  10000 (54.09%), best:    6489/10000 (64.89%), error:     15085.22852, best error:     10733.92480 on epoch:     0, epoch:   12/50, mu: 0.7051159143, learning_rate: 0.0000037826, weight_decay: 0.0001748409

epoch time: 4742.0063476563s, input_fired_time: 52.5646514893, output_fired_time: 23.4839782715, propagate_forward_time: 1445.3977050781, propagate_backward_time: 3174.3291015625, weight_update_time: 0.5626295805, other_time: 46.2312011719
epoch time: 298.2424316406s, input_fired_time: 1.3630131483, output_fired_time: 0.0000000000, propagate_forward_time: 288.4613037109, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4181213379
validation[          , genome  5847] predictions:    5312/  10000 (53.12%), best:    6489/10000 (64.89%), error:     15030.77832, best error:     10733.92480 on epoch:     0, epoch:   13/50, mu: 0.7181272507, learning_rate: 0.0000034882, weight_decay: 0.0001622023

epoch time: 4708.2670898438s, input_fired_time: 49.0584259033, output_fired_time: 22.8287410736, propagate_forward_time: 1433.3087158203, propagate_backward_time: 3171.8537597656, weight_update_time: 0.4678013921, other_time: 31.2175292969
epoch time: 295.3693237305s, input_fired_time: 3.2870650291, output_fired_time: 0.0000000000, propagate_forward_time: 283.6459960938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.4362487793
validation[          , genome  5847] predictions:    5321/  10000 (53.21%), best:    6489/10000 (64.89%), error:     15192.31348, best error:     10733.92480 on epoch:     0, epoch:   14/50, mu: 0.7305443287, learning_rate: 0.0000032168, weight_decay: 0.0001504774

epoch time: 4738.5771484375s, input_fired_time: 56.9181327820, output_fired_time: 21.3407554626, propagate_forward_time: 1413.3594970703, propagate_backward_time: 3208.2600097656, weight_update_time: 0.4555518925, other_time: 38.6989746094
epoch time: 292.0040283203s, input_fired_time: 8.4231090546, output_fired_time: 0.0000000000, propagate_forward_time: 278.0527648926, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5281677246
validation[          , genome  5847] predictions:    5351/  10000 (53.51%), best:    6489/10000 (64.89%), error:     15204.80762, best error:     10733.92480 on epoch:     0, epoch:   15/50, mu: 0.7423942685, learning_rate: 0.0000029665, weight_decay: 0.0001396000

epoch time: 4720.5488281250s, input_fired_time: 61.4127426147, output_fired_time: 23.6771812439, propagate_forward_time: 1433.9863281250, propagate_backward_time: 3171.2497558594, weight_update_time: 1.5867303610, other_time: 30.2229003906
epoch time: 291.8567504883s, input_fired_time: 4.3166031837, output_fired_time: 0.0000000000, propagate_forward_time: 285.1431884766, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3969726563
validation[          , genome  5847] predictions:    5215/  10000 (52.15%), best:    6489/10000 (64.89%), error:     15202.63672, best error:     10733.92480 on epoch:     0, epoch:   16/50, mu: 0.7537029982, learning_rate: 0.0000027356, weight_decay: 0.0001295088

epoch time: 4754.2099609375s, input_fired_time: 60.8493614197, output_fired_time: 21.8442592621, propagate_forward_time: 1428.5383300781, propagate_backward_time: 3209.7434082031, weight_update_time: 0.4941037297, other_time: 33.2348632813
epoch time: 290.6613159180s, input_fired_time: 1.3564603329, output_fired_time: 0.0000000000, propagate_forward_time: 284.1213073730, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1835632324
validation[          , genome  5847] predictions:    5262/  10000 (52.62%), best:    6489/10000 (64.89%), error:     15142.95996, best error:     10733.92480 on epoch:     0, epoch:   17/50, mu: 0.7644952536, learning_rate: 0.0000025227, weight_decay: 0.0001201472

epoch time: 4697.4306640625s, input_fired_time: 69.8381423950, output_fired_time: 22.3164119720, propagate_forward_time: 1388.2814941406, propagate_backward_time: 3184.4873046875, weight_update_time: 0.4815801382, other_time: 32.5070800781
epoch time: 293.2604980469s, input_fired_time: 4.1517796516, output_fired_time: 0.0000000000, propagate_forward_time: 281.6908569336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.4178466797
validation[          , genome  5847] predictions:    5259/  10000 (52.59%), best:    6489/10000 (64.89%), error:     15102.71387, best error:     10733.92480 on epoch:     0, epoch:   18/50, mu: 0.7747945786, learning_rate: 0.0000023264, weight_decay: 0.0001114622

epoch time: 4664.0009765625s, input_fired_time: 49.2101783752, output_fired_time: 23.0267410278, propagate_forward_time: 1371.7546386719, propagate_backward_time: 3185.9272460938, weight_update_time: 2.5857040882, other_time: 34.0822753906
epoch time: 285.6925354004s, input_fired_time: 1.1297802925, output_fired_time: 0.0000000000, propagate_forward_time: 275.0724487305, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.4902954102
validation[          , genome  5847] predictions:    5332/  10000 (53.32%), best:    6489/10000 (64.89%), error:     15054.26953, best error:     10733.92480 on epoch:     0, epoch:   19/50, mu: 0.7846235037, learning_rate: 0.0000021454, weight_decay: 0.0001034051

epoch time: 4564.7563476563s, input_fired_time: 54.4221038818, output_fired_time: 18.9140567780, propagate_forward_time: 1363.1676025391, propagate_backward_time: 3097.0986328125, weight_update_time: 3.3471965790, other_time: 31.1542968750
epoch time: 285.7680053711s, input_fired_time: 3.4770030975, output_fired_time: 0.0000000000, propagate_forward_time: 280.0323486328, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2586669922
validation[          , genome  5847] predictions:    5184/  10000 (51.84%), best:    6489/10000 (64.89%), error:     15122.89453, best error:     10733.92480 on epoch:     0, epoch:   20/50, mu: 0.7940035462, learning_rate: 0.0000019784, weight_decay: 0.0000959303

epoch time: 4976.1381835938s, input_fired_time: 58.2984962463, output_fired_time: 14.8504114151, propagate_forward_time: 1488.7424316406, propagate_backward_time: 3379.5729980469, weight_update_time: 3.4423208237, other_time: 34.6738281250
epoch time: 286.5705261230s, input_fired_time: 2.3881518841, output_fired_time: 0.0000000000, propagate_forward_time: 277.7893371582, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.3930358887
validation[          , genome  5847] predictions:    5298/  10000 (52.98%), best:    6489/10000 (64.89%), error:     14963.49805, best error:     10733.92480 on epoch:     0, epoch:   21/50, mu: 0.8029551506, learning_rate: 0.0000018245, weight_decay: 0.0000889959

epoch time: 4579.6240234375s, input_fired_time: 54.0570564270, output_fired_time: 21.1276912689, propagate_forward_time: 1385.6020507813, propagate_backward_time: 3082.2863769531, weight_update_time: 1.4801115990, other_time: 36.5505371094
epoch time: 287.5310974121s, input_fired_time: 3.4184730053, output_fired_time: 0.0000000000, propagate_forward_time: 277.6860656738, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.4265441895
validation[          , genome  5847] predictions:    5265/  10000 (52.65%), best:    6489/10000 (64.89%), error:     15133.70020, best error:     10733.92480 on epoch:     0, epoch:   22/50, mu: 0.8114979267, learning_rate: 0.0000016825, weight_decay: 0.0000825627

epoch time: 4574.9536132813s, input_fired_time: 66.7762222290, output_fired_time: 16.1480236053, propagate_forward_time: 1357.1419677734, propagate_backward_time: 3104.5681152344, weight_update_time: 0.4410837889, other_time: 30.3190917969
epoch time: 285.3950805664s, input_fired_time: 5.3297729492, output_fired_time: 0.0000000000, propagate_forward_time: 275.7727661133, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2925415039
validation[          , genome  5847] predictions:    5354/  10000 (53.54%), best:    6489/10000 (64.89%), error:     15024.33887, best error:     10733.92480 on epoch:     0, epoch:   23/50, mu: 0.8196505308, learning_rate: 0.0000015515, weight_decay: 0.0000765946

epoch time: 4544.3408203125s, input_fired_time: 48.7953948975, output_fired_time: 24.6081047058, propagate_forward_time: 1333.0275878906, propagate_backward_time: 3112.2604980469, weight_update_time: 0.4872701466, other_time: 25.6494140625
epoch time: 284.9017333984s, input_fired_time: 2.3686747551, output_fired_time: 0.0000000000, propagate_forward_time: 273.1651916504, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.3678588867
validation[          , genome  5847] predictions:    5344/  10000 (53.44%), best:    6489/10000 (64.89%), error:     15022.56055, best error:     10733.92480 on epoch:     0, epoch:   24/50, mu: 0.8274307847, learning_rate: 0.0000014308, weight_decay: 0.0000710579

epoch time: 4576.0058593750s, input_fired_time: 60.7093582153, output_fired_time: 12.9631538391, propagate_forward_time: 1393.8541259766, propagate_backward_time: 3079.0104980469, weight_update_time: 0.4398173392, other_time: 29.4685058594
epoch time: 280.7026062012s, input_fired_time: 2.3933632374, output_fired_time: 0.0000000000, propagate_forward_time: 269.9099426270, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.3992919922
validation[          , genome  5847] predictions:    5355/  10000 (53.55%), best:    6489/10000 (64.89%), error:     14982.12988, best error:     10733.92480 on epoch:     0, epoch:   25/50, mu: 0.8348556757, learning_rate: 0.0000013195, weight_decay: 0.0000659214

epoch time: 4612.8745117188s, input_fired_time: 47.3870887756, output_fired_time: 18.8415031433, propagate_forward_time: 1422.7869873047, propagate_backward_time: 3090.5637207031, weight_update_time: 0.5447891951, other_time: 33.2951660156
epoch time: 286.9052429199s, input_fired_time: 5.4478392601, output_fired_time: 0.0000000000, propagate_forward_time: 276.2295532227, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2278442383
validation[          , genome  5847] predictions:    5261/  10000 (52.61%), best:    6489/10000 (64.89%), error:     14992.45508, best error:     10733.92480 on epoch:     0, epoch:   26/50, mu: 0.8419414759, learning_rate: 0.0000012168, weight_decay: 0.0000611562

epoch time: 4561.8505859375s, input_fired_time: 53.2683639526, output_fired_time: 26.1681747437, propagate_forward_time: 1384.7724609375, propagate_backward_time: 3061.8776855469, weight_update_time: 1.4098027945, other_time: 35.7639160156
epoch time: 284.4627380371s, input_fired_time: 3.3297560215, output_fired_time: 0.0000000000, propagate_forward_time: 269.8107910156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 11.3222045898
validation[          , genome  5847] predictions:    5297/  10000 (52.97%), best:    6489/10000 (64.89%), error:     14883.02441, best error:     10733.92480 on epoch:     0, epoch:   27/50, mu: 0.8487036824, learning_rate: 0.0000011221, weight_decay: 0.0000567355

epoch time: 4570.5419921875s, input_fired_time: 56.6853218079, output_fired_time: 21.3478622437, propagate_forward_time: 1371.1188964844, propagate_backward_time: 3091.7553710938, weight_update_time: 0.5065907240, other_time: 29.6345214844
epoch time: 282.0367126465s, input_fired_time: 3.1678688526, output_fired_time: 0.0000000000, propagate_forward_time: 271.5321960449, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3366394043
validation[          , genome  5847] predictions:    5300/  10000 (53.00%), best:    6489/10000 (64.89%), error:     14876.87012, best error:     10733.92480 on epoch:     0, epoch:   28/50, mu: 0.8551570177, learning_rate: 0.0000010348, weight_decay: 0.0000526343

epoch time: 4569.5083007813s, input_fired_time: 57.9067649841, output_fired_time: 16.4811286926, propagate_forward_time: 1389.1267089844, propagate_backward_time: 3068.5090332031, weight_update_time: 1.5066857338, other_time: 37.4848632813
epoch time: 285.4176940918s, input_fired_time: 1.2250733376, output_fired_time: 0.0000000000, propagate_forward_time: 277.6228942871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5697326660
validation[          , genome  5847] predictions:    5276/  10000 (52.76%), best:    6489/10000 (64.89%), error:     14799.51465, best error:     10733.92480 on epoch:     0, epoch:   29/50, mu: 0.8613156080, learning_rate: 0.0000009542, weight_decay: 0.0000488296

epoch time: 4577.5302734375s, input_fired_time: 57.0351219177, output_fired_time: 21.0139408112, propagate_forward_time: 1388.8496093750, propagate_backward_time: 3075.9919433594, weight_update_time: 0.4857300520, other_time: 34.6394042969
epoch time: 285.0429382324s, input_fired_time: 3.1780176163, output_fired_time: 0.0000000000, propagate_forward_time: 276.4399414063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4249877930
validation[          , genome  5847] predictions:    5348/  10000 (53.48%), best:    6489/10000 (64.89%), error:     14836.83594, best error:     10733.92480 on epoch:     0, epoch:   30/50, mu: 0.8671929240, learning_rate: 0.0000008800, weight_decay: 0.0000452999

epoch time: 4551.8974609375s, input_fired_time: 62.6323852539, output_fired_time: 16.8542518616, propagate_forward_time: 1348.2534179688, propagate_backward_time: 3097.4294433594, weight_update_time: 0.4365307689, other_time: 26.7277832031
epoch time: 282.8591918945s, input_fired_time: 4.4182615280, output_fired_time: 0.0000000000, propagate_forward_time: 272.1939697266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2469482422
validation[          , genome  5847] predictions:    5399/  10000 (53.99%), best:    6489/10000 (64.89%), error:     14840.48145, best error:     10733.92480 on epoch:     0, epoch:   31/50, mu: 0.8728018403, learning_rate: 0.0000008115, weight_decay: 0.0000420253

epoch time: 4582.2778320313s, input_fired_time: 68.9214401245, output_fired_time: 20.7488994598, propagate_forward_time: 1376.1928710938, propagate_backward_time: 3088.6696777344, weight_update_time: 1.4848957062, other_time: 27.7448730469
epoch time: 287.7859802246s, input_fired_time: 4.4578671455, output_fired_time: 0.0000000000, propagate_forward_time: 275.9374389648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3906860352
validation[          , genome  5847] predictions:    5396/  10000 (53.96%), best:    6489/10000 (64.89%), error:     14768.47754, best error:     10733.92480 on epoch:     0, epoch:   32/50, mu: 0.8781545758, learning_rate: 0.0000007484, weight_decay: 0.0000389875

epoch time: 4550.2680664063s, input_fired_time: 51.5340309143, output_fired_time: 23.6684494019, propagate_forward_time: 1355.5883789063, propagate_backward_time: 3087.9853515625, weight_update_time: 2.5095684528, other_time: 31.4916992188
epoch time: 288.7019042969s, input_fired_time: 4.3411412239, output_fired_time: 0.0000000000, propagate_forward_time: 279.8537597656, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5069885254
validation[          , genome  5847] predictions:    5343/  10000 (53.43%), best:    6489/10000 (64.89%), error:     14683.36914, best error:     10733.92480 on epoch:     0, epoch:   33/50, mu: 0.8832628131, learning_rate: 0.0000006901, weight_decay: 0.0000361692

epoch time: 4569.5263671875s, input_fired_time: 58.5276069641, output_fired_time: 26.0555400848, propagate_forward_time: 1378.5832519531, propagate_backward_time: 3085.4343261719, weight_update_time: 0.4122898281, other_time: 20.9252929688
epoch time: 308.9614562988s, input_fired_time: 3.2930212021, output_fired_time: 0.0000000000, propagate_forward_time: 301.2185668945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4498596191
validation[          , genome  5847] predictions:    5418/  10000 (54.18%), best:    6489/10000 (64.89%), error:     14684.49023, best error:     10733.92480 on epoch:     0, epoch:   34/50, mu: 0.8881377578, learning_rate: 0.0000006364, weight_decay: 0.0000335547

epoch time: 4544.4711914063s, input_fired_time: 55.6359786987, output_fired_time: 27.1333427429, propagate_forward_time: 1409.6271972656, propagate_backward_time: 3025.1572265625, weight_update_time: 0.5466561317, other_time: 26.9177246094
epoch time: 279.7398986816s, input_fired_time: 3.0089650154, output_fired_time: 0.0000000000, propagate_forward_time: 272.2390136719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.4919128418
validation[          , genome  5847] predictions:    5338/  10000 (53.38%), best:    6489/10000 (64.89%), error:     14649.46680, best error:     10733.92480 on epoch:     0, epoch:   35/50, mu: 0.8927900195, learning_rate: 0.0000005869, weight_decay: 0.0000311292

epoch time: 4550.8730468750s, input_fired_time: 55.8973846436, output_fired_time: 16.9917335510, propagate_forward_time: 1344.9306640625, propagate_backward_time: 3095.3181152344, weight_update_time: 0.6595761180, other_time: 37.7351074219
epoch time: 284.1598815918s, input_fired_time: 7.4046282768, output_fired_time: 0.0000000000, propagate_forward_time: 274.2874755859, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4677734375
validation[          , genome  5847] predictions:    5333/  10000 (53.33%), best:    6489/10000 (64.89%), error:     14636.61133, best error:     10733.92480 on epoch:     0, epoch:   36/50, mu: 0.8972298503, learning_rate: 0.0000005412, weight_decay: 0.0000288790

epoch time: 4082.5915527344s, input_fired_time: 54.9345321655, output_fired_time: 12.4662132263, propagate_forward_time: 1211.3636474609, propagate_backward_time: 2772.0046386719, weight_update_time: 0.4353761375, other_time: 31.8225097656
epoch time: 187.4737701416s, input_fired_time: 1.8165667057, output_fired_time: 0.0000000000, propagate_forward_time: 181.9748229980, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6823730469
validation[          , genome  5847] predictions:    5349/  10000 (53.49%), best:    6489/10000 (64.89%), error:     14654.91113, best error:     10733.92480 on epoch:     0, epoch:   37/50, mu: 0.9014669061, learning_rate: 0.0000004991, weight_decay: 0.0000267914

epoch time: 3342.9191894531s, input_fired_time: 42.2672195435, output_fired_time: 19.6193408966, propagate_forward_time: 1010.6602783203, propagate_backward_time: 2249.2097167969, weight_update_time: 0.3296963274, other_time: 21.1623535156
epoch time: 188.7645263672s, input_fired_time: 1.8583012819, output_fired_time: 0.0000000000, propagate_forward_time: 185.2475738525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6586456299
validation[          , genome  5847] predictions:    5389/  10000 (53.89%), best:    6489/10000 (64.89%), error:     14590.37891, best error:     10733.92480 on epoch:     0, epoch:   38/50, mu: 0.9055104256, learning_rate: 0.0000004603, weight_decay: 0.0000248548

epoch time: 3039.0307617188s, input_fired_time: 33.6062431335, output_fired_time: 11.0618524551, propagate_forward_time: 928.6131591797, propagate_backward_time: 2026.6628417969, weight_update_time: 2.2978157997, other_time: 39.0866699219
epoch time: 180.2783660889s, input_fired_time: 2.8375453949, output_fired_time: 0.0000000000, propagate_forward_time: 174.9672393799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4735870361
validation[          , genome  5847] predictions:    5388/  10000 (53.88%), best:    6489/10000 (64.89%), error:     14573.13086, best error:     10733.92480 on epoch:     0, epoch:   39/50, mu: 0.9093692899, learning_rate: 0.0000004244, weight_decay: 0.0000230581

epoch time: 2942.4775390625s, input_fired_time: 25.4645290375, output_fired_time: 13.0333185196, propagate_forward_time: 888.7110595703, propagate_backward_time: 1987.1888427734, weight_update_time: 1.3066066504, other_time: 28.0798339844
epoch time: 180.4031829834s, input_fired_time: 5.8411331177, output_fired_time: 0.0000000000, propagate_forward_time: 173.0490875244, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5129547119
validation[          , genome  5847] predictions:    5380/  10000 (53.80%), best:    6489/10000 (64.89%), error:     14547.77930, best error:     10733.92480 on epoch:     0, epoch:   40/50, mu: 0.9130519032, learning_rate: 0.0000003914, weight_decay: 0.0000213913

epoch time: 2943.3161621094s, input_fired_time: 44.6616935730, output_fired_time: 9.9948377609, propagate_forward_time: 858.2350463867, propagate_backward_time: 2007.1654052734, weight_update_time: 0.3052223623, other_time: 23.2591552734
epoch time: 180.1444549561s, input_fired_time: 1.7200919390, output_fired_time: 0.0000000000, propagate_forward_time: 171.8768768311, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.5474853516
validation[          , genome  5847] predictions:    5353/  10000 (53.53%), best:    6489/10000 (64.89%), error:     14590.25098, best error:     10733.92480 on epoch:     0, epoch:   41/50, mu: 0.9165663123, learning_rate: 0.0000003610, weight_decay: 0.0000198450

epoch time: 2933.5495605469s, input_fired_time: 33.6351661682, output_fired_time: 17.0856399536, propagate_forward_time: 882.3527832031, propagate_backward_time: 1970.0382080078, weight_update_time: 0.3096475303, other_time: 30.4376220703
epoch time: 180.2885437012s, input_fired_time: 1.8324971199, output_fired_time: 0.0000000000, propagate_forward_time: 176.9254760742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5305633545
validation[          , genome  5847] predictions:    5405/  10000 (54.05%), best:    6489/10000 (64.89%), error:     14519.94043, best error:     10733.92480 on epoch:     0, epoch:   42/50, mu: 0.9199202061, learning_rate: 0.0000003329, weight_decay: 0.0000184105

epoch time: 2921.5009765625s, input_fired_time: 35.6322555542, output_fired_time: 15.0150337219, propagate_forward_time: 863.5266113281, propagate_backward_time: 1984.0604248047, weight_update_time: 0.3077262342, other_time: 23.2664794922
epoch time: 180.2589416504s, input_fired_time: 0.8228275180, output_fired_time: 0.0000000000, propagate_forward_time: 175.9041900635, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5319213867
validation[          , genome  5847] predictions:    5387/  10000 (53.87%), best:    6489/10000 (64.89%), error:     14483.31641, best error:     10733.92480 on epoch:     0, epoch:   43/50, mu: 0.9231209159, learning_rate: 0.0000003070, weight_decay: 0.0000170797

epoch time: 2936.7570800781s, input_fired_time: 35.8509979248, output_fired_time: 11.9584016800, propagate_forward_time: 869.0134887695, propagate_backward_time: 2004.8912353516, weight_update_time: 0.3018845916, other_time: 15.0428466797
epoch time: 181.3169403076s, input_fired_time: 2.8338003159, output_fired_time: 0.0000000000, propagate_forward_time: 175.9580383301, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5251007080
validation[          , genome  5847] predictions:    5382/  10000 (53.82%), best:    6489/10000 (64.89%), error:     14509.52930, best error:     10733.92480 on epoch:     0, epoch:   44/50, mu: 0.9261754155, learning_rate: 0.0000002831, weight_decay: 0.0000158451

epoch time: 2928.4191894531s, input_fired_time: 40.5075988770, output_fired_time: 18.0394821167, propagate_forward_time: 856.1378173828, propagate_backward_time: 1988.4931640625, weight_update_time: 2.3136467934, other_time: 25.2410888672
epoch time: 183.0708007813s, input_fired_time: 3.8353018761, output_fired_time: 0.0000000000, propagate_forward_time: 175.7096862793, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5258178711
validation[          , genome  5847] predictions:    5425/  10000 (54.25%), best:    6489/10000 (64.89%), error:     14480.72070, best error:     10733.92480 on epoch:     0, epoch:   45/50, mu: 0.9290904403, learning_rate: 0.0000002610, weight_decay: 0.0000146997

epoch time: 2936.5939941406s, input_fired_time: 34.5998840332, output_fired_time: 12.9414730072, propagate_forward_time: 865.2348632813, propagate_backward_time: 1996.6711425781, weight_update_time: 0.3067850769, other_time: 27.1467285156
epoch time: 180.3693237305s, input_fired_time: 0.8204151988, output_fired_time: 0.0000000000, propagate_forward_time: 176.0186309814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.5302734375
validation[          , genome  5847] predictions:    5416/  10000 (54.16%), best:    6489/10000 (64.89%), error:     14471.37695, best error:     10733.92480 on epoch:     0, epoch:   46/50, mu: 0.9318723083, learning_rate: 0.0000002407, weight_decay: 0.0000136371

epoch time: 3028.4350585938s, input_fired_time: 29.7971878052, output_fired_time: 18.2070827484, propagate_forward_time: 906.0659179688, propagate_backward_time: 2049.6130371094, weight_update_time: 0.3292121589, other_time: 24.7519531250
epoch time: 206.7396545410s, input_fired_time: 0.9638282657, output_fired_time: 0.0000000000, propagate_forward_time: 201.8601684570, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9156646729
validation[          , genome  5847] predictions:    5395/  10000 (53.95%), best:    6489/10000 (64.89%), error:     14452.19336, best error:     10733.92480 on epoch:     0, epoch:   47/50, mu: 0.9345271587, learning_rate: 0.0000002220, weight_decay: 0.0000126513

epoch time: 3712.0341796875s, input_fired_time: 45.6550178528, output_fired_time: 23.3001518250, propagate_forward_time: 1107.2308349609, propagate_backward_time: 2502.7119140625, weight_update_time: 1.4312620163, other_time: 33.1362304688
epoch time: 203.4792785645s, input_fired_time: 1.9584513903, output_fired_time: 0.0000000000, propagate_forward_time: 199.5574340820, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9633941650
validation[          , genome  5847] predictions:    5404/  10000 (54.04%), best:    6489/10000 (64.89%), error:     14459.89746, best error:     10733.92480 on epoch:     0, epoch:   48/50, mu: 0.9370607138, learning_rate: 0.0000002047, weight_decay: 0.0000117368

epoch time: 2971.7495117188s, input_fired_time: 37.6552238464, output_fired_time: 15.0199451447, propagate_forward_time: 895.8428344727, propagate_backward_time: 2003.9116210938, weight_update_time: 2.3254811764, other_time: 19.3198242188
epoch time: 180.7515258789s, input_fired_time: 1.8298368454, output_fired_time: 0.0000000000, propagate_forward_time: 176.4286956787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4929962158
validation[          , genome  5847] predictions:    5430/  10000 (54.30%), best:    6489/10000 (64.89%), error:     14461.13477, best error:     10733.92480 on epoch:     0, epoch:   49/50, mu: 0.9394785762, learning_rate: 0.0000001888, weight_decay: 0.0000108884

epoch time: 2882.4868164063s, input_fired_time: 43.4903068542, output_fired_time: 11.7989387512, propagate_forward_time: 850.8695068359, propagate_backward_time: 1950.3995361328, weight_update_time: 0.2977538109, other_time: 25.9287109375
epoch time: 169.1252288818s, input_fired_time: 0.9057996869, output_fired_time: 0.0000000000, propagate_forward_time: 162.6955261230, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5239105225
validation[          , genome  5847] predictions:    5416/  10000 (54.16%), best:    6489/10000 (64.89%), error:     14411.49219, best error:     10733.92480 on epoch:     0, epoch:   50/50, mu: 0.9417859912, learning_rate: 0.0000001741, weight_decay: 0.0000101013

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 879.3339843750s, input_fired_time: 8.4548406601, output_fired_time: 0.0000000000, propagate_forward_time: 849.3800659180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 21.4990844727
best training[          , genome  5847] predictions:   34827/  50000 (69.65%), best:    6489/10000 (64.89%), error:     49124.15234, best error:     10733.92480 on epoch:     0, epoch:   51/50, mu: 0.9439880252, learning_rate: 0.0000001606, weight_decay: 0.0000093712
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 176.6903381348s, input_fired_time: 2.9054167271, output_fired_time: 0.0000000000, propagate_forward_time: 163.8142547607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 9.9706726074
   testing[          , genome  5847] predictions:    6489/  10000 (64.89%), best:    6489/10000 (64.89%), error:     10733.92480, best error:     10733.92480 on epoch:     0, epoch:   51/50, mu: 0.9439880252, learning_rate: 0.0000001606, weight_decay: 0.0000093712
backpropagation finished successfully!
21:05:50 (8988): called boinc_finish(0)

</stderr_txt>
]]>