Nameexact_genome_1542692703_181_6406_2
Workunit4100896
Created27 Nov 2018, 21:42:00 UTC
Sent27 Nov 2018, 21:42:22 UTC
Report deadline17 Jan 2019, 11:21:39 UTC
Received28 Nov 2018, 13:05:56 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID73073
Run time12 hours 58 min 54 sec
CPU time11 hours 49 min 42 sec
Validate stateValid
Credit12,642.24
Device peak FLOPS4.61 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size548.39 MB
Peak swap size568.00 MB
Peak disk usage11.50 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_181_6406.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542692703_181_6406_2_r1394303412_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542692703_181_6406.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.484244
read mu: 0.484244
read mu_delta: 0.941689
read initial_learning_rate: 0.000360535
read learning_rate: 0.000360535
read learning_rate_delta: 0.920237
read initial_weight_decay: 0.000469629
read weight_decay: 0.000469629
read weight_decay_delta: 0.932314
read batch_size: 104
read epsilon: 1e-07
read alpha: 0.0652085
read input_dropout_probability: 0.00129196
read hidden_dropout_probability: 0.108281
read velocity_reset: 1272
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6406
read normal distribution: '0 1.99951684474945 -2.0102264881134'
generator_str: '1878135350'
read generator: 1878135350
read generated_by_map:
1 split_node 1
reading 87 nodes.
reading 332 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2106183844
post shuffle 1: 1679489302
epoch time: 853.123229980469s, input_fired_time: 12.0131282806396, output_fired_time: 4.25073385238647, propagate_forward_time: 315.242614746094, propagate_backward_time: 511.551025390625, weight_update_time: 0.241158649325371, other_time: 10.0657348632813
epoch time: 66.8407211303711s, input_fired_time: 0.706640422344208, output_fired_time: 0, propagate_forward_time: 64.5986633300781, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.53541564941406
validation[          , genome  6406] predictions:    5765/  10000 (57.65%), best:    5765/10000 (57.65%), error:     12220.08398, best error:     12220.08398 on epoch:     0, epoch:    0/50, mu: 0.4842438400, learning_rate: 0.0003605348, weight_decay: 0.0004696290

epoch time: 856.6805419922s, input_fired_time: 11.9421014786, output_fired_time: 5.2643132210, propagate_forward_time: 320.4551086426, propagate_backward_time: 508.9110717773, weight_update_time: 0.2385219485, other_time: 10.1079711914
epoch time: 67.5839767456s, input_fired_time: 0.7388097644, output_fired_time: 0.0000000000, propagate_forward_time: 65.3557281494, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4894409180
validation[          , genome  6406] predictions:    5802/  10000 (58.02%), best:    5765/10000 (57.65%), error:     12306.57422, best error:     12220.08398 on epoch:     0, epoch:    1/50, mu: 0.5137347579, learning_rate: 0.0003317773, weight_decay: 0.0004378417

epoch time: 859.8786010742s, input_fired_time: 12.9546909332, output_fired_time: 4.3201341629, propagate_forward_time: 320.3789062500, propagate_backward_time: 513.0053710938, weight_update_time: 0.2336951792, other_time: 9.2194824219
epoch time: 67.9443206787s, input_fired_time: 0.7287709117, output_fired_time: 0.0000000000, propagate_forward_time: 65.6294860840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5860671997
validation[          , genome  6406] predictions:    5752/  10000 (57.52%), best:    5765/10000 (57.65%), error:     12500.55273, best error:     12220.08398 on epoch:     0, epoch:    2/50, mu: 0.5415060520, learning_rate: 0.0003053136, weight_decay: 0.0004082060

epoch time: 870.7886962891s, input_fired_time: 13.1511974335, output_fired_time: 4.3580160141, propagate_forward_time: 323.1827392578, propagate_backward_time: 520.6300048828, weight_update_time: 0.2388306260, other_time: 9.4667358398
epoch time: 67.8998260498s, input_fired_time: 0.7022747397, output_fired_time: 0.0000000000, propagate_forward_time: 65.5430221558, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6545257568
validation[          , genome  6406] predictions:    5675/  10000 (56.75%), best:    5765/10000 (57.65%), error:     12626.22461, best error:     12220.08398 on epoch:     0, epoch:    3/50, mu: 0.5676579475, learning_rate: 0.0002809607, weight_decay: 0.0003805762

epoch time: 876.7676391602s, input_fired_time: 13.0104103088, output_fired_time: 4.3615398407, propagate_forward_time: 321.6812133789, propagate_backward_time: 527.2019042969, weight_update_time: 0.2604330480, other_time: 10.5125732422
epoch time: 67.7828521729s, input_fired_time: 0.7363234162, output_fired_time: 0.0000000000, propagate_forward_time: 65.4239501953, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6225814819
validation[          , genome  6406] predictions:    5730/  10000 (57.30%), best:    5765/10000 (57.65%), error:     12628.31445, best error:     12220.08398 on epoch:     0, epoch:    4/50, mu: 0.5922849178, learning_rate: 0.0002585503, weight_decay: 0.0003548165

epoch time: 876.8150634766s, input_fired_time: 12.1817321777, output_fired_time: 4.4284696579, propagate_forward_time: 325.6394653320, propagate_backward_time: 524.9716796875, weight_update_time: 0.2684453428, other_time: 9.5936889648
epoch time: 67.9474029541s, input_fired_time: 0.7290588021, output_fired_time: 0.0000000000, propagate_forward_time: 65.6409149170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5774307251
validation[          , genome  6406] predictions:    5770/  10000 (57.70%), best:    5765/10000 (57.65%), error:     12418.26270, best error:     12220.08398 on epoch:     0, epoch:    5/50, mu: 0.6154758930, learning_rate: 0.0002379275, weight_decay: 0.0003308004

epoch time: 877.0841064453s, input_fired_time: 13.2331314087, output_fired_time: 4.4672055244, propagate_forward_time: 327.7218017578, propagate_backward_time: 521.9479980469, weight_update_time: 0.2401928157, other_time: 9.7139282227
epoch time: 68.1264877319s, input_fired_time: 0.7256131172, output_fired_time: 0.0000000000, propagate_forward_time: 65.7664031982, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6344680786
validation[          , genome  6406] predictions:    5746/  10000 (57.46%), best:    5765/10000 (57.65%), error:     12677.47070, best error:     12220.08398 on epoch:     0, epoch:    6/50, mu: 0.6373145580, learning_rate: 0.0002189496, weight_decay: 0.0003084099

epoch time: 867.4445190430s, input_fired_time: 12.9569311142, output_fired_time: 4.3347373009, propagate_forward_time: 327.7776794434, propagate_backward_time: 511.9413452148, weight_update_time: 0.2609933019, other_time: 10.4338989258
epoch time: 65.8387451172s, input_fired_time: 0.6956343055, output_fired_time: 0.0000000000, propagate_forward_time: 63.5810775757, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5620346069
validation[          , genome  6406] predictions:    5560/  10000 (55.60%), best:    5765/10000 (57.65%), error:     12819.50000, best error:     12220.08398 on epoch:     0, epoch:    7/50, mu: 0.6578798294, learning_rate: 0.0002014854, weight_decay: 0.0002875348

epoch time: 871.8862915039s, input_fired_time: 13.0221138000, output_fired_time: 4.3883428574, propagate_forward_time: 315.4054565430, propagate_backward_time: 527.6217651367, weight_update_time: 0.2383989692, other_time: 11.4486083984
epoch time: 68.4270401001s, input_fired_time: 0.7065957189, output_fired_time: 0.0000000000, propagate_forward_time: 66.1421737671, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5782699585
validation[          , genome  6406] predictions:    5585/  10000 (55.85%), best:    5765/10000 (57.65%), error:     13041.54004, best error:     12220.08398 on epoch:     0, epoch:    8/50, mu: 0.6772459149, learning_rate: 0.0001854142, weight_decay: 0.0002680728

epoch time: 866.8424072266s, input_fired_time: 11.8902721405, output_fired_time: 4.2647304535, propagate_forward_time: 322.0422668457, propagate_backward_time: 519.2651367188, weight_update_time: 0.2497626096, other_time: 9.3800048828
epoch time: 66.5749740601s, input_fired_time: 0.7148764133, output_fired_time: 0.0000000000, propagate_forward_time: 64.3195037842, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5405960083
validation[          , genome  6406] predictions:    5510/  10000 (55.10%), best:    5765/10000 (57.65%), error:     12924.58691, best error:     12220.08398 on epoch:     0, epoch:    9/50, mu: 0.6954827905, learning_rate: 0.0001706250, weight_decay: 0.0002499280

epoch time: 866.7213134766s, input_fired_time: 12.0492334366, output_fired_time: 4.3601646423, propagate_forward_time: 318.9794311523, propagate_backward_time: 521.9231567383, weight_update_time: 0.2529663742, other_time: 9.4093017578
epoch time: 69.4583358765s, input_fired_time: 0.6886652708, output_fired_time: 0.0000000000, propagate_forward_time: 66.1451873779, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6244812012
validation[          , genome  6406] predictions:    5673/  10000 (56.73%), best:    5765/10000 (57.65%), error:     12795.54980, best error:     12220.08398 on epoch:     0, epoch:   10/50, mu: 0.7126562595, learning_rate: 0.0001570153, weight_decay: 0.0002330114

epoch time: 873.7493286133s, input_fired_time: 12.0041379929, output_fired_time: 4.3596210480, propagate_forward_time: 336.3465881348, propagate_backward_time: 511.6560668945, weight_update_time: 0.2367896736, other_time: 9.3828735352
epoch time: 67.4837875366s, input_fired_time: 0.6973894835, output_fired_time: 0.0000000000, propagate_forward_time: 65.2033538818, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5830459595
validation[          , genome  6406] predictions:    5690/  10000 (56.90%), best:    5765/10000 (57.65%), error:     12481.20703, best error:     12220.08398 on epoch:     0, epoch:   11/50, mu: 0.7288283110, learning_rate: 0.0001444913, weight_decay: 0.0002172398

epoch time: 865.7032470703s, input_fired_time: 13.9812870026, output_fired_time: 4.3550524712, propagate_forward_time: 321.7045593262, propagate_backward_time: 515.2513427734, weight_update_time: 0.2452827245, other_time: 10.4110107422
epoch time: 68.1387786865s, input_fired_time: 0.7308627963, output_fired_time: 0.0000000000, propagate_forward_time: 65.8794784546, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5284347534
validation[          , genome  6406] predictions:    5487/  10000 (54.87%), best:    5765/10000 (57.65%), error:     12855.69824, best error:     12220.08398 on epoch:     0, epoch:   12/50, mu: 0.7440573573, learning_rate: 0.0001329661, weight_decay: 0.0002025357

epoch time: 864.5324096680s, input_fired_time: 12.0143985748, output_fired_time: 5.3302965164, propagate_forward_time: 319.8347473145, propagate_backward_time: 517.9331054688, weight_update_time: 0.2376006395, other_time: 9.4197998047
epoch time: 67.2124252319s, input_fired_time: 0.6949513555, output_fired_time: 0.0000000000, propagate_forward_time: 64.9750366211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5424346924
validation[          , genome  6406] predictions:    5741/  10000 (57.41%), best:    5765/10000 (57.65%), error:     12295.34375, best error:     12220.08398 on epoch:     0, epoch:   13/50, mu: 0.7583984137, learning_rate: 0.0001223603, weight_decay: 0.0001888269

epoch time: 866.0385131836s, input_fired_time: 11.9768943787, output_fired_time: 4.2919001579, propagate_forward_time: 317.3054199219, propagate_backward_time: 522.9606323242, weight_update_time: 0.2425243109, other_time: 9.5037231445
epoch time: 67.5490417480s, input_fired_time: 0.7317263484, output_fired_time: 0.0000000000, propagate_forward_time: 65.2557296753, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5615844727
validation[          , genome  6406] predictions:    5732/  10000 (57.32%), best:    5765/10000 (57.65%), error:     12442.33691, best error:     12220.08398 on epoch:     0, epoch:   14/50, mu: 0.7719032168, learning_rate: 0.0001126004, weight_decay: 0.0001760460

epoch time: 867.5297241211s, input_fired_time: 12.9706077576, output_fired_time: 4.3094758987, propagate_forward_time: 335.1074523926, propagate_backward_time: 505.6793212891, weight_update_time: 0.2422762811, other_time: 9.4628906250
epoch time: 68.1228713989s, input_fired_time: 0.7234933972, output_fired_time: 0.0000000000, propagate_forward_time: 64.7501754761, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6492004395
validation[          , genome  6406] predictions:    5471/  10000 (54.71%), best:    5765/10000 (57.65%), error:     13164.71680, best error:     12220.08398 on epoch:     0, epoch:   15/50, mu: 0.7846205235, learning_rate: 0.0001036190, weight_decay: 0.0001641301

epoch time: 865.4899291992s, input_fired_time: 11.9866304398, output_fired_time: 4.3331575394, propagate_forward_time: 319.9334411621, propagate_backward_time: 519.7832031250, weight_update_time: 0.2721338868, other_time: 9.4534912109
epoch time: 66.5844955444s, input_fired_time: 0.6757590771, output_fired_time: 0.0000000000, propagate_forward_time: 64.3694915771, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5392456055
validation[          , genome  6406] predictions:    5729/  10000 (57.29%), best:    5765/10000 (57.65%), error:     12401.20801, best error:     12220.08398 on epoch:     0, epoch:   16/50, mu: 0.7965962887, learning_rate: 0.0000953540, weight_decay: 0.0001530208

epoch time: 865.9896240234s, input_fired_time: 11.9875926971, output_fired_time: 5.3728313446, propagate_forward_time: 330.1983337402, propagate_backward_time: 509.1492614746, weight_update_time: 0.2383597344, other_time: 9.2816467285
epoch time: 67.6859893799s, input_fired_time: 0.7045389414, output_fired_time: 0.0000000000, propagate_forward_time: 65.4062500000, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5752029419
validation[          , genome  6406] predictions:    5651/  10000 (56.51%), best:    5765/10000 (57.65%), error:     12392.26660, best error:     12220.08398 on epoch:     0, epoch:   17/50, mu: 0.8078737855, learning_rate: 0.0000877483, weight_decay: 0.0001426635

epoch time: 881.7865600586s, input_fired_time: 13.1713352203, output_fired_time: 4.3788533211, propagate_forward_time: 327.0173950195, propagate_backward_time: 527.8206176758, weight_update_time: 0.2419424355, other_time: 9.3983764648
epoch time: 67.7205276489s, input_fired_time: 0.6861249208, output_fired_time: 0.0000000000, propagate_forward_time: 65.5208053589, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5135955811
validation[          , genome  6406] predictions:    5779/  10000 (57.79%), best:    5765/10000 (57.65%), error:     12281.34473, best error:     12220.08398 on epoch:     0, epoch:   18/50, mu: 0.8184936643, learning_rate: 0.0000807492, weight_decay: 0.0001330071

epoch time: 926.5879516602s, input_fired_time: 13.7650241852, output_fired_time: 4.7145256996, propagate_forward_time: 341.9242858887, propagate_backward_time: 555.0078125000, weight_update_time: 0.2901869118, other_time: 11.1762695313
epoch time: 69.8618011475s, input_fired_time: 0.7481166124, output_fired_time: 0.0000000000, propagate_forward_time: 67.5359191895, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5777664185
validation[          , genome  6406] predictions:    5851/  10000 (58.51%), best:    5765/10000 (57.65%), error:     12302.92090, best error:     12220.08398 on epoch:     0, epoch:   19/50, mu: 0.8284943104, learning_rate: 0.0000743083, weight_decay: 0.0001240044

epoch time: 848.1053466797s, input_fired_time: 11.7981939316, output_fired_time: 4.2416667938, propagate_forward_time: 319.1011352539, propagate_backward_time: 500.8614196777, weight_update_time: 0.2296870798, other_time: 12.1029357910
epoch time: 63.2354850769s, input_fired_time: 0.6508181095, output_fired_time: 0.0000000000, propagate_forward_time: 61.1429939270, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4416732788
validation[          , genome  6406] predictions:    5803/  10000 (58.03%), best:    5803/10000 (58.03%), error:     12174.43262, best error:     12174.43262 on epoch:    20, epoch:   20/50, mu: 0.8379117846, learning_rate: 0.0000683813, weight_decay: 0.0001156111

epoch time: 815.1096191406s, input_fired_time: 11.2778253555, output_fired_time: 4.0258007050, propagate_forward_time: 302.1724853516, propagate_backward_time: 488.8392028809, weight_update_time: 0.2551163137, other_time: 8.7942810059
epoch time: 63.5987319946s, input_fired_time: 0.6569131613, output_fired_time: 0.0000000000, propagate_forward_time: 61.4965820313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4452362061
validation[          , genome  6406] predictions:    5816/  10000 (58.16%), best:    5816/10000 (58.16%), error:     12137.50195, best error:     12137.50195 on epoch:    21, epoch:   21/50, mu: 0.8467801213, learning_rate: 0.0000629269, weight_decay: 0.0001077858

epoch time: 813.6585693359s, input_fired_time: 12.3154382706, output_fired_time: 4.0375194550, propagate_forward_time: 301.9045104980, propagate_backward_time: 486.7092285156, weight_update_time: 0.2335209548, other_time: 8.6918640137
epoch time: 63.1131210327s, input_fired_time: 0.6540338993, output_fired_time: 0.0000000000, propagate_forward_time: 60.9860877991, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4729995728
validation[          , genome  6406] predictions:    5728/  10000 (57.28%), best:    5816/10000 (58.16%), error:     12265.09277, best error:     12137.50195 on epoch:    21, epoch:   22/50, mu: 0.8551313281, learning_rate: 0.0000579077, weight_decay: 0.0001004902

epoch time: 837.4621582031s, input_fired_time: 12.6941108704, output_fired_time: 4.1683397293, propagate_forward_time: 308.2651977539, propagate_backward_time: 503.4258728027, weight_update_time: 0.2272874117, other_time: 8.9086608887
epoch time: 67.4210281372s, input_fired_time: 0.7292460799, output_fired_time: 0.0000000000, propagate_forward_time: 64.1500854492, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5416946411
validation[          , genome  6406] predictions:    5909/  10000 (59.09%), best:    5909/10000 (59.09%), error:     11906.79785, best error:     11906.79785 on epoch:    23, epoch:   23/50, mu: 0.8629955649, learning_rate: 0.0000532888, weight_decay: 0.0000936885

epoch time: 886.4916992188s, input_fired_time: 13.2528848648, output_fired_time: 4.4393033981, propagate_forward_time: 339.6161804199, propagate_backward_time: 518.7440795898, weight_update_time: 0.2492844909, other_time: 10.4392700195
epoch time: 67.7608184814s, input_fired_time: 0.7063348293, output_fired_time: 0.0000000000, propagate_forward_time: 65.5114364624, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5430450439
validation[          , genome  6406] predictions:    5801/  10000 (58.01%), best:    5909/10000 (59.09%), error:     12243.45117, best error:     11906.79785 on epoch:    23, epoch:   24/50, mu: 0.8704012632, learning_rate: 0.0000490383, weight_decay: 0.0000873471

epoch time: 868.2563476563s, input_fired_time: 12.8697423935, output_fired_time: 4.3009576797, propagate_forward_time: 325.6746215820, propagate_backward_time: 515.0560302734, weight_update_time: 0.2428790778, other_time: 10.3549804688
epoch time: 68.7586212158s, input_fired_time: 0.6937460899, output_fired_time: 0.0000000000, propagate_forward_time: 66.5152969360, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5495758057
validation[          , genome  6406] predictions:    5877/  10000 (58.77%), best:    5909/10000 (59.09%), error:     12010.14648, best error:     11906.79785 on epoch:    23, epoch:   25/50, mu: 0.8773751259, learning_rate: 0.0000451268, weight_decay: 0.0000814349

epoch time: 870.9136962891s, input_fired_time: 11.9869699478, output_fired_time: 4.2323703766, propagate_forward_time: 321.3453369141, propagate_backward_time: 523.1169433594, weight_update_time: 0.2392029017, other_time: 10.2320556641
epoch time: 69.0954666138s, input_fired_time: 1.7384203672, output_fired_time: 0.0000000000, propagate_forward_time: 65.8177108765, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5393371582
validation[          , genome  6406] predictions:    5886/  10000 (58.86%), best:    5909/10000 (59.09%), error:     12127.30664, best error:     11906.79785 on epoch:    23, epoch:   26/50, mu: 0.8839423656, learning_rate: 0.0000415273, weight_decay: 0.0000759229

epoch time: 869.9216918945s, input_fired_time: 13.1892023087, output_fired_time: 4.3365640640, propagate_forward_time: 323.0801391602, propagate_backward_time: 520.0003051758, weight_update_time: 0.2470301837, other_time: 9.3154907227
epoch time: 68.2328186035s, input_fired_time: 0.6963911653, output_fired_time: 0.0000000000, propagate_forward_time: 66.0130691528, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5233612061
validation[          , genome  6406] predictions:    5957/  10000 (59.57%), best:    5957/10000 (59.57%), error:     11878.35352, best error:     11878.35352 on epoch:    27, epoch:   27/50, mu: 0.8901266456, learning_rate: 0.0000382150, weight_decay: 0.0000707840

epoch time: 871.9514160156s, input_fired_time: 12.1548719406, output_fired_time: 4.4358344078, propagate_forward_time: 323.0663146973, propagate_backward_time: 522.8563842773, weight_update_time: 0.2433539927, other_time: 9.4380493164
epoch time: 68.8191604614s, input_fired_time: 0.6920797229, output_fired_time: 0.0000000000, propagate_forward_time: 66.5719299316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5551528931
validation[          , genome  6406] predictions:    5783/  10000 (57.83%), best:    5957/10000 (59.57%), error:     12123.41602, best error:     11878.35352 on epoch:    27, epoch:   28/50, mu: 0.8959503174, learning_rate: 0.0000351668, weight_decay: 0.0000659929

epoch time: 879.5666503906s, input_fired_time: 12.3056125641, output_fired_time: 4.3693542480, propagate_forward_time: 328.9938354492, propagate_backward_time: 522.2836914063, weight_update_time: 0.2448059469, other_time: 11.6141967773
epoch time: 68.6296920776s, input_fired_time: 0.7093479633, output_fired_time: 0.0000000000, propagate_forward_time: 66.2903442383, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6299972534
validation[          , genome  6406] predictions:    5992/  10000 (59.92%), best:    5992/10000 (59.92%), error:     11848.68555, best error:     11848.68555 on epoch:    29, epoch:   29/50, mu: 0.9014344215, learning_rate: 0.0000323618, weight_decay: 0.0000615261

epoch time: 878.2872924805s, input_fired_time: 13.2477712631, output_fired_time: 5.3718814850, propagate_forward_time: 331.9983825684, propagate_backward_time: 518.0717773438, weight_update_time: 0.2495890707, other_time: 9.5975341797
epoch time: 69.5505065918s, input_fired_time: 0.7172446847, output_fired_time: 0.0000000000, propagate_forward_time: 67.2322692871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6009902954
validation[          , genome  6406] predictions:    5937/  10000 (59.37%), best:    5992/10000 (59.92%), error:     11871.83496, best error:     11848.68555 on epoch:    29, epoch:   30/50, mu: 0.9065987468, learning_rate: 0.0000297805, weight_decay: 0.0000573617

epoch time: 879.4503784180s, input_fired_time: 12.1708459854, output_fired_time: 4.4268956184, propagate_forward_time: 331.0534362793, propagate_backward_time: 522.1608886719, weight_update_time: 0.2635233402, other_time: 9.6383056641
epoch time: 68.2188796997s, input_fired_time: 0.7432752252, output_fired_time: 0.0000000000, propagate_forward_time: 65.8848724365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5907287598
validation[          , genome  6406] predictions:    6001/  10000 (60.01%), best:    6001/10000 (60.01%), error:     11768.46484, best error:     11768.46484 on epoch:    31, epoch:   31/50, mu: 0.9114618897, learning_rate: 0.0000274051, weight_decay: 0.0000534791

epoch time: 877.1713867188s, input_fired_time: 13.2402896881, output_fired_time: 4.3935289383, propagate_forward_time: 324.2610473633, propagate_backward_time: 525.5993041992, weight_update_time: 0.2430302054, other_time: 9.6771850586
epoch time: 66.9161605835s, input_fired_time: 0.7342879772, output_fired_time: 0.0000000000, propagate_forward_time: 64.5679168701, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6139526367
validation[          , genome  6406] predictions:    5936/  10000 (59.36%), best:    5936/10000 (59.36%), error:     11756.26172, best error:     11756.26172 on epoch:    32, epoch:   32/50, mu: 0.9160414934, learning_rate: 0.0000252192, weight_decay: 0.0000498593

epoch time: 870.8050537109s, input_fired_time: 15.1290693283, output_fired_time: 4.3313689232, propagate_forward_time: 316.1510314941, propagate_backward_time: 524.5231933594, weight_update_time: 0.2522137165, other_time: 10.6704101563
epoch time: 67.7936401367s, input_fired_time: 0.7263423800, output_fired_time: 0.0000000000, propagate_forward_time: 65.4152145386, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6520843506
validation[          , genome  6406] predictions:    6038/  10000 (60.38%), best:    6038/10000 (60.38%), error:     11694.87109, best error:     11694.87109 on epoch:    33, epoch:   33/50, mu: 0.9203540683, learning_rate: 0.0000232076, weight_decay: 0.0000464845

epoch time: 873.0779418945s, input_fired_time: 12.1545667648, output_fired_time: 4.3919739723, propagate_forward_time: 314.1193237305, propagate_backward_time: 532.7780761719, weight_update_time: 0.2527990639, other_time: 9.6340332031
epoch time: 65.9489669800s, input_fired_time: 0.6993155479, output_fired_time: 0.0000000000, propagate_forward_time: 63.6465759277, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6030731201
validation[          , genome  6406] predictions:    6044/  10000 (60.44%), best:    6044/10000 (60.44%), error:     11655.33105, best error:     11655.33105 on epoch:    34, epoch:   34/50, mu: 0.9244151711, learning_rate: 0.0000213565, weight_decay: 0.0000433382

epoch time: 870.7672729492s, input_fired_time: 13.1339702606, output_fired_time: 4.3223748207, propagate_forward_time: 341.6847229004, propagate_backward_time: 502.0054931641, weight_update_time: 0.2454993874, other_time: 9.6206970215
epoch time: 68.1174011230s, input_fired_time: 0.7380695939, output_fired_time: 0.0000000000, propagate_forward_time: 65.7069091797, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6724243164
validation[          , genome  6406] predictions:    5867/  10000 (58.67%), best:    6044/10000 (60.44%), error:     11932.13770, best error:     11655.33105 on epoch:    34, epoch:   35/50, mu: 0.9282394648, learning_rate: 0.0000196530, weight_decay: 0.0000404048

epoch time: 890.4636230469s, input_fired_time: 13.5932092667, output_fired_time: 5.4575595856, propagate_forward_time: 323.7208251953, propagate_backward_time: 537.0003051758, weight_update_time: 0.2696720064, other_time: 10.6917114258
epoch time: 67.6613540649s, input_fired_time: 0.7261649966, output_fired_time: 0.0000000000, propagate_forward_time: 65.3684310913, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5667572021
validation[          , genome  6406] predictions:    6031/  10000 (60.31%), best:    6031/10000 (60.31%), error:     11651.73047, best error:     11651.73047 on epoch:    36, epoch:   36/50, mu: 0.9318407774, learning_rate: 0.0000180854, weight_decay: 0.0000376700

epoch time: 876.1549072266s, input_fired_time: 12.2496223450, output_fired_time: 4.4010424614, propagate_forward_time: 331.7751159668, propagate_backward_time: 518.1525268555, weight_update_time: 0.2557992041, other_time: 9.5765991211
epoch time: 67.7227478027s, input_fired_time: 0.6874221563, output_fired_time: 0.0000000000, propagate_forward_time: 65.4663467407, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5689773560
validation[          , genome  6406] predictions:    6036/  10000 (60.36%), best:    6031/10000 (60.31%), error:     11661.78125, best error:     11651.73047 on epoch:    36, epoch:   37/50, mu: 0.9352320433, learning_rate: 0.0000166429, weight_decay: 0.0000351202

epoch time: 873.8929443359s, input_fired_time: 14.1485919952, output_fired_time: 4.3389883041, propagate_forward_time: 315.7174987793, propagate_backward_time: 530.0720825195, weight_update_time: 0.2565596998, other_time: 9.6157836914
epoch time: 66.4438095093s, input_fired_time: 0.6837170720, output_fired_time: 0.0000000000, propagate_forward_time: 64.1511383057, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6089553833
validation[          , genome  6406] predictions:    5952/  10000 (59.52%), best:    6031/10000 (60.31%), error:     11814.76660, best error:     11651.73047 on epoch:    36, epoch:   38/50, mu: 0.9384256005, learning_rate: 0.0000153154, weight_decay: 0.0000327431

epoch time: 870.2084960938s, input_fired_time: 12.1699666977, output_fired_time: 5.2691488266, propagate_forward_time: 321.3085021973, propagate_backward_time: 521.7388305664, weight_update_time: 0.2479897887, other_time: 9.7219848633
epoch time: 67.6775360107s, input_fired_time: 0.7267706990, output_fired_time: 0.0000000000, propagate_forward_time: 65.3301086426, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6206588745
validation[          , genome  6406] predictions:    6047/  10000 (60.47%), best:    6047/10000 (60.47%), error:     11648.89941, best error:     11648.89941 on epoch:    39, epoch:   39/50, mu: 0.9414329529, learning_rate: 0.0000140938, weight_decay: 0.0000305268

epoch time: 873.2320556641s, input_fired_time: 14.1739883423, output_fired_time: 4.4459710121, propagate_forward_time: 318.2844543457, propagate_backward_time: 526.7091674805, weight_update_time: 0.2417896241, other_time: 9.6184692383
epoch time: 67.7649078369s, input_fired_time: 0.7013348937, output_fired_time: 0.0000000000, propagate_forward_time: 65.4305801392, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6329956055
validation[          , genome  6406] predictions:    6049/  10000 (60.49%), best:    6049/10000 (60.49%), error:     11575.63965, best error:     11575.63965 on epoch:    40, epoch:   40/50, mu: 0.9442649484, learning_rate: 0.0000129696, weight_decay: 0.0000284606

epoch time: 870.3616333008s, input_fired_time: 12.1144733429, output_fired_time: 5.2881817818, propagate_forward_time: 330.4854736328, propagate_backward_time: 512.9301147461, weight_update_time: 0.2472722977, other_time: 9.5433349609
epoch time: 67.8228530884s, input_fired_time: 0.7255171537, output_fired_time: 0.0000000000, propagate_forward_time: 65.4189910889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6783447266
validation[          , genome  6406] predictions:    6092/  10000 (60.92%), best:    6092/10000 (60.92%), error:     11487.81250, best error:     11487.81250 on epoch:    41, epoch:   41/50, mu: 0.9469317794, learning_rate: 0.0000119351, weight_decay: 0.0000265342

epoch time: 871.3839721680s, input_fired_time: 13.2237682343, output_fired_time: 4.2836585045, propagate_forward_time: 318.1521606445, propagate_backward_time: 526.0325317383, weight_update_time: 0.2471699268, other_time: 9.6918945313
epoch time: 67.0144805908s, input_fired_time: 1.7073711157, output_fired_time: 0.0000000000, propagate_forward_time: 63.7078132629, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5992927551
validation[          , genome  6406] predictions:    6094/  10000 (60.94%), best:    6092/10000 (60.92%), error:     11516.10645, best error:     11487.81250 on epoch:    41, epoch:   42/50, mu: 0.9494431019, learning_rate: 0.0000109831, weight_decay: 0.0000247382

epoch time: 874.0103149414s, input_fired_time: 12.1937322617, output_fired_time: 4.3281903267, propagate_forward_time: 327.6597290039, propagate_backward_time: 518.1032104492, weight_update_time: 0.2787215412, other_time: 11.7254638672
epoch time: 67.4562911987s, input_fired_time: 0.7301456928, output_fired_time: 0.0000000000, propagate_forward_time: 65.0939483643, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6321945190
validation[          , genome  6406] predictions:    6075/  10000 (60.75%), best:    6092/10000 (60.92%), error:     11565.90820, best error:     11487.81250 on epoch:    41, epoch:   43/50, mu: 0.9518079758, learning_rate: 0.0000101071, weight_decay: 0.0000230638

epoch time: 872.1775512695s, input_fired_time: 13.1304292679, output_fired_time: 5.3789176941, propagate_forward_time: 330.8855590820, propagate_backward_time: 510.3449096680, weight_update_time: 0.2524068058, other_time: 12.4377441406
epoch time: 67.4703521729s, input_fired_time: 0.7169849277, output_fired_time: 0.0000000000, propagate_forward_time: 65.2157974243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5375671387
validation[          , genome  6406] predictions:    6033/  10000 (60.33%), best:    6092/10000 (60.92%), error:     11567.03418, best error:     11487.81250 on epoch:    41, epoch:   44/50, mu: 0.9540349841, learning_rate: 0.0000093009, weight_decay: 0.0000215027

epoch time: 868.7199096680s, input_fired_time: 14.2488441467, output_fired_time: 4.2468113899, propagate_forward_time: 326.2550354004, propagate_backward_time: 513.7426757813, weight_update_time: 0.2667541504, other_time: 10.2265625000
epoch time: 66.2409057617s, input_fired_time: 0.6950609684, output_fired_time: 0.0000000000, propagate_forward_time: 64.0338973999, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5119476318
validation[          , genome  6406] predictions:    6066/  10000 (60.66%), best:    6092/10000 (60.92%), error:     11573.99316, best error:     11487.81250 on epoch:    41, epoch:   45/50, mu: 0.9561321139, learning_rate: 0.0000085590, weight_decay: 0.0000200473

epoch time: 873.8410644531s, input_fired_time: 12.3393936157, output_fired_time: 4.3339085579, propagate_forward_time: 321.4246520996, propagate_backward_time: 526.3556518555, weight_update_time: 0.2601424754, other_time: 9.3873901367
epoch time: 67.9532699585s, input_fired_time: 0.7180512547, output_fired_time: 0.0000000000, propagate_forward_time: 65.6227569580, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6124649048
validation[          , genome  6406] predictions:    6103/  10000 (61.03%), best:    6103/10000 (61.03%), error:     11469.33789, best error:     11469.33789 on epoch:    46, epoch:   46/50, mu: 0.9581069946, learning_rate: 0.0000078763, weight_decay: 0.0000186903

epoch time: 974.0992431641s, input_fired_time: 14.8899574280, output_fired_time: 5.7919635773, propagate_forward_time: 357.6496887207, propagate_backward_time: 584.3436279297, weight_update_time: 0.3049678206, other_time: 11.4239501953
epoch time: 84.6938629150s, input_fired_time: 0.8329886794, output_fired_time: 0.0000000000, propagate_forward_time: 81.9384155273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9224624634
validation[          , genome  6406] predictions:    6103/  10000 (61.03%), best:    6103/10000 (61.03%), error:     11446.22168, best error:     11446.22168 on epoch:    47, epoch:   47/50, mu: 0.9599667192, learning_rate: 0.0000072481, weight_decay: 0.0000174253

epoch time: 1098.4622802734s, input_fired_time: 16.5901241302, output_fired_time: 5.7381505966, propagate_forward_time: 404.8688964844, propagate_backward_time: 659.3903198242, weight_update_time: 0.2811619341, other_time: 11.8748168945
epoch time: 83.8021469116s, input_fired_time: 0.9311712980, output_fired_time: 0.0000000000, propagate_forward_time: 80.0374450684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8335342407
validation[          , genome  6406] predictions:    6102/  10000 (61.02%), best:    6103/10000 (61.03%), error:     11459.33301, best error:     11446.22168 on epoch:    47, epoch:   48/50, mu: 0.9617179632, learning_rate: 0.0000066700, weight_decay: 0.0000162458

epoch time: 1105.7060546875s, input_fired_time: 15.5279903412, output_fired_time: 5.3804941177, propagate_forward_time: 412.9365539551, propagate_backward_time: 659.7407226563, weight_update_time: 0.2965992391, other_time: 12.1203613281
epoch time: 86.4421234131s, input_fired_time: 0.8362198472, output_fired_time: 0.0000000000, propagate_forward_time: 82.5813827515, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0245208740
validation[          , genome  6406] predictions:    6133/  10000 (61.33%), best:    6103/10000 (61.03%), error:     11488.83203, best error:     11446.22168 on epoch:    47, epoch:   49/50, mu: 0.9633671045, learning_rate: 0.0000061379, weight_decay: 0.0000151462

epoch time: 1110.3813476563s, input_fired_time: 17.5478267670, output_fired_time: 5.6756658554, propagate_forward_time: 411.1708679199, propagate_backward_time: 663.7310180664, weight_update_time: 0.3218660653, other_time: 12.2559204102
epoch time: 86.9659194946s, input_fired_time: 0.9138283730, output_fired_time: 0.0000000000, propagate_forward_time: 84.0191726685, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0329208374
validation[          , genome  6406] predictions:    6051/  10000 (60.51%), best:    6103/10000 (61.03%), error:     11505.71680, best error:     11446.22168 on epoch:    47, epoch:   50/50, mu: 0.9649201035, learning_rate: 0.0000056484, weight_decay: 0.0000141210

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 425.5036010742s, input_fired_time: 4.5117077827, output_fired_time: 0.0000000000, propagate_forward_time: 410.9272766113, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.0646057129
best training[          , genome  6406] predictions:   32047/  50000 (64.09%), best:    6103/10000 (61.03%), error:     53880.93750, best error:     11446.22168 on epoch:    47, epoch:   51/50, mu: 0.9663825035, learning_rate: 0.0000051978, weight_decay: 0.0000131652
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 85.9337615967s, input_fired_time: 0.9739300609, output_fired_time: 0.0000000000, propagate_forward_time: 82.8339614868, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1258697510
   testing[          , genome  6406] predictions:    6103/  10000 (61.03%), best:    6103/10000 (61.03%), error:     11446.22168, best error:     11446.22168 on epoch:    47, epoch:   51/50, mu: 0.9663825035, learning_rate: 0.0000051978, weight_decay: 0.0000131652
backpropagation finished successfully!
13:04:42 (15052): called boinc_finish(0)

</stderr_txt>
]]>