Nameexact_genome_1542383404_173_6109_2
Workunit4093825
Created16 Nov 2018, 22:15:14 UTC
Sent16 Nov 2018, 22:44:45 UTC
Report deadline24 Dec 2018, 19:24:30 UTC
Received16 Dec 2018, 17:35:12 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID56670
Run time18 hours 46 min 5 sec
CPU time15 hours 16 min 42 sec
Validate stateValid
Credit9,465.24
Device peak FLOPS3.25 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size595.95 MB
Peak swap size617.46 MB
Peak disk usage13.99 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_173_6109.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_173_6109_2_r45274338_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_173_6109.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550101
read mu: 0.550101
read mu_delta: 0.96956
read initial_learning_rate: 0.00182586
read learning_rate: 0.00182586
read learning_rate_delta: 0.913947
read initial_weight_decay: 0.000344293
read weight_decay: 0.000344293
read weight_decay_delta: 0.914535
read batch_size: 121
read epsilon: 1e-07
read alpha: 0.131153
read input_dropout_probability: 0.00164786
read hidden_dropout_probability: 0.0700594
read velocity_reset: 1794
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6109
read normal distribution: '0 0.889634609222412 1.04321122169495'
generator_str: '295535268'
read generator: 295535268
read generated_by_map:
1 add_edge 1
reading 105 nodes.
reading 444 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 2079057412
post shuffle 1: 482412882
epoch time: 2170.8330078125s, input_fired_time: 29.7972621917725, output_fired_time: 11.7064352035522, propagate_forward_time: 804.062072753906, propagate_backward_time: 1309.21997070313, weight_update_time: 0.39920237660408, other_time: 16.047119140625
epoch time: 168.398864746094s, input_fired_time: 2.11796021461487, output_fired_time: 0, propagate_forward_time: 163.539199829102, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.74169921875
validation[          , genome  6109] predictions:    5523/  10000 (55.23%), best:    5523/10000 (55.23%), error:     12921.63965, best error:     12921.63965 on epoch:     0, epoch:    0/25, mu: 0.5501011610, learning_rate: 0.0018258635, weight_decay: 0.0003442926

epoch time: 2292.6447753906s, input_fired_time: 32.3439712524, output_fired_time: 12.6539535522, propagate_forward_time: 843.9260864258, propagate_backward_time: 1386.3184814453, weight_update_time: 0.4779694080, other_time: 17.4022216797
epoch time: 152.6790771484s, input_fired_time: 1.8778200150, output_fired_time: 0.0000000000, propagate_forward_time: 148.2516784668, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5495758057
validation[          , genome  6109] predictions:    5028/  10000 (50.28%), best:    5523/10000 (55.23%), error:     16289.28223, best error:     12921.63965 on epoch:     0, epoch:    1/25, mu: 0.5634915829, learning_rate: 0.0016687425, weight_decay: 0.0003148676

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_173_6109.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_173_6109_2_r45274338_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.550101
read mu: 0.576474
read mu_delta: 0.96956
read initial_learning_rate: 0.00182586
read learning_rate: 0.00152514
read learning_rate_delta: 0.913947
read initial_weight_decay: 0.000344293
read weight_decay: 0.000287957
read weight_decay_delta: 0.914535
read batch_size: 121
read epsilon: 1e-07
read alpha: 0.131153
read input_dropout_probability: 0.00164786
read hidden_dropout_probability: 0.0700594
read velocity_reset: 1794
read epoch: 2
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 5523
read best_validation_error: 12921.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6109
read normal distribution: '0 0.889634609222412 1.04321122169495'
generator_str: '654984645'
read generator: 654984645
read generated_by_map:
1 add_edge 1
reading 105 nodes.
reading 444 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2295.85400390625s, input_fired_time: 32.4330635070801, output_fired_time: 12.6241245269775, propagate_forward_time: 853.126892089844, propagate_backward_time: 1380.54504394531, weight_update_time: 0.425582468509674, other_time: 17.1248779296875
epoch time: 169.850860595703s, input_fired_time: 2.24552750587463, output_fired_time: 0, propagate_forward_time: 164.674362182617, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.93096923828125
validation[          , genome  6109] predictions:    5743/  10000 (57.43%), best:    5743/10000 (57.43%), error:     12289.43359, best error:     12289.43359 on epoch:     2, epoch:    2/25, mu: 0.5764744282, learning_rate: 0.0015251422, weight_decay: 0.0002879575

epoch time: 2691.1369628906s, input_fired_time: 39.0301132202, output_fired_time: 14.8457641602, propagate_forward_time: 985.9822998047, propagate_backward_time: 1629.5517578125, weight_update_time: 0.4358580410, other_time: 21.7271728516
epoch time: 190.4887084961s, input_fired_time: 2.4891288280, output_fired_time: 0.0000000000, propagate_forward_time: 184.6812133789, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3183593750
validation[          , genome  6109] predictions:    5774/  10000 (57.74%), best:    5774/10000 (57.74%), error:     12258.45215, best error:     12258.45215 on epoch:     3, epoch:    3/25, mu: 0.5890620351, learning_rate: 0.0013938992, weight_decay: 0.0002633471

epoch time: 2207.5131835938s, input_fired_time: 31.2592411041, output_fired_time: 11.8314933777, propagate_forward_time: 819.2165527344, propagate_backward_time: 1328.6141357422, weight_update_time: 0.4285308719, other_time: 16.5916748047
epoch time: 160.9436035156s, input_fired_time: 2.0812418461, output_fired_time: 0.0000000000, propagate_forward_time: 156.0222167969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8401489258
validation[          , genome  6109] predictions:    5765/  10000 (57.65%), best:    5774/10000 (57.74%), error:     12293.02734, best error:     12258.45215 on epoch:     3, epoch:    4/25, mu: 0.6012665033, learning_rate: 0.0012739501, weight_decay: 0.0002408402

epoch time: 2065.2041015625s, input_fired_time: 29.2566413879, output_fired_time: 11.0649919510, propagate_forward_time: 769.9685058594, propagate_backward_time: 1239.2089843750, weight_update_time: 0.3845604062, other_time: 15.7050781250
epoch time: 167.4560852051s, input_fired_time: 2.1550602913, output_fired_time: 0.0000000000, propagate_forward_time: 162.4425048828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8585205078
validation[          , genome  6109] predictions:    5687/  10000 (56.87%), best:    5774/10000 (57.74%), error:     12446.83008, best error:     12258.45215 on epoch:     3, epoch:    5/25, mu: 0.6130994558, learning_rate: 0.0011643230, weight_decay: 0.0002202567

epoch time: 2145.9580078125s, input_fired_time: 30.4032688141, output_fired_time: 11.5310640335, propagate_forward_time: 796.5787963867, propagate_backward_time: 1291.1844482422, weight_update_time: 0.4203622043, other_time: 16.2603759766
epoch time: 198.0283660889s, input_fired_time: 2.5876874924, output_fired_time: 0.0000000000, propagate_forward_time: 191.9685668945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4721069336
validation[          , genome  6109] predictions:    5943/  10000 (59.43%), best:    5943/10000 (59.43%), error:     11571.76758, best error:     11571.76758 on epoch:     6, epoch:    6/25, mu: 0.6245722175, learning_rate: 0.0010641295, weight_decay: 0.0002014325

epoch time: 2595.9470214844s, input_fired_time: 38.3700485229, output_fired_time: 14.1030569077, propagate_forward_time: 947.1753540039, propagate_backward_time: 1575.9005126953, weight_update_time: 0.4990997612, other_time: 20.3980712891
epoch time: 200.2290344238s, input_fired_time: 2.6306891441, output_fired_time: 0.0000000000, propagate_forward_time: 193.7868804932, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8114624023
validation[          , genome  6109] predictions:    5822/  10000 (58.22%), best:    5943/10000 (59.43%), error:     11975.93945, best error:     11571.76758 on epoch:     6, epoch:    7/25, mu: 0.6356957555, learning_rate: 0.0009725580, weight_decay: 0.0001842170

epoch time: 2322.4924316406s, input_fired_time: 33.6363296509, output_fired_time: 12.6028509140, propagate_forward_time: 854.5199584961, propagate_backward_time: 1403.6552734375, weight_update_time: 0.4322932065, other_time: 18.0781250000
epoch time: 183.9705047607s, input_fired_time: 2.5068643093, output_fired_time: 0.0000000000, propagate_forward_time: 178.2684936523, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1951446533
validation[          , genome  6109] predictions:    6033/  10000 (60.33%), best:    6033/10000 (60.33%), error:     11526.40527, best error:     11526.40527 on epoch:     8, epoch:    8/25, mu: 0.6464806795, learning_rate: 0.0008888665, weight_decay: 0.0001684729

epoch time: 2102.0600585938s, input_fired_time: 29.8028507233, output_fired_time: 11.4094877243, propagate_forward_time: 780.7263793945, propagate_backward_time: 1264.6939697266, weight_update_time: 0.3970451653, other_time: 15.4276123047
epoch time: 172.6328125000s, input_fired_time: 2.2239820957, output_fired_time: 0.0000000000, propagate_forward_time: 167.5153350830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8934936523
validation[          , genome  6109] predictions:    6130/  10000 (61.30%), best:    6130/10000 (61.30%), error:     11138.38379, best error:     11138.38379 on epoch:     9, epoch:    9/25, mu: 0.6569373608, learning_rate: 0.0008123770, weight_decay: 0.0001540744

epoch time: 2387.1191406250s, input_fired_time: 34.2044563293, output_fired_time: 12.9698104858, propagate_forward_time: 879.7183837891, propagate_backward_time: 1441.8590087891, weight_update_time: 0.4337191284, other_time: 18.3676757813
epoch time: 198.5967559814s, input_fired_time: 2.5162637234, output_fired_time: 0.0000000000, propagate_forward_time: 192.4478607178, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.6326293945
validation[          , genome  6109] predictions:    6133/  10000 (61.33%), best:    6133/10000 (61.33%), error:     11098.74902, best error:     11098.74902 on epoch:    10, epoch:   10/25, mu: 0.6670756936, learning_rate: 0.0007424695, weight_decay: 0.0001409064

epoch time: 2323.1850585938s, input_fired_time: 33.3035697937, output_fired_time: 12.6930360794, propagate_forward_time: 856.7881469727, propagate_backward_time: 1402.9396972656, weight_update_time: 0.4130516648, other_time: 17.4606933594
epoch time: 168.2686004639s, input_fired_time: 2.3226127625, output_fired_time: 0.0000000000, propagate_forward_time: 163.2152709961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7307128906
validation[          , genome  6109] predictions:    6160/  10000 (61.60%), best:    6133/10000 (61.33%), error:     11299.42676, best error:     11098.74902 on epoch:    10, epoch:   11/25, mu: 0.6769054532, learning_rate: 0.0006785778, weight_decay: 0.0001288638

epoch time: 2231.8569335938s, input_fired_time: 32.4846725464, output_fired_time: 12.3389654160, propagate_forward_time: 824.6398925781, propagate_backward_time: 1345.8864746094, weight_update_time: 0.4307700694, other_time: 16.5070800781
epoch time: 159.1752014160s, input_fired_time: 2.0763339996, output_fired_time: 0.0000000000, propagate_forward_time: 154.5092926025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5895690918
validation[          , genome  6109] predictions:    5839/  10000 (58.39%), best:    6133/10000 (61.33%), error:     12173.31250, best error:     11098.74902 on epoch:    10, epoch:   12/25, mu: 0.6864359975, learning_rate: 0.0006201842, weight_decay: 0.0001178505

epoch time: 2198.4160156250s, input_fired_time: 31.6547641754, output_fired_time: 12.1039066315, propagate_forward_time: 815.1956176758, propagate_backward_time: 1322.9799804688, weight_update_time: 0.4697380066, other_time: 16.4816894531
epoch time: 198.4801330566s, input_fired_time: 2.4710316658, output_fired_time: 0.0000000000, propagate_forward_time: 192.5790100098, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.4300842285
validation[          , genome  6109] predictions:    6232/  10000 (62.32%), best:    6232/10000 (62.32%), error:     10883.57129, best error:     10883.57129 on epoch:    13, epoch:   13/25, mu: 0.6956764460, learning_rate: 0.0005668155, weight_decay: 0.0001077784

epoch time: 2119.5654296875s, input_fired_time: 30.2770156860, output_fired_time: 11.4476861954, propagate_forward_time: 786.9708251953, propagate_backward_time: 1275.1059570313, weight_update_time: 0.3716264367, other_time: 15.7637939453
epoch time: 161.1760864258s, input_fired_time: 2.1066815853, output_fired_time: 0.0000000000, propagate_forward_time: 156.4744415283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5949707031
validation[          , genome  6109] predictions:    6324/  10000 (63.24%), best:    6324/10000 (63.24%), error:     10557.36328, best error:     10557.36328 on epoch:    14, epoch:   14/25, mu: 0.7046356201, learning_rate: 0.0005180393, weight_decay: 0.0000985671

epoch time: 2247.9003906250s, input_fired_time: 32.4790344238, output_fired_time: 12.2717399597, propagate_forward_time: 830.6464233398, propagate_backward_time: 1355.5485839844, weight_update_time: 0.4668618441, other_time: 16.9545898438
epoch time: 158.6616973877s, input_fired_time: 2.0856199265, output_fired_time: 0.0000000000, propagate_forward_time: 154.0063171387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5697631836
validation[          , genome  6109] predictions:    6087/  10000 (60.87%), best:    6324/10000 (63.24%), error:     11505.72266, best error:     10557.36328 on epoch:    14, epoch:   15/25, mu: 0.7133220434, learning_rate: 0.0004734605, weight_decay: 0.0000901430

epoch time: 2385.1455078125s, input_fired_time: 34.7849693298, output_fired_time: 13.0921487808, propagate_forward_time: 876.0819702148, propagate_backward_time: 1442.3414306641, weight_update_time: 0.4523914754, other_time: 18.8450927734
epoch time: 202.7918701172s, input_fired_time: 2.7200622559, output_fired_time: 0.0000000000, propagate_forward_time: 196.2453765869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.8264312744
validation[          , genome  6109] predictions:    6261/  10000 (62.61%), best:    6324/10000 (63.24%), error:     10864.23828, best error:     10557.36328 on epoch:    14, epoch:   16/25, mu: 0.7217440605, learning_rate: 0.0004327178, weight_decay: 0.0000824389

epoch time: 2303.9592285156s, input_fired_time: 33.1680068970, output_fired_time: 12.5901460648, propagate_forward_time: 850.6086425781, propagate_backward_time: 1390.3144531250, weight_update_time: 0.4656707942, other_time: 17.2780761719
epoch time: 161.6358489990s, input_fired_time: 2.1389849186, output_fired_time: 0.0000000000, propagate_forward_time: 156.7949829102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7018737793
validation[          , genome  6109] predictions:    6364/  10000 (63.64%), best:    6364/10000 (63.64%), error:     10483.51660, best error:     10483.51660 on epoch:    17, epoch:   17/25, mu: 0.7299097180, learning_rate: 0.0003954812, weight_decay: 0.0000753933

epoch time: 2308.1674804688s, input_fired_time: 33.3112831116, output_fired_time: 12.6162004471, propagate_forward_time: 853.2587890625, propagate_backward_time: 1391.9046630859, weight_update_time: 0.4085017443, other_time: 17.0765380859
epoch time: 161.0323638916s, input_fired_time: 2.1888573170, output_fired_time: 0.0000000000, propagate_forward_time: 156.2544860840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5890197754
validation[          , genome  6109] predictions:    6350/  10000 (63.50%), best:    6364/10000 (63.64%), error:     10598.20605, best error:     10483.51660 on epoch:    17, epoch:   18/25, mu: 0.7378268242, learning_rate: 0.0003614489, weight_decay: 0.0000689498

epoch time: 2475.1108398438s, input_fired_time: 36.1514472961, output_fired_time: 13.2252244949, propagate_forward_time: 906.0258178711, propagate_backward_time: 1500.9963378906, weight_update_time: 0.4516619444, other_time: 18.7119140625
epoch time: 161.7775115967s, input_fired_time: 2.1397390366, output_fired_time: 0.0000000000, propagate_forward_time: 156.7320861816, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9056854248
validation[          , genome  6109] predictions:    6229/  10000 (62.29%), best:    6364/10000 (63.64%), error:     10830.51953, best error:     10483.51660 on epoch:    17, epoch:   19/25, mu: 0.7455029488, learning_rate: 0.0003303451, weight_decay: 0.0000630570

epoch time: 2298.2058105469s, input_fired_time: 32.8491516113, output_fired_time: 12.4935369492, propagate_forward_time: 851.0946044922, propagate_backward_time: 1384.0290527344, weight_update_time: 0.3821263611, other_time: 17.7393798828
epoch time: 158.7785949707s, input_fired_time: 2.1699068546, output_fired_time: 0.0000000000, propagate_forward_time: 154.0075988770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6010894775
validation[          , genome  6109] predictions:    6399/  10000 (63.99%), best:    6399/10000 (63.99%), error:     10421.74512, best error:     10421.74512 on epoch:    20, epoch:   20/25, mu: 0.7529453635, learning_rate: 0.0003019180, weight_decay: 0.0000576678

epoch time: 2446.0734863281s, input_fired_time: 35.6793060303, output_fired_time: 13.2650814056, propagate_forward_time: 896.0485839844, propagate_backward_time: 1481.9647216797, weight_update_time: 0.4801561236, other_time: 19.1158447266
epoch time: 159.3664703369s, input_fired_time: 2.0908560753, output_fired_time: 0.0000000000, propagate_forward_time: 154.5446929932, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7309265137
validation[          , genome  6109] predictions:    6397/  10000 (63.97%), best:    6397/10000 (63.97%), error:     10376.10742, best error:     10376.10742 on epoch:    21, epoch:   21/25, mu: 0.7601612806, learning_rate: 0.0002759370, weight_decay: 0.0000527393

epoch time: 2140.3474121094s, input_fired_time: 30.7491340637, output_fired_time: 11.6348400116, propagate_forward_time: 796.8413085938, propagate_backward_time: 1285.2957763672, weight_update_time: 0.4289114475, other_time: 15.8265380859
epoch time: 213.8933563232s, input_fired_time: 3.0313451290, output_fired_time: 0.0000000000, propagate_forward_time: 207.0872650146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7747497559
validation[          , genome  6109] predictions:    6439/  10000 (64.39%), best:    6439/10000 (64.39%), error:     10299.25781, best error:     10299.25781 on epoch:    22, epoch:   22/25, mu: 0.7671574950, learning_rate: 0.0002521918, weight_decay: 0.0000482319

epoch time: 2621.3139648438s, input_fired_time: 39.1007156372, output_fired_time: 14.3527584076, propagate_forward_time: 960.7664184570, propagate_backward_time: 1585.9180908203, weight_update_time: 0.5247754455, other_time: 21.1759033203
epoch time: 262.8938598633s, input_fired_time: 4.1171226501, output_fired_time: 0.0000000000, propagate_forward_time: 253.4467010498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3300323486
validation[          , genome  6109] predictions:    6415/  10000 (64.15%), best:    6439/10000 (64.39%), error:     10527.34668, best error:     10299.25781 on epoch:    22, epoch:   23/25, mu: 0.7739408016, learning_rate: 0.0002304900, weight_decay: 0.0000441097

epoch time: 3291.7592773438s, input_fired_time: 50.0099449158, output_fired_time: 18.5802459717, propagate_forward_time: 1181.3780517578, propagate_backward_time: 2012.8822021484, weight_update_time: 0.6265979409, other_time: 28.9086914063
epoch time: 244.6785125732s, input_fired_time: 3.6040780544, output_fired_time: 0.0000000000, propagate_forward_time: 236.0428161621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0316162109
validation[          , genome  6109] predictions:    6429/  10000 (64.29%), best:    6429/10000 (64.29%), error:     10224.59668, best error:     10224.59668 on epoch:    24, epoch:   24/25, mu: 0.7805175781, learning_rate: 0.0002106556, weight_decay: 0.0000403399

epoch time: 2906.2504882813s, input_fired_time: 43.5946807861, output_fired_time: 16.4830188751, propagate_forward_time: 1054.7136230469, propagate_backward_time: 1766.4285888672, weight_update_time: 0.5584387779, other_time: 25.0306396484
epoch time: 226.1434631348s, input_fired_time: 3.0045392513, output_fired_time: 0.0000000000, propagate_forward_time: 218.8711090088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2678222656
validation[          , genome  6109] predictions:    6380/  10000 (63.80%), best:    6429/10000 (64.29%), error:     10566.40527, best error:     10224.59668 on epoch:    24, epoch:   25/25, mu: 0.7868942022, learning_rate: 0.0001925281, weight_decay: 0.0000368922

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 999.3626708984s, input_fired_time: 14.4629068375, output_fired_time: 0.0000000000, propagate_forward_time: 965.4378051758, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 19.4619750977
best training[          , genome  6109] predictions:   35594/  50000 (71.19%), best:    6429/10000 (64.29%), error:     42121.49219, best error:     10224.59668 on epoch:    24, epoch:   26/25, mu: 0.7930766940, learning_rate: 0.0001759605, weight_decay: 0.0000337392
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 167.4480285645s, input_fired_time: 2.4228417873, output_fired_time: 0.0000000000, propagate_forward_time: 161.6920318604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3331604004
   testing[          , genome  6109] predictions:    6429/  10000 (64.29%), best:    6429/10000 (64.29%), error:     10224.59668, best error:     10224.59668 on epoch:    24, epoch:   26/25, mu: 0.7930766940, learning_rate: 0.0001759605, weight_decay: 0.0000337392
backpropagation finished successfully!
10:34:47 (1352): called boinc_finish(0)

</stderr_txt>
]]>