Nameexact_genome_1542727803_170_6409_0
Workunit4101336
Created20 Nov 2018, 20:17:16 UTC
Sent20 Nov 2018, 20:18:09 UTC
Report deadline20 Dec 2018, 6:15:09 UTC
Received23 Nov 2018, 7:07:19 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID88573
Run time13 hours 49 min 56 sec
CPU time13 hours 49 min 56 sec
Validate stateValid
Credit7,353.65
Device peak FLOPS3.35 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size572.89 MB
Peak swap size571.64 MB
Peak disk usage12.52 MB

Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6409.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6409_0_r2032315885_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_170_6409.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.462669
read mu: 0.462669
read mu_delta: 0.93806
read initial_learning_rate: 0.000231681
read learning_rate: 0.000231681
read learning_rate_delta: 0.903435
read initial_weight_decay: 0.000189602
read weight_decay: 0.000189602
read weight_decay_delta: 0.923235
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.049622
read input_dropout_probability: 0.00152739
read hidden_dropout_probability: 0.0755906
read velocity_reset: 1387
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6409
read normal distribution: '1 0.485888063907623 -0.977158308029175'
generator_str: '1768325499'
read generator: 1768325499
read generated_by_map:
1 change_size 1
reading 120 nodes.
reading 447 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 1220470860
post shuffle 1: 2069547512
epoch time: 1668.85607910156s, input_fired_time: 23.8368167877197, output_fired_time: 11.9220132827759, propagate_forward_time: 661.980346679688, propagate_backward_time: 954.6494140625, weight_update_time: 0.510562658309937, other_time: 16.467529296875
epoch time: 108.495948791504s, input_fired_time: 1.66841244697571, output_fired_time: 0, propagate_forward_time: 104.041702270508, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.78583526611328
validation[          , genome  6409] predictions:    6235/  10000 (62.35%), best:    6235/10000 (62.35%), error:     10995.70313, best error:     10995.70313 on epoch:     0, epoch:    0/25, mu: 0.4626686871, learning_rate: 0.0002316809, weight_decay: 0.0001896016

epoch time: 1648.5502929688s, input_fired_time: 23.7779598236, output_fired_time: 11.8386096954, propagate_forward_time: 656.2091064453, propagate_backward_time: 940.3030395508, weight_update_time: 0.5067865252, other_time: 16.4215698242
epoch time: 187.5652618408s, input_fired_time: 2.3611662388, output_fired_time: 0.0000000000, propagate_forward_time: 181.9763488770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2277526855
validation[          , genome  6409] predictions:    6268/  10000 (62.68%), best:    6268/10000 (62.68%), error:     10762.83594, best error:     10762.83594 on epoch:     1, epoch:    1/25, mu: 0.4953315854, learning_rate: 0.0002093086, weight_decay: 0.0001750469

epoch time: 1757.8259277344s, input_fired_time: 24.3033237457, output_fired_time: 12.5750312805, propagate_forward_time: 701.6086425781, propagate_backward_time: 1002.5965576172, weight_update_time: 0.5326548815, other_time: 16.7423095703
epoch time: 186.7480621338s, input_fired_time: 2.4047758579, output_fired_time: 0.0000000000, propagate_forward_time: 180.9753875732, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3679046631
validation[          , genome  6409] predictions:    6250/  10000 (62.50%), best:    6268/10000 (62.68%), error:     10897.15039, best error:     10762.83594 on epoch:     1, epoch:    2/25, mu: 0.5259713531, learning_rate: 0.0001890967, weight_decay: 0.0001616095

epoch time: 1891.0095214844s, input_fired_time: 25.2781200409, output_fired_time: 13.5552387238, propagate_forward_time: 760.5904541016, propagate_backward_time: 1073.8261718750, weight_update_time: 0.5788941979, other_time: 17.7595214844
epoch time: 114.2642440796s, input_fired_time: 1.6566382647, output_fired_time: 0.0000000000, propagate_forward_time: 109.9214706421, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6861343384
validation[          , genome  6409] predictions:    6201/  10000 (62.01%), best:    6268/10000 (62.68%), error:     11079.21973, best error:     10762.83594 on epoch:     1, epoch:    3/25, mu: 0.5547133088, learning_rate: 0.0001708366, weight_decay: 0.0001492036

epoch time: 1916.9781494141s, input_fired_time: 25.4865417480, output_fired_time: 13.7774066925, propagate_forward_time: 769.3474731445, propagate_backward_time: 1090.2469482422, weight_update_time: 0.5880562067, other_time: 18.1197509766
epoch time: 122.9302673340s, input_fired_time: 1.7322703600, output_fired_time: 0.0000000000, propagate_forward_time: 118.4419174194, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7560806274
validation[          , genome  6409] predictions:    6109/  10000 (61.09%), best:    6268/10000 (62.68%), error:     11226.51465, best error:     10762.83594 on epoch:     1, epoch:    4/25, mu: 0.5816749930, learning_rate: 0.0001543397, weight_decay: 0.0001377501

epoch time: 1812.0587158203s, input_fired_time: 24.5001544952, output_fired_time: 12.8738832474, propagate_forward_time: 725.4080200195, propagate_backward_time: 1032.4870605469, weight_update_time: 0.5427616239, other_time: 16.7895507813
epoch time: 110.3630981445s, input_fired_time: 1.6793735027, output_fired_time: 0.0000000000, propagate_forward_time: 105.7337570190, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9499664307
validation[          , genome  6409] predictions:    6149/  10000 (61.49%), best:    6268/10000 (62.68%), error:     11207.23145, best error:     10762.83594 on epoch:     1, epoch:    5/25, mu: 0.6069666743, learning_rate: 0.0001394359, weight_decay: 0.0001271758

epoch time: 1682.7574462891s, input_fired_time: 24.2578754425, output_fired_time: 12.2366609573, propagate_forward_time: 669.3082885742, propagate_backward_time: 959.4943847656, weight_update_time: 0.5309899449, other_time: 17.4601440430
epoch time: 125.6095581055s, input_fired_time: 1.8209818602, output_fired_time: 0.0000000000, propagate_forward_time: 120.8139953613, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9745788574
validation[          , genome  6409] predictions:    6206/  10000 (62.06%), best:    6268/10000 (62.68%), error:     11084.03125, best error:     10762.83594 on epoch:     1, epoch:    6/25, mu: 0.6306917667, learning_rate: 0.0001259713, weight_decay: 0.0001174132

epoch time: 1746.4537353516s, input_fired_time: 24.1167182922, output_fired_time: 12.4708318710, propagate_forward_time: 696.9439697266, propagate_backward_time: 996.4130859375, weight_update_time: 0.5258884430, other_time: 16.5091552734
epoch time: 135.8058776855s, input_fired_time: 1.8497866392, output_fired_time: 0.0000000000, propagate_forward_time: 131.0997314453, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8563537598
validation[          , genome  6409] predictions:    6217/  10000 (62.17%), best:    6268/10000 (62.68%), error:     11143.73145, best error:     10762.83594 on epoch:     1, epoch:    7/25, mu: 0.6529473066, learning_rate: 0.0001138068, weight_decay: 0.0001084000

epoch time: 1542.8647460938s, input_fired_time: 23.2094860077, output_fired_time: 11.0785856247, propagate_forward_time: 609.8472900391, propagate_backward_time: 882.3917236328, weight_update_time: 0.4886945486, other_time: 16.3376464844
epoch time: 119.5962219238s, input_fired_time: 1.7654035091, output_fired_time: 0.0000000000, propagate_forward_time: 114.8782043457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9526138306
validation[          , genome  6409] predictions:    6193/  10000 (61.93%), best:    6268/10000 (62.68%), error:     11131.88184, best error:     10762.83594 on epoch:     1, epoch:    8/25, mu: 0.6738243699, learning_rate: 0.0001028171, weight_decay: 0.0001000787

epoch time: 1674.0250244141s, input_fired_time: 24.1679611206, output_fired_time: 12.0839548111, propagate_forward_time: 666.2083129883, propagate_backward_time: 954.2812500000, weight_update_time: 0.5297961235, other_time: 17.2835083008
epoch time: 167.8381500244s, input_fired_time: 2.2164659500, output_fired_time: 0.0000000000, propagate_forward_time: 162.3827362061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2389526367
validation[          , genome  6409] predictions:    6246/  10000 (62.46%), best:    6268/10000 (62.68%), error:     10942.67285, best error:     10762.83594 on epoch:     1, epoch:    9/25, mu: 0.6934083104, learning_rate: 0.0000928885, weight_decay: 0.0000923962

epoch time: 1785.8682861328s, input_fired_time: 24.6880950928, output_fired_time: 12.8794479370, propagate_forward_time: 711.9951171875, propagate_backward_time: 1018.7562255859, weight_update_time: 0.5575437546, other_time: 17.5494384766
epoch time: 107.1090545654s, input_fired_time: 1.6354490519, output_fired_time: 0.0000000000, propagate_forward_time: 102.6898803711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7837219238
validation[          , genome  6409] predictions:    6275/  10000 (62.75%), best:    6268/10000 (62.68%), error:     10979.07617, best error:     10762.83594 on epoch:     1, epoch:   10/25, mu: 0.7117791772, learning_rate: 0.0000839187, weight_decay: 0.0000853034

epoch time: 1706.2448730469s, input_fired_time: 24.0637283325, output_fired_time: 12.2306003571, propagate_forward_time: 679.5122070313, propagate_backward_time: 973.6263427734, weight_update_time: 0.5234120488, other_time: 16.8120117188
epoch time: 108.8859786987s, input_fired_time: 1.6043249369, output_fired_time: 0.0000000000, propagate_forward_time: 104.5277633667, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7538909912
validation[          , genome  6409] predictions:    6338/  10000 (63.38%), best:    6268/10000 (62.68%), error:     10880.62012, best error:     10762.83594 on epoch:     1, epoch:   11/25, mu: 0.7290121913, learning_rate: 0.0000758151, weight_decay: 0.0000787552

epoch time: 1571.0229492188s, input_fired_time: 23.2177581787, output_fired_time: 11.2205123901, propagate_forward_time: 622.6058959961, propagate_backward_time: 897.8061523438, weight_update_time: 0.4871104956, other_time: 16.1726684570
epoch time: 157.9899749756s, input_fired_time: 2.0705735683, output_fired_time: 0.0000000000, propagate_forward_time: 152.8828430176, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0365600586
validation[          , genome  6409] predictions:    6217/  10000 (62.17%), best:    6268/10000 (62.68%), error:     11001.81055, best error:     10762.83594 on epoch:     1, epoch:   12/25, mu: 0.7451778054, learning_rate: 0.0000684940, weight_decay: 0.0000727096

epoch time: 1838.4273681641s, input_fired_time: 24.7469863892, output_fired_time: 13.1378126144, propagate_forward_time: 736.5593261719, propagate_backward_time: 1046.7603759766, weight_update_time: 0.5571324825, other_time: 17.2229003906
epoch time: 132.1060638428s, input_fired_time: 1.8738650084, output_fired_time: 0.0000000000, propagate_forward_time: 127.2712097168, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9609832764
validation[          , genome  6409] predictions:    6269/  10000 (62.69%), best:    6268/10000 (62.68%), error:     10920.28223, best error:     10762.83594 on epoch:     1, epoch:   13/25, mu: 0.7603421211, learning_rate: 0.0000618799, weight_decay: 0.0000671280

epoch time: 1829.4949951172s, input_fired_time: 24.6532077789, output_fired_time: 13.0714721680, propagate_forward_time: 733.5304565430, propagate_backward_time: 1041.2268066406, weight_update_time: 0.5478580594, other_time: 17.0129394531
epoch time: 165.7221527100s, input_fired_time: 2.1802139282, output_fired_time: 0.0000000000, propagate_forward_time: 160.3335266113, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2084045410
validation[          , genome  6409] predictions:    6268/  10000 (62.68%), best:    6268/10000 (62.68%), error:     11004.82520, best error:     10762.83594 on epoch:     1, epoch:   14/25, mu: 0.7745671272, learning_rate: 0.0000559045, weight_decay: 0.0000619750

epoch time: 1825.3701171875s, input_fired_time: 24.9007549286, output_fired_time: 13.1082592010, propagate_forward_time: 730.1005249023, propagate_backward_time: 1039.6096191406, weight_update_time: 0.5687335730, other_time: 17.6508789063
epoch time: 141.2633972168s, input_fired_time: 1.9501339197, output_fired_time: 0.0000000000, propagate_forward_time: 136.2714843750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0417785645
validation[          , genome  6409] predictions:    6150/  10000 (61.50%), best:    6268/10000 (62.68%), error:     11228.02051, best error:     10762.83594 on epoch:     1, epoch:   15/25, mu: 0.7879110575, learning_rate: 0.0000505060, weight_decay: 0.0000572175

epoch time: 1804.3251953125s, input_fired_time: 24.5563316345, output_fired_time: 12.9422292709, propagate_forward_time: 719.6332397461, propagate_backward_time: 1030.0373535156, weight_update_time: 0.5471339226, other_time: 17.1560058594
epoch time: 131.8251342773s, input_fired_time: 1.8200151920, output_fired_time: 0.0000000000, propagate_forward_time: 127.1429367065, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8621749878
validation[          , genome  6409] predictions:    6260/  10000 (62.60%), best:    6268/10000 (62.68%), error:     10911.94824, best error:     10762.83594 on epoch:     1, epoch:   16/25, mu: 0.8004284501, learning_rate: 0.0000456289, weight_decay: 0.0000528252

epoch time: 1859.0192871094s, input_fired_time: 25.0151462555, output_fired_time: 13.3024187088, propagate_forward_time: 745.3380126953, propagate_backward_time: 1057.7712402344, weight_update_time: 0.5688672662, other_time: 17.5925292969
epoch time: 185.2924346924s, input_fired_time: 2.3730905056, output_fired_time: 0.0000000000, propagate_forward_time: 179.5490264893, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3703155518
validation[          , genome  6409] predictions:    6266/  10000 (62.66%), best:    6268/10000 (62.68%), error:     10991.35645, best error:     10762.83594 on epoch:     1, epoch:   17/25, mu: 0.8121705055, learning_rate: 0.0000412228, weight_decay: 0.0000487701

epoch time: 1799.2532958984s, input_fired_time: 24.7772216797, output_fired_time: 12.9698266983, propagate_forward_time: 718.2687988281, propagate_backward_time: 1025.6724853516, weight_update_time: 0.5608329177, other_time: 17.5649414063
epoch time: 159.2770690918s, input_fired_time: 2.1257159710, output_fired_time: 0.0000000000, propagate_forward_time: 153.9717559814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1795959473
validation[          , genome  6409] predictions:    6208/  10000 (62.08%), best:    6268/10000 (62.68%), error:     11004.45313, best error:     10762.83594 on epoch:     1, epoch:   18/25, mu: 0.8231852651, learning_rate: 0.0000372421, weight_decay: 0.0000450263

epoch time: 1470.1826171875s, input_fired_time: 23.0322532654, output_fired_time: 10.6410856247, propagate_forward_time: 579.4778442383, propagate_backward_time: 840.6044921875, weight_update_time: 0.4808721840, other_time: 16.4269409180
epoch time: 164.0636901855s, input_fired_time: 2.1749646664, output_fired_time: 0.0000000000, propagate_forward_time: 158.6893768311, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1993560791
validation[          , genome  6409] predictions:    6319/  10000 (63.19%), best:    6268/10000 (62.68%), error:     10856.28027, best error:     10762.83594 on epoch:     1, epoch:   19/25, mu: 0.8335177898, learning_rate: 0.0000336458, weight_decay: 0.0000415699

epoch time: 1751.5064697266s, input_fired_time: 24.4202079773, output_fired_time: 12.6090230942, propagate_forward_time: 699.1564331055, propagate_backward_time: 998.1719360352, weight_update_time: 0.5418261886, other_time: 17.1489257813
epoch time: 103.8401947021s, input_fired_time: 1.5607229471, output_fired_time: 0.0000000000, propagate_forward_time: 99.5034713745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7760009766
validation[          , genome  6409] predictions:    6331/  10000 (63.31%), best:    6268/10000 (62.68%), error:     10801.31348, best error:     10762.83594 on epoch:     1, epoch:   20/25, mu: 0.8432102799, learning_rate: 0.0000303968, weight_decay: 0.0000383788

epoch time: 1757.9931640625s, input_fired_time: 24.3475856781, output_fired_time: 12.6120529175, propagate_forward_time: 701.3215332031, propagate_backward_time: 1002.6086425781, weight_update_time: 0.5345239639, other_time: 17.1033935547
epoch time: 182.5377197266s, input_fired_time: 2.3101956844, output_fired_time: 0.0000000000, propagate_forward_time: 177.0245513916, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2029724121
validation[          , genome  6409] predictions:    6317/  10000 (63.17%), best:    6268/10000 (62.68%), error:     10839.12207, best error:     10762.83594 on epoch:     1, epoch:   21/25, mu: 0.8523024321, learning_rate: 0.0000274615, weight_decay: 0.0000354326

epoch time: 1683.4652099609s, input_fired_time: 23.9121093750, output_fired_time: 12.1113624573, propagate_forward_time: 669.1434936523, propagate_backward_time: 961.5017700195, weight_update_time: 0.5158163905, other_time: 16.7965087891
epoch time: 153.2574310303s, input_fired_time: 2.0981810093, output_fired_time: 0.0000000000, propagate_forward_time: 147.9735565186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1856994629
validation[          , genome  6409] predictions:    6277/  10000 (62.77%), best:    6268/10000 (62.68%), error:     10840.84668, best error:     10762.83594 on epoch:     1, epoch:   22/25, mu: 0.8608314395, learning_rate: 0.0000248097, weight_decay: 0.0000327127

epoch time: 1670.0969238281s, input_fired_time: 24.2877311707, output_fired_time: 12.1528921127, propagate_forward_time: 663.4765625000, propagate_backward_time: 952.5466918945, weight_update_time: 0.5302042365, other_time: 17.6331176758
epoch time: 139.5728912354s, input_fired_time: 1.9564086199, output_fired_time: 0.0000000000, propagate_forward_time: 134.5446777344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0718078613
validation[          , genome  6409] predictions:    6271/  10000 (62.71%), best:    6268/10000 (62.68%), error:     10914.53125, best error:     10762.83594 on epoch:     1, epoch:   23/25, mu: 0.8688321710, learning_rate: 0.0000224139, weight_decay: 0.0000302015

epoch time: 1798.0897216797s, input_fired_time: 25.0447330475, output_fired_time: 13.0748090744, propagate_forward_time: 716.0213012695, propagate_backward_time: 1025.6551513672, weight_update_time: 0.5634174943, other_time: 18.2938232422
epoch time: 157.7101898193s, input_fired_time: 2.1412563324, output_fired_time: 0.0000000000, propagate_forward_time: 152.3532562256, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2156829834
validation[          , genome  6409] predictions:    6384/  10000 (63.84%), best:    6268/10000 (62.68%), error:     10772.10254, best error:     10762.83594 on epoch:     1, epoch:   24/25, mu: 0.8763372898, learning_rate: 0.0000202495, weight_decay: 0.0000278831

epoch time: 1668.3968505859s, input_fired_time: 23.9414463043, output_fired_time: 12.0563201904, propagate_forward_time: 660.3247680664, propagate_backward_time: 954.8240966797, weight_update_time: 0.5173661709, other_time: 17.2503051758
epoch time: 157.8247833252s, input_fired_time: 2.1311318874, output_fired_time: 0.0000000000, propagate_forward_time: 151.9537506104, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.7398986816
validation[          , genome  6409] predictions:    6203/  10000 (62.03%), best:    6268/10000 (62.68%), error:     11008.84863, best error:     10762.83594 on epoch:     1, epoch:   25/25, mu: 0.8833775520, learning_rate: 0.0000182941, weight_decay: 0.0000257427

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 737.4912109375s, input_fired_time: 9.9651508331, output_fired_time: 0.0000000000, propagate_forward_time: 712.1679077148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 15.3581542969
best training[          , genome  6409] predictions:   34859/  50000 (69.72%), best:    6268/10000 (62.68%), error:     46297.75000, best error:     10762.83594 on epoch:     1, epoch:   26/25, mu: 0.8899817467, learning_rate: 0.0000165276, weight_decay: 0.0000237665
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 153.9889221191s, input_fired_time: 2.1220967770, output_fired_time: 0.0000000000, propagate_forward_time: 148.5169525146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3498687744
   testing[          , genome  6409] predictions:    6268/  10000 (62.68%), best:    6268/10000 (62.68%), error:     10762.83594, best error:     10762.83594 on epoch:     1, epoch:   26/25, mu: 0.8899817467, learning_rate: 0.0000165276, weight_decay: 0.0000237665
backpropagation finished successfully!
01:04:51 (3232): called boinc_finish(0)

</stderr_txt>
]]>