Nameexact_genome_1542383404_179_6104_1
Workunit4093799
Created16 Nov 2018, 21:39:20 UTC
Sent16 Nov 2018, 21:57:34 UTC
Report deadline19 Jan 2019, 23:04:04 UTC
Received20 Nov 2018, 5:20:27 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID30770
Run time22 hours 18 min 29 sec
CPU time22 hours 11 min 52 sec
Validate stateValid
Credit16,011.55
Device peak FLOPS4.85 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size572.72 MB
Peak swap size594.32 MB
Peak disk usage15.66 MB

Stderr output

<core_client_version>7.6.33</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104_1_r446561101_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542383404_179_6104.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.512452
read mu: 0.512452
read mu_delta: 0.954665
read initial_learning_rate: 0.00186827
read learning_rate: 0.00186827
read learning_rate_delta: 0.913556
read initial_weight_decay: 0.000439731
read weight_decay: 0.000439731
read weight_decay_delta: 0.936346
read batch_size: 107
read epsilon: 1e-07
read alpha: 0.118664
read input_dropout_probability: 0.00133047
read hidden_dropout_probability: 0.0964702
read velocity_reset: 1748
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6104
read normal distribution: '0 -0.355088144540787 0.925766110420227'
generator_str: '662050602'
read generator: 662050602
read generated_by_map:
1 change_size 1
reading 107 nodes.
reading 565 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 971692707
post shuffle 1: 189589552
epoch time: 1413.30639648438s, input_fired_time: 18.9323558807373, output_fired_time: 8.10397434234619, propagate_forward_time: 539.046508789063, propagate_backward_time: 834.912048339844, weight_update_time: 0.43188750743866, other_time: 12.3114624023438
epoch time: 112.811058044434s, input_fired_time: 1.28871548175812, output_fired_time: 0, propagate_forward_time: 109.349349975586, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.17298889160156
validation[          , genome  6104] predictions:    5594/  10000 (55.94%), best:    5594/10000 (55.94%), error:     13014.06250, best error:     13014.06250 on epoch:     0, epoch:    0/50, mu: 0.5124523640, learning_rate: 0.0018682743, weight_decay: 0.0004397306

epoch time: 1419.8350830078s, input_fired_time: 19.1010646820, output_fired_time: 8.2036437988, propagate_forward_time: 541.6519775391, propagate_backward_time: 838.3946533203, weight_update_time: 0.4222978950, other_time: 12.4837646484
epoch time: 112.1149444580s, input_fired_time: 1.2723549604, output_fired_time: 0.0000000000, propagate_forward_time: 108.7258605957, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1167297363
validation[          , genome  6104] predictions:    5773/  10000 (57.73%), best:    5773/10000 (57.73%), error:     12116.42871, best error:     12116.42871 on epoch:     1, epoch:    1/50, mu: 0.5341020823, learning_rate: 0.0017067726, weight_decay: 0.0004117398

epoch time: 1412.0239257813s, input_fired_time: 18.9907283783, output_fired_time: 8.1554594040, propagate_forward_time: 539.8110961914, propagate_backward_time: 832.4772949219, weight_update_time: 0.4329389930, other_time: 12.5892944336
epoch time: 112.0612716675s, input_fired_time: 1.2773150206, output_fired_time: 0.0000000000, propagate_forward_time: 108.6510772705, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1328811646
validation[          , genome  6104] predictions:    5926/  10000 (59.26%), best:    5926/10000 (59.26%), error:     11786.67383, best error:     11786.67383 on epoch:     2, epoch:    2/50, mu: 0.5547702909, learning_rate: 0.0015592318, weight_decay: 0.0003855307

epoch time: 1426.1071777344s, input_fired_time: 19.2227783203, output_fired_time: 8.2620372772, propagate_forward_time: 544.2435913086, propagate_backward_time: 841.6785888672, weight_update_time: 0.4416877627, other_time: 12.7001342773
epoch time: 113.4067459106s, input_fired_time: 1.2983518839, output_fired_time: 0.0000000000, propagate_forward_time: 109.9010314941, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2073593140
validation[          , genome  6104] predictions:    5839/  10000 (58.39%), best:    5839/10000 (58.39%), error:     11688.76074, best error:     11688.76074 on epoch:     3, epoch:    3/50, mu: 0.5745015144, learning_rate: 0.0014244451, weight_decay: 0.0003609900

epoch time: 1421.2695312500s, input_fired_time: 19.1914253235, output_fired_time: 8.2222747803, propagate_forward_time: 542.3511962891, propagate_backward_time: 838.8473510742, weight_update_time: 0.4351136982, other_time: 12.6572875977
epoch time: 111.8991165161s, input_fired_time: 1.3062206507, output_fired_time: 0.0000000000, propagate_forward_time: 108.4115142822, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1813812256
validation[          , genome  6104] predictions:    5700/  10000 (57.00%), best:    5839/10000 (58.39%), error:     12101.68750, best error:     11688.76074 on epoch:     3, epoch:    4/50, mu: 0.5933381915, learning_rate: 0.0013013099, weight_decay: 0.0003380114

epoch time: 1414.8795166016s, input_fired_time: 19.1428394318, output_fired_time: 8.1610393524, propagate_forward_time: 539.4784545898, propagate_backward_time: 835.5673217773, weight_update_time: 0.4400329888, other_time: 12.5299072266
epoch time: 112.4314193726s, input_fired_time: 1.2960290909, output_fired_time: 0.0000000000, propagate_forward_time: 108.9343643188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2010269165
validation[          , genome  6104] predictions:    6070/  10000 (60.70%), best:    6070/10000 (60.70%), error:     11176.89453, best error:     11176.89453 on epoch:     5, epoch:    5/50, mu: 0.6113209128, learning_rate: 0.0011888191, weight_decay: 0.0003164955

epoch time: 1426.8695068359s, input_fired_time: 19.3356361389, output_fired_time: 8.2214422226, propagate_forward_time: 545.0816040039, propagate_backward_time: 841.6191406250, weight_update_time: 0.4391409755, other_time: 12.6116333008
epoch time: 112.8962631226s, input_fired_time: 1.3230657578, output_fired_time: 0.0000000000, propagate_forward_time: 109.4259567261, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1472396851
validation[          , genome  6104] predictions:    6024/  10000 (60.24%), best:    6070/10000 (60.70%), error:     11431.93848, best error:     11176.89453 on epoch:     5, epoch:    6/50, mu: 0.6284884214, learning_rate: 0.0010860524, weight_decay: 0.0002963491

epoch time: 1410.0677490234s, input_fired_time: 19.1386127472, output_fired_time: 8.1274833679, propagate_forward_time: 538.3361206055, propagate_backward_time: 832.0332641602, weight_update_time: 0.4294342101, other_time: 12.4322509766
epoch time: 110.8425292969s, input_fired_time: 1.2913182974, output_fired_time: 0.0000000000, propagate_forward_time: 107.4571151733, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0940933228
validation[          , genome  6104] predictions:    6043/  10000 (60.43%), best:    6070/10000 (60.70%), error:     11189.08203, best error:     11176.89453 on epoch:     5, epoch:    7/50, mu: 0.6448776126, learning_rate: 0.0009921694, weight_decay: 0.0002774852

epoch time: 1406.5461425781s, input_fired_time: 19.1393260956, output_fired_time: 8.0807580948, propagate_forward_time: 536.9148559570, propagate_backward_time: 830.1715087891, weight_update_time: 0.4307921231, other_time: 12.2396850586
epoch time: 111.0626449585s, input_fired_time: 1.2921366692, output_fired_time: 0.0000000000, propagate_forward_time: 107.6501922607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1203155518
validation[          , genome  6104] predictions:    6013/  10000 (60.13%), best:    6070/10000 (60.70%), error:     11676.64551, best error:     11176.89453 on epoch:     5, epoch:    8/50, mu: 0.6605237722, learning_rate: 0.0009064020, weight_decay: 0.0002598220

epoch time: 1421.2502441406s, input_fired_time: 19.2438850403, output_fired_time: 8.1829061508, propagate_forward_time: 542.5805053711, propagate_backward_time: 838.8823852539, weight_update_time: 0.4406845570, other_time: 12.3605957031
epoch time: 112.1991348267s, input_fired_time: 1.3040812016, output_fired_time: 0.0000000000, propagate_forward_time: 108.7798309326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1152191162
validation[          , genome  6104] predictions:    6197/  10000 (61.97%), best:    6197/10000 (61.97%), error:     10767.65820, best error:     10767.65820 on epoch:     9, epoch:    9/50, mu: 0.6754606366, learning_rate: 0.0008280487, weight_decay: 0.0002432832

epoch time: 1420.9431152344s, input_fired_time: 19.2348213196, output_fired_time: 8.1915864944, propagate_forward_time: 541.9776611328, propagate_backward_time: 839.1340942383, weight_update_time: 0.4306990504, other_time: 12.4049682617
epoch time: 111.3463668823s, input_fired_time: 1.2998065948, output_fired_time: 0.0000000000, propagate_forward_time: 107.9542922974, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0922698975
validation[          , genome  6104] predictions:    6172/  10000 (61.72%), best:    6197/10000 (61.97%), error:     10944.30566, best error:     10767.65820 on epoch:     9, epoch:   10/50, mu: 0.6897203326, learning_rate: 0.0007564686, weight_decay: 0.0002277972

epoch time: 1412.7709960938s, input_fired_time: 19.1927509308, output_fired_time: 8.1265602112, propagate_forward_time: 539.8521118164, propagate_backward_time: 833.2238769531, weight_update_time: 0.4310767353, other_time: 12.3756713867
epoch time: 111.3466186523s, input_fired_time: 1.2897778749, output_fired_time: 0.0000000000, propagate_forward_time: 107.9654006958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0914382935
validation[          , genome  6104] predictions:    6075/  10000 (60.75%), best:    6197/10000 (61.97%), error:     11367.01563, best error:     10767.65820 on epoch:     9, epoch:   11/50, mu: 0.7033335567, learning_rate: 0.0006910762, weight_decay: 0.0002132969

epoch time: 1416.0566406250s, input_fired_time: 19.2020263672, output_fired_time: 8.1564378738, propagate_forward_time: 540.3989868164, propagate_backward_time: 835.9871215820, weight_update_time: 0.4348262548, other_time: 12.3120117188
epoch time: 112.7129592896s, input_fired_time: 1.3057496548, output_fired_time: 0.0000000000, propagate_forward_time: 109.3118972778, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0953140259
validation[          , genome  6104] predictions:    6072/  10000 (60.72%), best:    6197/10000 (61.97%), error:     11509.91016, best error:     10767.65820 on epoch:     9, epoch:   12/50, mu: 0.7163296342, learning_rate: 0.0006313365, weight_decay: 0.0001997196

epoch time: 1422.3244628906s, input_fired_time: 19.3247089386, output_fired_time: 8.2030134201, propagate_forward_time: 542.7702636719, propagate_backward_time: 839.6318359375, weight_update_time: 0.4304764271, other_time: 12.3946533203
epoch time: 111.5942535400s, input_fired_time: 1.3127373457, output_fired_time: 0.0000000000, propagate_forward_time: 108.1972427368, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0842742920
validation[          , genome  6104] predictions:    6438/  10000 (64.38%), best:    6438/10000 (64.38%), error:     10220.60352, best error:     10220.60352 on epoch:    13, epoch:   13/50, mu: 0.7287365198, learning_rate: 0.0005767611, weight_decay: 0.0001870065

epoch time: 1410.6722412109s, input_fired_time: 19.1638126373, output_fired_time: 8.0907535553, propagate_forward_time: 538.7985839844, propagate_backward_time: 832.3199462891, weight_update_time: 0.4302916527, other_time: 12.2991943359
epoch time: 111.1077728271s, input_fired_time: 1.3130259514, output_fired_time: 0.0000000000, propagate_forward_time: 107.7105712891, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0841751099
validation[          , genome  6104] predictions:    6292/  10000 (62.92%), best:    6438/10000 (64.38%), error:     10560.50781, best error:     10220.60352 on epoch:    13, epoch:   14/50, mu: 0.7405809164, learning_rate: 0.0005269034, weight_decay: 0.0001751028

epoch time: 1412.8035888672s, input_fired_time: 19.2250499725, output_fired_time: 8.1506633759, propagate_forward_time: 539.0464477539, propagate_backward_time: 834.0445556641, weight_update_time: 0.4427803457, other_time: 12.3368530273
epoch time: 112.3506164551s, input_fired_time: 1.3199450970, output_fired_time: 0.0000000000, propagate_forward_time: 108.9024963379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1281738281
validation[          , genome  6104] predictions:    6233/  10000 (62.33%), best:    6438/10000 (64.38%), error:     10773.95996, best error:     10220.60352 on epoch:    13, epoch:   15/50, mu: 0.7518883944, learning_rate: 0.0004813556, weight_decay: 0.0001639567

epoch time: 1419.9752197266s, input_fired_time: 19.3283920288, output_fired_time: 8.1923484802, propagate_forward_time: 542.7168579102, propagate_backward_time: 837.3845214844, weight_update_time: 0.4270998836, other_time: 12.3530883789
epoch time: 111.6660995483s, input_fired_time: 1.3187630177, output_fired_time: 0.0000000000, propagate_forward_time: 108.2345123291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1128234863
validation[          , genome  6104] predictions:    6356/  10000 (63.56%), best:    6438/10000 (64.38%), error:     10231.29297, best error:     10220.60352 on epoch:    13, epoch:   16/50, mu: 0.7626832128, learning_rate: 0.0004397451, weight_decay: 0.0001535201

epoch time: 1414.9896240234s, input_fired_time: 19.2750434875, output_fired_time: 8.1945543289, propagate_forward_time: 539.9067382813, propagate_backward_time: 835.4170532227, weight_update_time: 0.4255650342, other_time: 12.1962280273
epoch time: 111.5841674805s, input_fired_time: 1.3205686808, output_fired_time: 0.0000000000, propagate_forward_time: 108.1918563843, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0717391968
validation[          , genome  6104] predictions:    6366/  10000 (63.66%), best:    6438/10000 (64.38%), error:     10297.07715, best error:     10220.60352 on epoch:    13, epoch:   17/50, mu: 0.7729886770, learning_rate: 0.0004017316, weight_decay: 0.0001437479

epoch time: 1416.8845214844s, input_fired_time: 19.2693271637, output_fired_time: 8.1801624298, propagate_forward_time: 541.1120605469, propagate_backward_time: 836.0449218750, weight_update_time: 0.4294513166, other_time: 12.2780761719
epoch time: 112.8817825317s, input_fired_time: 1.3237731457, output_fired_time: 0.0000000000, propagate_forward_time: 109.4225311279, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1354751587
validation[          , genome  6104] predictions:    6325/  10000 (63.25%), best:    6438/10000 (64.38%), error:     10539.40918, best error:     10220.60352 on epoch:    13, epoch:   18/50, mu: 0.7828269005, learning_rate: 0.0003670042, weight_decay: 0.0001345977

epoch time: 1426.0040283203s, input_fired_time: 19.4531002045, output_fired_time: 8.2620401382, propagate_forward_time: 544.2371826172, propagate_backward_time: 841.6837768555, weight_update_time: 0.4344744384, other_time: 12.3678588867
epoch time: 111.3801269531s, input_fired_time: 1.3276925087, output_fired_time: 0.0000000000, propagate_forward_time: 107.9457702637, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1066665649
validation[          , genome  6104] predictions:    6192/  10000 (61.92%), best:    6438/10000 (64.38%), error:     10919.88477, best error:     10220.60352 on epoch:    13, epoch:   19/50, mu: 0.7922191024, learning_rate: 0.0003352788, weight_decay: 0.0001260299

epoch time: 1419.5236816406s, input_fired_time: 19.3781032562, output_fired_time: 8.1508054733, propagate_forward_time: 542.5578613281, propagate_backward_time: 837.0536499023, weight_update_time: 0.4333626926, other_time: 12.3833618164
epoch time: 112.2265167236s, input_fired_time: 1.3330509663, output_fired_time: 0.0000000000, propagate_forward_time: 108.7782897949, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1151733398
validation[          , genome  6104] predictions:    6579/  10000 (65.79%), best:    6579/10000 (65.79%), error:      9961.17480, best error:      9961.17480 on epoch:    20, epoch:   20/50, mu: 0.8011855483, learning_rate: 0.0003062959, weight_decay: 0.0001180076

epoch time: 1417.3992919922s, input_fired_time: 19.3032684326, output_fired_time: 8.1891632080, propagate_forward_time: 540.9945678711, propagate_backward_time: 836.5790405273, weight_update_time: 0.4289743602, other_time: 12.3332519531
epoch time: 114.2128219604s, input_fired_time: 1.3520997763, output_fired_time: 0.0000000000, propagate_forward_time: 110.6980056763, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1627197266
validation[          , genome  6104] predictions:    6385/  10000 (63.85%), best:    6579/10000 (65.79%), error:     10321.46680, best error:      9961.17480 on epoch:    20, epoch:   21/50, mu: 0.8097454906, learning_rate: 0.0002798183, weight_decay: 0.0001104959

epoch time: 1431.2659912109s, input_fired_time: 19.5070209503, output_fired_time: 8.1972026825, propagate_forward_time: 547.7781372070, propagate_backward_time: 843.3589477539, weight_update_time: 0.4341415763, other_time: 12.4246826172
epoch time: 112.7650070190s, input_fired_time: 1.3336306810, output_fired_time: 0.0000000000, propagate_forward_time: 109.3027191162, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1286544800
validation[          , genome  6104] predictions:    6345/  10000 (63.45%), best:    6579/10000 (65.79%), error:     10421.78125, best error:      9961.17480 on epoch:    20, epoch:   22/50, mu: 0.8179173470, learning_rate: 0.0002556296, weight_decay: 0.0001034623

epoch time: 1425.6230468750s, input_fired_time: 19.4998874664, output_fired_time: 8.1482620239, propagate_forward_time: 545.5845947266, propagate_backward_time: 840.0087890625, weight_update_time: 0.4306338131, other_time: 12.3814697266
epoch time: 112.1815795898s, input_fired_time: 1.3288261890, output_fired_time: 0.0000000000, propagate_forward_time: 108.7015380859, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1512145996
validation[          , genome  6104] predictions:    6474/  10000 (64.74%), best:    6579/10000 (65.79%), error:     10136.33789, best error:      9961.17480 on epoch:    20, epoch:   23/50, mu: 0.8257187605, learning_rate: 0.0002335319, weight_decay: 0.0000968765

epoch time: 1417.6072998047s, input_fired_time: 19.3896121979, output_fired_time: 8.1374549866, propagate_forward_time: 540.4725341797, propagate_backward_time: 837.2584228516, weight_update_time: 0.4325749278, other_time: 12.3492431641
epoch time: 111.5542984009s, input_fired_time: 1.3235313892, output_fired_time: 0.0000000000, propagate_forward_time: 108.1176300049, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1131362915
validation[          , genome  6104] predictions:    6337/  10000 (63.37%), best:    6579/10000 (65.79%), error:     10471.31055, best error:      9961.17480 on epoch:    20, epoch:   24/50, mu: 0.8331664801, learning_rate: 0.0002133444, weight_decay: 0.0000907099

epoch time: 1417.9151611328s, input_fired_time: 19.3414478302, output_fired_time: 8.1179313660, propagate_forward_time: 541.3096313477, propagate_backward_time: 836.8083496094, weight_update_time: 0.4329720438, other_time: 12.3378295898
epoch time: 112.2648391724s, input_fired_time: 1.3436480761, output_fired_time: 0.0000000000, propagate_forward_time: 108.8155975342, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1055908203
validation[          , genome  6104] predictions:    6340/  10000 (63.40%), best:    6579/10000 (65.79%), error:     10473.27441, best error:      9961.17480 on epoch:    20, epoch:   25/50, mu: 0.8402765393, learning_rate: 0.0001949020, weight_decay: 0.0000849358

epoch time: 1427.3764648438s, input_fired_time: 19.4889488220, output_fired_time: 8.2107419968, propagate_forward_time: 545.1486206055, propagate_backward_time: 842.1931762695, weight_update_time: 0.4238702357, other_time: 12.3350830078
epoch time: 112.2695693970s, input_fired_time: 1.3393763304, output_fired_time: 0.0000000000, propagate_forward_time: 108.8103790283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1198120117
validation[          , genome  6104] predictions:    6321/  10000 (63.21%), best:    6579/10000 (65.79%), error:     10504.16895, best error:      9961.17480 on epoch:    20, epoch:   26/50, mu: 0.8470642567, learning_rate: 0.0001780538, weight_decay: 0.0000795293

epoch time: 1418.2065429688s, input_fired_time: 19.3906421661, output_fired_time: 8.1727113724, propagate_forward_time: 541.2965698242, propagate_backward_time: 836.9028320313, weight_update_time: 0.4356120527, other_time: 12.4437866211
epoch time: 112.1819152832s, input_fired_time: 1.3317451477, output_fired_time: 0.0000000000, propagate_forward_time: 108.7766571045, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0735168457
validation[          , genome  6104] predictions:    6525/  10000 (65.25%), best:    6525/10000 (65.25%), error:      9925.30664, best error:      9925.30664 on epoch:    27, epoch:   27/50, mu: 0.8535442948, learning_rate: 0.0001626621, weight_decay: 0.0000744669

epoch time: 1414.2092285156s, input_fired_time: 19.3783092499, output_fired_time: 8.1207962036, propagate_forward_time: 539.8441772461, propagate_backward_time: 834.5176391602, weight_update_time: 0.4314790666, other_time: 12.3482666016
epoch time: 112.4179763794s, input_fired_time: 1.3354120255, output_fired_time: 0.0000000000, propagate_forward_time: 108.9797515869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1028137207
validation[          , genome  6104] predictions:    6531/  10000 (65.31%), best:    6525/10000 (65.25%), error:     10186.86426, best error:      9925.30664 on epoch:    27, epoch:   28/50, mu: 0.8597305417, learning_rate: 0.0001486009, weight_decay: 0.0000697267

epoch time: 1428.9173583984s, input_fired_time: 19.5920238495, output_fired_time: 8.2262706757, propagate_forward_time: 545.3526611328, propagate_backward_time: 843.3179931641, weight_update_time: 0.4389173687, other_time: 12.4283447266
epoch time: 112.4287414551s, input_fired_time: 1.3499057293, output_fired_time: 0.0000000000, propagate_forward_time: 108.9479293823, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1309051514
validation[          , genome  6104] predictions:    6517/  10000 (65.17%), best:    6517/10000 (65.17%), error:      9876.65430, best error:      9876.65430 on epoch:    29, epoch:   29/50, mu: 0.8656363487, learning_rate: 0.0001357552, weight_decay: 0.0000652883

epoch time: 1422.9888916016s, input_fired_time: 19.4715442657, output_fired_time: 8.1497783661, propagate_forward_time: 543.0082397461, propagate_backward_time: 839.9722900391, weight_update_time: 0.4290544391, other_time: 12.3870239258
epoch time: 112.4652252197s, input_fired_time: 1.3510868549, output_fired_time: 0.0000000000, propagate_forward_time: 108.9951171875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1190185547
validation[          , genome  6104] predictions:    6571/  10000 (65.71%), best:    6571/10000 (65.71%), error:      9866.27148, best error:      9866.27148 on epoch:    30, epoch:   30/50, mu: 0.8712744117, learning_rate: 0.0001240199, weight_decay: 0.0000611324

epoch time: 1421.1456298828s, input_fired_time: 19.4772415161, output_fired_time: 8.1940231323, propagate_forward_time: 543.1961059570, propagate_backward_time: 837.9135742188, weight_update_time: 0.4346856773, other_time: 12.3646850586
epoch time: 113.9232406616s, input_fired_time: 1.3648279905, output_fired_time: 0.0000000000, propagate_forward_time: 110.4361572266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1222534180
validation[          , genome  6104] predictions:    6611/  10000 (66.11%), best:    6611/10000 (66.11%), error:      9695.07227, best error:      9695.07227 on epoch:    31, epoch:   31/50, mu: 0.8766568303, learning_rate: 0.0001132991, weight_decay: 0.0000572411

epoch time: 1424.7567138672s, input_fired_time: 19.5524139404, output_fired_time: 8.1183586121, propagate_forward_time: 544.9978637695, propagate_backward_time: 839.6547851563, weight_update_time: 0.4338013232, other_time: 12.4332885742
epoch time: 112.3629760742s, input_fired_time: 1.3564990759, output_fired_time: 0.0000000000, propagate_forward_time: 108.8892288208, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1172485352
validation[          , genome  6104] predictions:    6626/  10000 (66.26%), best:    6611/10000 (66.11%), error:      9761.42383, best error:      9695.07227 on epoch:    31, epoch:   32/50, mu: 0.8817952871, learning_rate: 0.0001035050, weight_decay: 0.0000535974

epoch time: 1414.1086425781s, input_fired_time: 19.4175796509, output_fired_time: 8.1078033447, propagate_forward_time: 539.6499633789, propagate_backward_time: 834.6469116211, weight_update_time: 0.4305937588, other_time: 12.2863769531
epoch time: 112.3285751343s, input_fired_time: 1.3502084017, output_fired_time: 0.0000000000, propagate_forward_time: 108.8410110474, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1373519897
validation[          , genome  6104] predictions:    6649/  10000 (66.49%), best:    6649/10000 (66.49%), error:      9634.26465, best error:      9634.26465 on epoch:    33, epoch:   33/50, mu: 0.8867007494, learning_rate: 0.0000945576, weight_decay: 0.0000501857

epoch time: 1410.2147216797s, input_fired_time: 19.3869495392, output_fired_time: 8.0810995102, propagate_forward_time: 538.9651489258, propagate_backward_time: 831.5697021484, weight_update_time: 0.4312816560, other_time: 12.2118530273
epoch time: 111.4141845703s, input_fired_time: 1.3458513021, output_fired_time: 0.0000000000, propagate_forward_time: 107.9973449707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0709915161
validation[          , genome  6104] predictions:    6572/  10000 (65.72%), best:    6649/10000 (66.49%), error:      9746.67383, best error:      9634.26465 on epoch:    33, epoch:   34/50, mu: 0.8913838267, learning_rate: 0.0000863836, weight_decay: 0.0000469912

epoch time: 1435.6199951172s, input_fired_time: 19.5620059967, output_fired_time: 8.2198371887, propagate_forward_time: 547.9159545898, propagate_backward_time: 847.4366455078, weight_update_time: 0.4341183901, other_time: 12.4855346680
epoch time: 110.4274444580s, input_fired_time: 1.3371285200, output_fired_time: 0.0000000000, propagate_forward_time: 106.9818954468, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1084213257
validation[          , genome  6104] predictions:    6609/  10000 (66.09%), best:    6649/10000 (66.49%), error:      9715.60449, best error:      9634.26465 on epoch:    33, epoch:   35/50, mu: 0.8958545923, learning_rate: 0.0000789163, weight_decay: 0.0000440000

epoch time: 1405.1004638672s, input_fired_time: 19.2964515686, output_fired_time: 8.0789270401, propagate_forward_time: 535.1144409180, propagate_backward_time: 830.3363037109, weight_update_time: 0.4325440824, other_time: 12.2742309570
epoch time: 111.4079132080s, input_fired_time: 1.3350983858, output_fired_time: 0.0000000000, propagate_forward_time: 107.9529800415, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1198348999
validation[          , genome  6104] predictions:    6697/  10000 (66.97%), best:    6697/10000 (66.97%), error:      9530.93555, best error:      9530.93555 on epoch:    36, epoch:   36/50, mu: 0.9001227021, learning_rate: 0.0000720944, weight_decay: 0.0000411992

epoch time: 1417.0792236328s, input_fired_time: 19.4485530853, output_fired_time: 8.1283702850, propagate_forward_time: 540.5159301758, propagate_backward_time: 836.6605834961, weight_update_time: 0.4256770015, other_time: 12.3256835938
epoch time: 112.4340744019s, input_fired_time: 1.3573893309, output_fired_time: 0.0000000000, propagate_forward_time: 108.9710998535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1055831909
validation[          , genome  6104] predictions:    6617/  10000 (66.17%), best:    6697/10000 (66.97%), error:      9749.89453, best error:      9530.93555 on epoch:    36, epoch:   37/50, mu: 0.9041972756, learning_rate: 0.0000658622, weight_decay: 0.0000385767

epoch time: 1431.6701660156s, input_fired_time: 19.5919322968, output_fired_time: 8.2309846878, propagate_forward_time: 546.8703002930, propagate_backward_time: 844.5359497070, weight_update_time: 0.4338734746, other_time: 12.4410400391
epoch time: 111.5104751587s, input_fired_time: 1.3575377464, output_fired_time: 0.0000000000, propagate_forward_time: 108.0613555908, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0915832520
validation[          , genome  6104] predictions:    6718/  10000 (67.18%), best:    6718/10000 (67.18%), error:      9429.76563, best error:      9429.76563 on epoch:    38, epoch:   38/50, mu: 0.9080871344, learning_rate: 0.0000601688, weight_decay: 0.0000361211

epoch time: 1412.0478515625s, input_fired_time: 19.4033889771, output_fired_time: 8.0975780487, propagate_forward_time: 540.2104492188, propagate_backward_time: 832.1264648438, weight_update_time: 0.4278800488, other_time: 12.2099609375
epoch time: 111.0162277222s, input_fired_time: 1.3467446566, output_fired_time: 0.0000000000, propagate_forward_time: 107.5758743286, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0936050415
validation[          , genome  6104] predictions:    6702/  10000 (67.02%), best:    6718/10000 (67.18%), error:      9509.35742, best error:      9429.76563 on epoch:    38, epoch:   39/50, mu: 0.9118006825, learning_rate: 0.0000549676, weight_decay: 0.0000338218

epoch time: 1407.9793701172s, input_fired_time: 19.3477458954, output_fired_time: 8.0842180252, propagate_forward_time: 537.4194946289, propagate_backward_time: 830.9418945313, weight_update_time: 0.4274148643, other_time: 12.1859741211
epoch time: 111.0135879517s, input_fired_time: 1.3490780592, output_fired_time: 0.0000000000, propagate_forward_time: 107.5664672852, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0980453491
validation[          , genome  6104] predictions:    6691/  10000 (66.91%), best:    6718/10000 (67.18%), error:      9517.79492, best error:      9429.76563 on epoch:    38, epoch:   40/50, mu: 0.9153458476, learning_rate: 0.0000502159, weight_decay: 0.0000316689

epoch time: 1417.9117431641s, input_fired_time: 19.5023841858, output_fired_time: 8.1528873444, propagate_forward_time: 541.3143310547, propagate_backward_time: 836.5300292969, weight_update_time: 0.4382040501, other_time: 12.4121093750
epoch time: 111.4897613525s, input_fired_time: 1.3474485874, output_fired_time: 0.0000000000, propagate_forward_time: 108.0179595947, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1243515015
validation[          , genome  6104] predictions:    6733/  10000 (67.33%), best:    6718/10000 (67.18%), error:      9441.39258, best error:      9429.76563 on epoch:    38, epoch:   41/50, mu: 0.9187303185, learning_rate: 0.0000458751, weight_decay: 0.0000296531

epoch time: 1401.7958984375s, input_fired_time: 19.3289089203, output_fired_time: 8.0763454437, propagate_forward_time: 535.2197875977, propagate_backward_time: 827.0330200195, weight_update_time: 0.4236812294, other_time: 12.1379394531
epoch time: 109.1637496948s, input_fired_time: 1.3272261620, output_fired_time: 0.0000000000, propagate_forward_time: 105.7994766235, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0370483398
validation[          , genome  6104] predictions:    6735/  10000 (67.35%), best:    6735/10000 (67.35%), error:      9398.12793, best error:      9398.12793 on epoch:    42, epoch:   42/50, mu: 0.9219613671, learning_rate: 0.0000419094, weight_decay: 0.0000277655

epoch time: 1403.7878417969s, input_fired_time: 19.3586120605, output_fired_time: 8.0649814606, propagate_forward_time: 536.3562011719, propagate_backward_time: 827.8836059570, weight_update_time: 0.4281864464, other_time: 12.1244506836
epoch time: 110.8080902100s, input_fired_time: 1.3380639553, output_fired_time: 0.0000000000, propagate_forward_time: 107.3625030518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1075210571
validation[          , genome  6104] predictions:    6682/  10000 (66.82%), best:    6735/10000 (67.35%), error:      9587.67480, best error:      9398.12793 on epoch:    42, epoch:   43/50, mu: 0.9250459075, learning_rate: 0.0000382866, weight_decay: 0.0000259981

epoch time: 1414.6182861328s, input_fired_time: 19.4549350739, output_fired_time: 8.1305322647, propagate_forward_time: 539.7177124023, propagate_backward_time: 835.1270751953, weight_update_time: 0.4251818061, other_time: 12.1880493164
epoch time: 111.9026031494s, input_fired_time: 1.3586091995, output_fired_time: 0.0000000000, propagate_forward_time: 108.4472351074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0967559814
validation[          , genome  6104] predictions:    6731/  10000 (67.31%), best:    6735/10000 (67.35%), error:      9477.17578, best error:      9398.12793 on epoch:    42, epoch:   44/50, mu: 0.9279906154, learning_rate: 0.0000349769, weight_decay: 0.0000243432

epoch time: 1411.1936035156s, input_fired_time: 19.4251976013, output_fired_time: 8.1260900497, propagate_forward_time: 539.1941528320, propagate_backward_time: 832.1441650391, weight_update_time: 0.4350956678, other_time: 12.3040161133
epoch time: 110.2700653076s, input_fired_time: 1.3530783653, output_fired_time: 0.0000000000, propagate_forward_time: 106.8499526978, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0670318604
validation[          , genome  6104] predictions:    6715/  10000 (67.15%), best:    6735/10000 (67.35%), error:      9441.52637, best error:      9398.12793 on epoch:    42, epoch:   45/50, mu: 0.9308018088, learning_rate: 0.0000319534, weight_decay: 0.0000227937

epoch time: 1407.9975585938s, input_fired_time: 19.3660488129, output_fired_time: 8.0722780228, propagate_forward_time: 538.1816406250, propagate_backward_time: 830.1504516602, weight_update_time: 0.4228377938, other_time: 12.2271118164
epoch time: 150.2339782715s, input_fired_time: 1.7162057161, output_fired_time: 0.0000000000, propagate_forward_time: 146.0067138672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5110626221
validation[          , genome  6104] predictions:    6750/  10000 (67.50%), best:    6750/10000 (67.50%), error:      9369.13867, best error:      9369.13867 on epoch:    46, epoch:   46/50, mu: 0.9334855676, learning_rate: 0.0000291912, weight_decay: 0.0000213427

epoch time: 1879.7762451172s, input_fired_time: 22.9631977081, output_fired_time: 10.5831317902, propagate_forward_time: 727.0541992188, propagate_backward_time: 1104.4510498047, weight_update_time: 0.5510995984, other_time: 14.7246093750
epoch time: 149.0993041992s, input_fired_time: 1.7147214413, output_fired_time: 0.0000000000, propagate_forward_time: 144.8524780273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5321044922
validation[          , genome  6104] predictions:    6723/  10000 (67.23%), best:    6750/10000 (67.50%), error:      9438.68359, best error:      9369.13867 on epoch:    46, epoch:   47/50, mu: 0.9360476732, learning_rate: 0.0000266678, weight_decay: 0.0000199842

epoch time: 1857.7180175781s, input_fired_time: 22.7679233551, output_fired_time: 10.4580488205, propagate_forward_time: 718.9506225586, propagate_backward_time: 1091.0520019531, weight_update_time: 0.5409286618, other_time: 14.4895019531
epoch time: 149.2230834961s, input_fired_time: 1.7263453007, output_fired_time: 0.0000000000, propagate_forward_time: 145.0300598145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4666748047
validation[          , genome  6104] predictions:    6776/  10000 (67.76%), best:    6750/10000 (67.50%), error:      9393.16309, best error:      9369.13867 on epoch:    46, epoch:   48/50, mu: 0.9384936094, learning_rate: 0.0000243625, weight_decay: 0.0000187121

epoch time: 1663.6502685547s, input_fired_time: 21.3652915955, output_fired_time: 9.4540176392, propagate_forward_time: 640.6090087891, propagate_backward_time: 978.7279663086, weight_update_time: 0.4904112816, other_time: 13.4940795898
epoch time: 110.3391418457s, input_fired_time: 1.3484147787, output_fired_time: 0.0000000000, propagate_forward_time: 106.9325790405, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0581512451
validation[          , genome  6104] predictions:    6716/  10000 (67.16%), best:    6716/10000 (67.16%), error:      9364.96680, best error:      9364.96680 on epoch:    49, epoch:   49/50, mu: 0.9408286810, learning_rate: 0.0000222565, weight_decay: 0.0000175210

epoch time: 1533.7022705078s, input_fired_time: 20.4973564148, output_fired_time: 8.7292728424, propagate_forward_time: 588.9576416016, propagate_backward_time: 902.2666625977, weight_update_time: 0.4743999839, other_time: 13.2514038086
epoch time: 130.1011352539s, input_fired_time: 1.5554487705, output_fired_time: 0.0000000000, propagate_forward_time: 126.1873168945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3583679199
validation[          , genome  6104] predictions:    6756/  10000 (67.56%), best:    6756/10000 (67.56%), error:      9350.97852, best error:      9350.97852 on epoch:    50, epoch:   50/50, mu: 0.9430578947, learning_rate: 0.0000203325, weight_decay: 0.0000164057

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 673.1708374023s, input_fired_time: 7.9979562759, output_fired_time: 0.0000000000, propagate_forward_time: 652.6327514648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.5401000977
best training[          , genome  6104] predictions:   37397/  50000 (74.79%), best:    6756/10000 (67.56%), error:     38121.75000, best error:      9350.97852 on epoch:    50, epoch:   51/50, mu: 0.9451860189, learning_rate: 0.0000185749, weight_decay: 0.0000153614
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 138.4420166016s, input_fired_time: 1.6356134415, output_fired_time: 0.0000000000, propagate_forward_time: 134.2125091553, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5938873291
   testing[          , genome  6104] predictions:    6756/  10000 (67.56%), best:    6756/10000 (67.56%), error:      9350.97852, best error:      9350.97852 on epoch:    50, epoch:   51/50, mu: 0.9451860189, learning_rate: 0.0000185749, weight_decay: 0.0000153614
backpropagation finished successfully!
23:19:35 (11128): called boinc_finish(0)

</stderr_txt>
]]>