Nameexact_genome_1542031203_186_5582_0
Workunit4081010
Created12 Nov 2018, 17:49:54 UTC
Sent12 Nov 2018, 18:02:07 UTC
Report deadline7 Mar 2019, 1:58:39 UTC
Received16 Nov 2018, 12:41:38 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID62094
Run time1 days 2 hours 4 min 36 sec
CPU time1 days 2 hours 4 min 36 sec
Validate stateInitial
Credit0.00
Device peak FLOPS5.53 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size552.52 MB
Peak swap size572.34 MB
Peak disk usage11.64 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
ing_rate: 0.0000213011, weight_decay: 0.0000672339

epoch time: 1816.8225097656s, input_fired_time: 27.7343845367, output_fired_time: 12.2424268723, propagate_forward_time: 585.7951049805, propagate_backward_time: 1175.4024658203, weight_update_time: 0.2320622653, other_time: 15.6480712891
epoch time: 120.5157394409s, input_fired_time: 0.8339213133, output_fired_time: 0.0000000000, propagate_forward_time: 118.5474319458, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1343841553
validation[          , genome  5582] predictions:    6167/  10000 (61.67%), best:    6215/10000 (62.15%), error:     11232.81348, best error:     11141.57813 on epoch:    39, epoch:   42/100, mu: 0.9523416758, learning_rate: 0.0000199847, weight_decay: 0.0000640130

epoch time: 1802.8991699219s, input_fired_time: 29.7518482208, output_fired_time: 12.0909957886, propagate_forward_time: 586.8935546875, propagate_backward_time: 1158.5679931641, weight_update_time: 0.2312413305, other_time: 15.5948486328
epoch time: 121.9871368408s, input_fired_time: 0.8268057108, output_fired_time: 0.0000000000, propagate_forward_time: 119.0578918457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1024398804
validation[          , genome  5582] predictions:    6270/  10000 (62.70%), best:    6270/10000 (62.70%), error:     11070.72754, best error:     11070.72754 on epoch:    43, epoch:   43/100, mu: 0.9545211196, learning_rate: 0.0000187497, weight_decay: 0.0000609465

epoch time: 1817.4528808594s, input_fired_time: 28.5443744659, output_fired_time: 14.2134389877, propagate_forward_time: 601.8026123047, propagate_backward_time: 1158.5653076172, weight_update_time: 0.2276926041, other_time: 14.3271484375
epoch time: 123.5592727661s, input_fired_time: 2.8525578976, output_fired_time: 0.0000000000, propagate_forward_time: 118.5750427246, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1316757202
validation[          , genome  5582] predictions:    6217/  10000 (62.17%), best:    6270/10000 (62.70%), error:     11152.14746, best error:     11070.72754 on epoch:    43, epoch:   44/100, mu: 0.9565744400, learning_rate: 0.0000175911, weight_decay: 0.0000580268

epoch time: 1798.6407470703s, input_fired_time: 28.5394515991, output_fired_time: 8.1670856476, propagate_forward_time: 570.3463134766, propagate_backward_time: 1182.0588378906, weight_update_time: 0.2269391418, other_time: 9.5290527344
epoch time: 119.9920883179s, input_fired_time: 0.8413212895, output_fired_time: 0.0000000000, propagate_forward_time: 114.0039520264, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.1468124390
validation[          , genome  5582] predictions:    6255/  10000 (62.55%), best:    6255/10000 (62.55%), error:     11048.96973, best error:     11048.96973 on epoch:    45, epoch:   45/100, mu: 0.9585089087, learning_rate: 0.0000165040, weight_decay: 0.0000552470

epoch time: 1802.4992675781s, input_fired_time: 28.8757266998, output_fired_time: 8.6823320389, propagate_forward_time: 597.6870117188, propagate_backward_time: 1155.5867919922, weight_update_time: 2.2460334301, other_time: 11.6673583984
epoch time: 122.4358520508s, input_fired_time: 1.8618215322, output_fired_time: 0.0000000000, propagate_forward_time: 119.4731445313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1008834839
validation[          , genome  5582] predictions:    6181/  10000 (61.81%), best:    6255/10000 (62.55%), error:     11209.04980, best error:     11048.96973 on epoch:    45, epoch:   46/100, mu: 0.9603314400, learning_rate: 0.0000154841, weight_decay: 0.0000526004

epoch time: 1835.1435546875s, input_fired_time: 26.9354934692, output_fired_time: 9.3870077133, propagate_forward_time: 610.3345336914, propagate_backward_time: 1174.8796386719, weight_update_time: 0.2311854362, other_time: 13.6069335938
epoch time: 121.9301300049s, input_fired_time: 2.8536741734, output_fired_time: 0.0000000000, propagate_forward_time: 116.9967727661, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0796813965
validation[          , genome  5582] predictions:    6278/  10000 (62.78%), best:    6278/10000 (62.78%), error:     10976.70801, best error:     10976.70801 on epoch:    47, epoch:   47/100, mu: 0.9620485306, learning_rate: 0.0000145272, weight_decay: 0.0000500806

epoch time: 1862.7856445313s, input_fired_time: 29.9034996033, output_fired_time: 12.5920352936, propagate_forward_time: 621.7022094727, propagate_backward_time: 1181.6180419922, weight_update_time: 1.2535727024, other_time: 16.9698486328
epoch time: 120.5081329346s, input_fired_time: 0.8397954106, output_fired_time: 0.0000000000, propagate_forward_time: 117.5398330688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1285018921
validation[          , genome  5582] predictions:    6285/  10000 (62.85%), best:    6285/10000 (62.85%), error:     10975.13184, best error:     10975.13184 on epoch:    48, epoch:   48/100, mu: 0.9636662006, learning_rate: 0.0000136295, weight_decay: 0.0000476814

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582_0_r505125911_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.529426
read mu: 0.96519
read mu_delta: 0.942125
read initial_learning_rate: 0.000291229
read learning_rate: 1.27872e-05
read learning_rate_delta: 0.938203
read initial_weight_decay: 0.000503154
read weight_decay: 4.53972e-05
read weight_decay_delta: 0.952095
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0776802
read input_dropout_probability: 0.00118188
read hidden_dropout_probability: 0.105026
read velocity_reset: 1916
read epoch: 49
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 48
read number_validation_images: 10000
read best_validation_predictions: 6285
read best_validation_error: 10975.1
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5582
read normal distribution: '1 0 0'
generator_str: '1684772930'
read generator: 1684772930
read generated_by_map:
1 crossover 1
reading 90 nodes.
reading 336 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1725.02795410156s, input_fired_time: 27.6096973419189, output_fired_time: 8.89001941680908, propagate_forward_time: 541.283996582031, propagate_backward_time: 1131.02587890625, weight_update_time: 0.217872351408005, other_time: 16.21826171875
epoch time: 111.57984161377s, input_fired_time: 1.79489636421204, output_fired_time: 0, propagate_forward_time: 107.777984619141, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.0069580078125
validation[          , genome  5582] predictions:    6293/  10000 (62.93%), best:    6285/10000 (62.85%), error:     11000.90430, best error:     10975.13184 on epoch:    48, epoch:   49/100, mu: 0.9651902914, learning_rate: 0.0000127872, weight_decay: 0.0000453972

epoch time: 1710.6105957031s, input_fired_time: 25.4256610870, output_fired_time: 7.7946286201, propagate_forward_time: 529.4310302734, propagate_backward_time: 1134.7261962891, weight_update_time: 0.2122723013, other_time: 13.2330322266
epoch time: 112.7677307129s, input_fired_time: 0.7995075583, output_fired_time: 0.0000000000, propagate_forward_time: 110.9408035278, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0274200439
validation[          , genome  5582] predictions:    6225/  10000 (62.25%), best:    6285/10000 (62.85%), error:     11017.64648, best error:     10975.13184 on epoch:    48, epoch:   50/100, mu: 0.9666261673, learning_rate: 0.0000119970, weight_decay: 0.0000432225

epoch time: 1729.5738525391s, input_fired_time: 29.6276397705, output_fired_time: 9.9061050415, propagate_forward_time: 545.2197875977, propagate_backward_time: 1133.5317382813, weight_update_time: 1.1259889603, other_time: 11.2885742188
epoch time: 110.8771591187s, input_fired_time: 1.6976679564, output_fired_time: 0.0000000000, propagate_forward_time: 108.0891647339, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0903244019
validation[          , genome  5582] predictions:    6263/  10000 (62.63%), best:    6285/10000 (62.85%), error:     10977.52637, best error:     10975.13184 on epoch:    48, epoch:   51/100, mu: 0.9679788947, learning_rate: 0.0000112556, weight_decay: 0.0000411519

epoch time: 1713.5186767578s, input_fired_time: 25.4320545197, output_fired_time: 8.8863439560, propagate_forward_time: 536.4851684570, propagate_backward_time: 1129.4160156250, weight_update_time: 0.2141381055, other_time: 13.2990722656
epoch time: 108.3521881104s, input_fired_time: 0.7937124968, output_fired_time: 0.0000000000, propagate_forward_time: 105.5184402466, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0400390625
validation[          , genome  5582] predictions:    6292/  10000 (62.92%), best:    6292/10000 (62.92%), error:     10898.40332, best error:     10898.40332 on epoch:    52, epoch:   52/100, mu: 0.9692533612, learning_rate: 0.0000105601, weight_decay: 0.0000391805

epoch time: 1717.4986572266s, input_fired_time: 25.5220146179, output_fired_time: 9.8960361481, propagate_forward_time: 538.9693603516, propagate_backward_time: 1127.7518310547, weight_update_time: 0.2162616402, other_time: 15.3594970703
epoch time: 111.5811309814s, input_fired_time: 0.7945961952, output_fired_time: 0.0000000000, propagate_forward_time: 109.7384185791, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0481185913
validation[          , genome  5582] predictions:    6220/  10000 (62.20%), best:    6292/10000 (62.92%), error:     10991.42090, best error:     10898.40332 on epoch:    52, epoch:   53/100, mu: 0.9704540968, learning_rate: 0.0000099075, weight_decay: 0.0000373035

epoch time: 1730.2707519531s, input_fired_time: 35.6557426453, output_fired_time: 6.8819627762, propagate_forward_time: 531.9680786133, propagate_backward_time: 1137.3885498047, weight_update_time: 0.2189551443, other_time: 18.3763427734
epoch time: 113.8326568604s, input_fired_time: 3.8210437298, output_fired_time: 0.0000000000, propagate_forward_time: 107.9406738281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0709381104
validation[          , genome  5582] predictions:    6257/  10000 (62.57%), best:    6292/10000 (62.92%), error:     10959.83594, best error:     10898.40332 on epoch:    52, epoch:   54/100, mu: 0.9715853333, learning_rate: 0.0000092952, weight_decay: 0.0000355165

epoch time: 1733.2440185547s, input_fired_time: 22.4814147949, output_fired_time: 10.8675880432, propagate_forward_time: 545.4791870117, propagate_backward_time: 1135.8865966797, weight_update_time: 0.2236013561, other_time: 18.5291748047
epoch time: 113.6771011353s, input_fired_time: 0.8073398471, output_fired_time: 0.0000000000, propagate_forward_time: 111.8074417114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0623168945
validation[          , genome  5582] predictions:    6266/  10000 (62.66%), best:    6292/10000 (62.92%), error:     11019.94727, best error:     10898.40332 on epoch:    52, epoch:   55/100, mu: 0.9726510644, learning_rate: 0.0000087208, weight_decay: 0.0000338151

epoch time: 1731.4199218750s, input_fired_time: 28.3395538330, output_fired_time: 9.8616638184, propagate_forward_time: 539.5608520508, propagate_backward_time: 1140.3248291016, weight_update_time: 0.2166164964, other_time: 13.3328857422
epoch time: 112.2842559814s, input_fired_time: 2.8118185997, output_fired_time: 0.0000000000, propagate_forward_time: 105.3982543945, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0741806030
validation[          , genome  5582] predictions:    6229/  10000 (62.29%), best:    6292/10000 (62.92%), error:     10978.60059, best error:     10898.40332 on epoch:    52, epoch:   56/100, mu: 0.9736551046, learning_rate: 0.0000081819, weight_decay: 0.0000321952

epoch time: 1752.9069824219s, input_fired_time: 27.6814785004, output_fired_time: 11.9416732788, propagate_forward_time: 553.2133178711, propagate_backward_time: 1147.8200683594, weight_update_time: 0.2130878419, other_time: 12.2504882813
epoch time: 118.0145568848s, input_fired_time: 0.8293415308, output_fired_time: 0.0000000000, propagate_forward_time: 116.1025695801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0826492310
validation[          , genome  5582] predictions:    6317/  10000 (63.17%), best:    6317/10000 (63.17%), error:     10886.86621, best error:     10886.86621 on epoch:    57, epoch:   57/100, mu: 0.9746010900, learning_rate: 0.0000076763, weight_decay: 0.0000306528

epoch time: 1835.1634521484s, input_fired_time: 27.8467922211, output_fired_time: 9.1698322296, propagate_forward_time: 572.0097656250, propagate_backward_time: 1201.4733886719, weight_update_time: 0.2252392322, other_time: 24.6636962891
epoch time: 116.4923019409s, input_fired_time: 1.8258581161, output_fired_time: 0.0000000000, propagate_forward_time: 113.5740737915, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0923690796
validation[          , genome  5582] predictions:    6296/  10000 (62.96%), best:    6317/10000 (63.17%), error:     10899.15137, best error:     10886.86621 on epoch:    57, epoch:   58/100, mu: 0.9754922986, learning_rate: 0.0000072019, weight_decay: 0.0000291844

epoch time: 1867.5447998047s, input_fired_time: 29.8615245819, output_fired_time: 10.1832828522, propagate_forward_time: 601.4116821289, propagate_backward_time: 1214.3729248047, weight_update_time: 0.2267405987, other_time: 11.7154541016
epoch time: 129.6438140869s, input_fired_time: 0.8389915824, output_fired_time: 0.0000000000, propagate_forward_time: 125.6859741211, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1188507080
validation[          , genome  5582] predictions:    6311/  10000 (63.11%), best:    6311/10000 (63.11%), error:     10846.73926, best error:     10846.73926 on epoch:    59, epoch:   59/100, mu: 0.9763319492, learning_rate: 0.0000067568, weight_decay: 0.0000277863

epoch time: 1800.4136962891s, input_fired_time: 27.7657318115, output_fired_time: 10.1689176559, propagate_forward_time: 570.9121093750, propagate_backward_time: 1180.9941406250, weight_update_time: 0.2234331518, other_time: 10.5727539063
epoch time: 117.8601074219s, input_fired_time: 0.8282082677, output_fired_time: 0.0000000000, propagate_forward_time: 115.9501190186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0817794800
validation[          , genome  5582] predictions:    6301/  10000 (63.01%), best:    6311/10000 (63.11%), error:     10871.69043, best error:     10846.73926 on epoch:    59, epoch:   60/100, mu: 0.9771229625, learning_rate: 0.0000063393, weight_decay: 0.0000264552

epoch time: 1822.7694091797s, input_fired_time: 24.6262302399, output_fired_time: 11.0973176956, propagate_forward_time: 557.8272094727, propagate_backward_time: 1214.8232421875, weight_update_time: 0.2202670425, other_time: 14.3955078125
epoch time: 120.2910614014s, input_fired_time: 1.8447486162, output_fired_time: 0.0000000000, propagate_forward_time: 116.3187103271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1276016235
validation[          , genome  5582] predictions:    6225/  10000 (62.25%), best:    6311/10000 (63.11%), error:     10946.44531, best error:     10846.73926 on epoch:    59, epoch:   61/100, mu: 0.9778681993, learning_rate: 0.0000059475, weight_decay: 0.0000251879

epoch time: 1771.1386718750s, input_fired_time: 25.7503700256, output_fired_time: 13.9428720474, propagate_forward_time: 559.1427612305, propagate_backward_time: 1156.0126953125, weight_update_time: 0.2182050347, other_time: 16.2900390625
epoch time: 129.9897003174s, input_fired_time: 0.8408384323, output_fired_time: 0.0000000000, propagate_forward_time: 125.0585784912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0902862549
validation[          , genome  5582] predictions:    6306/  10000 (63.06%), best:    6311/10000 (63.11%), error:     10890.83984, best error:     10846.73926 on epoch:    59, epoch:   62/100, mu: 0.9785703421, learning_rate: 0.0000055800, weight_decay: 0.0000239812

epoch time: 1883.7692871094s, input_fired_time: 30.2277278900, output_fired_time: 11.4140443802, propagate_forward_time: 631.0908203125, propagate_backward_time: 1197.0979003906, weight_update_time: 1.2469400167, other_time: 13.9387207031
epoch time: 131.7605285645s, input_fired_time: 2.9170908928, output_fired_time: 0.0000000000, propagate_forward_time: 125.6446533203, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1987915039
validation[          , genome  5582] predictions:    6347/  10000 (63.47%), best:    6347/10000 (63.47%), error:     10840.84668, best error:     10840.84668 on epoch:    63, epoch:   63/100, mu: 0.9792318344, learning_rate: 0.0000052352, weight_decay: 0.0000228324

epoch time: 1905.6879882813s, input_fired_time: 29.1348400116, output_fired_time: 11.5712194443, propagate_forward_time: 624.6703491211, propagate_backward_time: 1226.1755371094, weight_update_time: 0.2464829385, other_time: 14.1359863281
epoch time: 128.7881469727s, input_fired_time: 0.8965492845, output_fired_time: 0.0000000000, propagate_forward_time: 125.7029647827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1886291504
validation[          , genome  5582] predictions:    6342/  10000 (63.42%), best:    6342/10000 (63.42%), error:     10798.87305, best error:     10798.87305 on epoch:    64, epoch:   64/100, mu: 0.9798550606, learning_rate: 0.0000049116, weight_decay: 0.0000217386

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582_0_r505125911_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.529426
read mu: 0.980442
read mu_delta: 0.942125
read initial_learning_rate: 0.000291229
read learning_rate: 4.60812e-06
read learning_rate_delta: 0.938203
read initial_weight_decay: 0.000503154
read weight_decay: 2.06972e-05
read weight_decay_delta: 0.952095
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0776802
read input_dropout_probability: 0.00118188
read hidden_dropout_probability: 0.105026
read velocity_reset: 1916
read epoch: 65
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 64
read number_validation_images: 10000
read best_validation_predictions: 6342
read best_validation_error: 10798.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5582
read normal distribution: '1 0 0'
generator_str: '1606662666'
read generator: 1606662666
read generated_by_map:
1 crossover 1
reading 90 nodes.
reading 336 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1619.53540039063s, input_fired_time: 32.3957023620605, output_fired_time: 8.44886589050293, propagate_forward_time: 478.439727783203, propagate_backward_time: 1084.31420898438, weight_update_time: 0.202707797288895, other_time: 15.9368896484375
epoch time: 99.6597442626953s, input_fired_time: 1.76609003543854, output_fired_time: 0, propagate_forward_time: 95.9511108398438, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.94254302978516
validation[          , genome  5582] predictions:    6302/  10000 (63.02%), best:    6342/10000 (63.42%), error:     10831.04004, best error:     10798.87305 on epoch:    64, epoch:   65/100, mu: 0.9804422259, learning_rate: 0.0000046081, weight_decay: 0.0000206972

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582_0_r505125911_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.529426
read mu: 0.980995
read mu_delta: 0.942125
read initial_learning_rate: 0.000291229
read learning_rate: 4.32335e-06
read learning_rate_delta: 0.938203
read initial_weight_decay: 0.000503154
read weight_decay: 1.97057e-05
read weight_decay_delta: 0.952095
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0776802
read input_dropout_probability: 0.00118188
read hidden_dropout_probability: 0.105026
read velocity_reset: 1916
read epoch: 66
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 64
read number_validation_images: 10000
read best_validation_predictions: 6342
read best_validation_error: 10798.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5582
read normal distribution: '1 0 0'
generator_str: '795349868'
read generator: 795349868
read generated_by_map:
1 crossover 1
reading 90 nodes.
reading 336 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1765.60339355469s, input_fired_time: 30.6468677520752, output_fired_time: 10.0267763137817, propagate_forward_time: 556.828125, propagate_backward_time: 1154.75817871094, weight_update_time: 0.224111422896385, other_time: 13.343505859375
epoch time: 108.602005004883s, input_fired_time: 1.84286785125732, output_fired_time: 0, propagate_forward_time: 104.79069519043, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.96844482421875
validation[          , genome  5582] predictions:    6318/  10000 (63.18%), best:    6342/10000 (63.42%), error:     10862.71289, best error:     10798.87305 on epoch:    64, epoch:   66/100, mu: 0.9809953570, learning_rate: 0.0000043234, weight_decay: 0.0000197057

epoch time: 1692.8590087891s, input_fired_time: 20.8469734192, output_fired_time: 14.7395429611, propagate_forward_time: 523.2048339844, propagate_backward_time: 1121.8707275391, weight_update_time: 1.1645128727, other_time: 12.1970214844
epoch time: 110.5211257935s, input_fired_time: 0.7957544923, output_fired_time: 0.0000000000, propagate_forward_time: 107.6393203735, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0860519409
validation[          , genome  5582] predictions:    6314/  10000 (63.14%), best:    6342/10000 (63.42%), error:     10858.50488, best error:     10798.87305 on epoch:    64, epoch:   67/100, mu: 0.9815164804, learning_rate: 0.0000040562, weight_decay: 0.0000187617

epoch time: 1687.2863769531s, input_fired_time: 24.5348262787, output_fired_time: 8.5794792175, propagate_forward_time: 523.3375244141, propagate_backward_time: 1115.7009277344, weight_update_time: 0.2122976035, other_time: 15.1336669922
epoch time: 109.9571228027s, input_fired_time: 1.7597503662, output_fired_time: 0.0000000000, propagate_forward_time: 106.2085952759, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9887771606
validation[          , genome  5582] predictions:    6365/  10000 (63.65%), best:    6342/10000 (63.42%), error:     10845.70020, best error:     10798.87305 on epoch:    64, epoch:   68/100, mu: 0.9820074439, learning_rate: 0.0000038055, weight_decay: 0.0000178629

epoch time: 1702.2825927734s, input_fired_time: 26.3749561310, output_fired_time: 8.8790264130, propagate_forward_time: 523.8934326172, propagate_backward_time: 1127.9875488281, weight_update_time: 1.1942447424, other_time: 15.1475830078
epoch time: 111.8303527832s, input_fired_time: 0.7945012450, output_fired_time: 0.0000000000, propagate_forward_time: 110.0091781616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0266723633
validation[          , genome  5582] predictions:    6343/  10000 (63.43%), best:    6342/10000 (63.42%), error:     10924.40234, best error:     10798.87305 on epoch:    64, epoch:   69/100, mu: 0.9824700356, learning_rate: 0.0000035704, weight_decay: 0.0000170072

epoch time: 1700.8341064453s, input_fired_time: 27.3046474457, output_fired_time: 10.8376665115, propagate_forward_time: 525.5164794922, propagate_backward_time: 1124.9302978516, weight_update_time: 0.2132713646, other_time: 12.2449951172
epoch time: 108.7051849365s, input_fired_time: 1.8720065355, output_fired_time: 0.0000000000, propagate_forward_time: 105.8420944214, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9910812378
validation[          , genome  5582] predictions:    6344/  10000 (63.44%), best:    6344/10000 (63.44%), error:     10779.28223, best error:     10779.28223 on epoch:    70, epoch:   70/100, mu: 0.9829058051, learning_rate: 0.0000033497, weight_decay: 0.0000161925

epoch time: 1694.1630859375s, input_fired_time: 23.4830131531, output_fired_time: 6.7663407326, propagate_forward_time: 525.4901123047, propagate_backward_time: 1127.4016113281, weight_update_time: 0.2117772996, other_time: 11.0219726563
epoch time: 107.2012252808s, input_fired_time: 0.7979080677, output_fired_time: 0.0000000000, propagate_forward_time: 104.3311080933, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0722122192
validation[          , genome  5582] predictions:    6330/  10000 (63.30%), best:    6344/10000 (63.44%), error:     10854.56445, best error:     10779.28223 on epoch:    70, epoch:   71/100, mu: 0.9833163619, learning_rate: 0.0000031427, weight_decay: 0.0000154168

epoch time: 1678.0869140625s, input_fired_time: 28.5535068512, output_fired_time: 9.5969467163, propagate_forward_time: 513.0516357422, propagate_backward_time: 1114.7114257813, weight_update_time: 0.2076518238, other_time: 12.1734619141
epoch time: 108.6620712280s, input_fired_time: 1.7530696392, output_fired_time: 0.0000000000, propagate_forward_time: 105.9196548462, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9893493652
validation[          , genome  5582] predictions:    6356/  10000 (63.56%), best:    6344/10000 (63.44%), error:     10804.76953, best error:     10779.28223 on epoch:    70, epoch:   72/100, mu: 0.9837031960, learning_rate: 0.0000029485, weight_decay: 0.0000146782

epoch time: 1694.2797851563s, input_fired_time: 23.4472656250, output_fired_time: 11.8868618011, propagate_forward_time: 525.4303588867, propagate_backward_time: 1123.4555664063, weight_update_time: 0.2069135010, other_time: 10.0598144531
epoch time: 110.6385650635s, input_fired_time: 2.7150785923, output_fired_time: 0.0000000000, propagate_forward_time: 105.9506149292, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9728698730
validation[          , genome  5582] predictions:    6406/  10000 (64.06%), best:    6406/10000 (64.06%), error:     10773.94727, best error:     10773.94727 on epoch:    73, epoch:   73/100, mu: 0.9840676188, learning_rate: 0.0000027663, weight_decay: 0.0000139750

epoch time: 1685.3314208984s, input_fired_time: 22.5068321228, output_fired_time: 14.0305480957, propagate_forward_time: 525.2430419922, propagate_backward_time: 1107.4639892578, weight_update_time: 1.1635466814, other_time: 16.0870361328
epoch time: 109.0147094727s, input_fired_time: 4.8814783096, output_fired_time: 0.0000000000, propagate_forward_time: 103.1102600098, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0229721069
validation[          , genome  5582] predictions:    6401/  10000 (64.01%), best:    6401/10000 (64.01%), error:     10756.84668, best error:     10756.84668 on epoch:    74, epoch:   74/100, mu: 0.9844109416, learning_rate: 0.0000025953, weight_decay: 0.0000133056

epoch time: 1699.2873535156s, input_fired_time: 22.4875793457, output_fired_time: 8.8320932388, propagate_forward_time: 534.9754638672, propagate_backward_time: 1119.8750000000, weight_update_time: 1.1851841211, other_time: 13.1171875000
epoch time: 108.6363830566s, input_fired_time: 2.7978630066, output_fired_time: 0.0000000000, propagate_forward_time: 104.8446884155, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9938278198
validation[          , genome  5582] predictions:    6347/  10000 (63.47%), best:    6347/10000 (63.47%), error:     10754.75391, best error:     10754.75391 on epoch:    75, epoch:   75/100, mu: 0.9847344160, learning_rate: 0.0000024350, weight_decay: 0.0000126682

epoch time: 1674.1156005859s, input_fired_time: 27.6988410950, output_fired_time: 9.6594686508, propagate_forward_time: 521.6785888672, propagate_backward_time: 1103.1241455078, weight_update_time: 0.2097412348, other_time: 11.9545898438
epoch time: 108.9369583130s, input_fired_time: 2.8141021729, output_fired_time: 0.0000000000, propagate_forward_time: 104.1459350586, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9769210815
validation[          , genome  5582] predictions:    6329/  10000 (63.29%), best:    6347/10000 (63.47%), error:     10759.70508, best error:     10754.75391 on epoch:    75, epoch:   76/100, mu: 0.9850391746, learning_rate: 0.0000022845, weight_decay: 0.0000120613

epoch time: 1686.7103271484s, input_fired_time: 19.6364173889, output_fired_time: 10.5360078812, propagate_forward_time: 529.9310302734, propagate_backward_time: 1115.7265625000, weight_update_time: 0.2115198076, other_time: 10.8802490234
epoch time: 108.0347595215s, input_fired_time: 1.7522065639, output_fired_time: 0.0000000000, propagate_forward_time: 103.8933410645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3892135620
validation[          , genome  5582] predictions:    6320/  10000 (63.20%), best:    6347/10000 (63.47%), error:     10781.90820, best error:     10754.75391 on epoch:    75, epoch:   77/100, mu: 0.9853262901, learning_rate: 0.0000021433, weight_decay: 0.0000114835

epoch time: 1686.4595947266s, input_fired_time: 16.3872089386, output_fired_time: 11.6087980270, propagate_forward_time: 533.4589843750, propagate_backward_time: 1116.0793457031, weight_update_time: 0.2120776027, other_time: 8.9252929688
epoch time: 110.2720565796s, input_fired_time: 0.8047593236, output_fired_time: 0.0000000000, propagate_forward_time: 108.4445648193, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0227355957
validation[          , genome  5582] predictions:    6429/  10000 (64.29%), best:    6429/10000 (64.29%), error:     10712.85254, best error:     10712.85254 on epoch:    78, epoch:   78/100, mu: 0.9855967760, learning_rate: 0.0000020109, weight_decay: 0.0000109334

epoch time: 1676.8022460938s, input_fired_time: 24.9896526337, output_fired_time: 8.8195276260, propagate_forward_time: 528.0220336914, propagate_backward_time: 1103.8309326172, weight_update_time: 0.2109152675, other_time: 11.1400146484
epoch time: 108.1685791016s, input_fired_time: 1.8619021177, output_fired_time: 0.0000000000, propagate_forward_time: 104.3781890869, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9284896851
validation[          , genome  5582] predictions:    6355/  10000 (63.55%), best:    6429/10000 (64.29%), error:     10768.72656, best error:     10712.85254 on epoch:    78, epoch:   79/100, mu: 0.9858515859, learning_rate: 0.0000018866, weight_decay: 0.0000104096

epoch time: 1678.4201660156s, input_fired_time: 23.2310562134, output_fired_time: 5.6625518799, propagate_forward_time: 525.1702270508, propagate_backward_time: 1114.2386474609, weight_update_time: 1.2722710371, other_time: 10.1175537109
epoch time: 110.5249481201s, input_fired_time: 1.7511121035, output_fired_time: 0.0000000000, propagate_forward_time: 106.8381423950, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9356918335
validation[          , genome  5582] predictions:    6325/  10000 (63.25%), best:    6429/10000 (64.29%), error:     10827.58105, best error:     10712.85254 on epoch:    78, epoch:   80/100, mu: 0.9860916734, learning_rate: 0.0000017700, weight_decay: 0.0000099109

epoch time: 1692.0761718750s, input_fired_time: 30.8689270020, output_fired_time: 12.1031837463, propagate_forward_time: 521.5925903320, propagate_backward_time: 1119.6070556641, weight_update_time: 0.2062822878, other_time: 7.9044189453
epoch time: 109.1901245117s, input_fired_time: 1.7597198486, output_fired_time: 0.0000000000, propagate_forward_time: 103.4132614136, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.0171432495
validation[          , genome  5582] predictions:    6392/  10000 (63.92%), best:    6429/10000 (64.29%), error:     10721.22461, best error:     10712.85254 on epoch:    78, epoch:   81/100, mu: 0.9863178730, learning_rate: 0.0000016606, weight_decay: 0.0000094361

epoch time: 1692.7957763672s, input_fired_time: 20.5603446960, output_fired_time: 6.5884709358, propagate_forward_time: 530.6559448242, propagate_backward_time: 1121.9938964844, weight_update_time: 0.2077335417, other_time: 12.9970703125
epoch time: 110.4031753540s, input_fired_time: 0.8164930940, output_fired_time: 0.0000000000, propagate_forward_time: 108.5547256470, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0319595337
validation[          , genome  5582] predictions:    6360/  10000 (63.60%), best:    6429/10000 (64.29%), error:     10740.27832, best error:     10712.85254 on epoch:    78, epoch:   82/100, mu: 0.9865309596, learning_rate: 0.0000015580, weight_decay: 0.0000089841

epoch time: 1688.5555419922s, input_fired_time: 23.3242321014, output_fired_time: 7.7516036034, propagate_forward_time: 521.4088745117, propagate_backward_time: 1124.0291748047, weight_update_time: 2.1456341743, other_time: 12.0416259766
epoch time: 108.1046752930s, input_fired_time: 2.7263193130, output_fired_time: 0.0000000000, propagate_forward_time: 103.4094467163, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9689102173
validation[          , genome  5582] predictions:    6388/  10000 (63.88%), best:    6429/10000 (64.29%), error:     10741.43945, best error:     10712.85254 on epoch:    78, epoch:   83/100, mu: 0.9867317080, learning_rate: 0.0000014617, weight_decay: 0.0000085537

epoch time: 1677.6153564453s, input_fired_time: 18.4129600525, output_fired_time: 10.7123517990, propagate_forward_time: 521.9649047852, propagate_backward_time: 1112.5100097656, weight_update_time: 1.2863520384, other_time: 14.0151367188
epoch time: 108.8395156860s, input_fired_time: 1.7619647980, output_fired_time: 0.0000000000, propagate_forward_time: 101.9878997803, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0896530151
validation[          , genome  5582] predictions:    6423/  10000 (64.23%), best:    6423/10000 (64.23%), error:     10697.53223, best error:     10697.53223 on epoch:    84, epoch:   84/100, mu: 0.9869208336, learning_rate: 0.0000013714, weight_decay: 0.0000081440

epoch time: 1693.7011718750s, input_fired_time: 23.5969467163, output_fired_time: 9.5225706100, propagate_forward_time: 532.7338867188, propagate_backward_time: 1114.6706542969, weight_update_time: 0.2133528441, other_time: 13.1771240234
epoch time: 108.9021606445s, input_fired_time: 0.8040070534, output_fired_time: 0.0000000000, propagate_forward_time: 105.0445404053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0536117554
validation[          , genome  5582] predictions:    6333/  10000 (63.33%), best:    6423/10000 (64.23%), error:     10779.32129, best error:     10697.53223 on epoch:    84, epoch:   85/100, mu: 0.9870990515, learning_rate: 0.0000012866, weight_decay: 0.0000077538

epoch time: 1683.3536376953s, input_fired_time: 23.4116630554, output_fired_time: 10.7380399704, propagate_forward_time: 520.6793823242, propagate_backward_time: 1115.3160400391, weight_update_time: 0.2075715661, other_time: 13.2086181641
epoch time: 110.5477523804s, input_fired_time: 2.9238848686, output_fired_time: 0.0000000000, propagate_forward_time: 106.6112594604, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0126113892
validation[          , genome  5582] predictions:    6447/  10000 (64.47%), best:    6447/10000 (64.47%), error:     10642.19531, best error:     10642.19531 on epoch:    86, epoch:   86/100, mu: 0.9872669578, learning_rate: 0.0000012071, weight_decay: 0.0000073824

epoch time: 1701.8126220703s, input_fired_time: 32.4628181458, output_fired_time: 6.7684826851, propagate_forward_time: 527.8445434570, propagate_backward_time: 1125.6485595703, weight_update_time: 0.2112536877, other_time: 9.0882568359
epoch time: 106.7802047729s, input_fired_time: 1.7635273933, output_fired_time: 0.0000000000, propagate_forward_time: 102.9290847778, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0875930786
validation[          , genome  5582] predictions:    6392/  10000 (63.92%), best:    6447/10000 (64.47%), error:     10690.07715, best error:     10642.19531 on epoch:    86, epoch:   87/100, mu: 0.9874251485, learning_rate: 0.0000011325, weight_decay: 0.0000070287

epoch time: 1760.6770019531s, input_fired_time: 24.3498382568, output_fired_time: 9.6508369446, propagate_forward_time: 563.6654052734, propagate_backward_time: 1150.9289550781, weight_update_time: 1.1905040741, other_time: 12.0819091797
epoch time: 133.5087280273s, input_fired_time: 2.9128172398, output_fired_time: 0.0000000000, propagate_forward_time: 126.4216995239, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.1742172241
validation[          , genome  5582] predictions:    6380/  10000 (63.80%), best:    6447/10000 (64.47%), error:     10698.12402, best error:     10642.19531 on epoch:    86, epoch:   88/100, mu: 0.9875741601, learning_rate: 0.0000010625, weight_decay: 0.0000066920

epoch time: 1826.1912841797s, input_fired_time: 36.0922088623, output_fired_time: 9.0907745361, propagate_forward_time: 587.0775146484, propagate_backward_time: 1182.9578857422, weight_update_time: 0.2259407640, other_time: 10.9729003906
epoch time: 125.2532958984s, input_fired_time: 0.8735876679, output_fired_time: 0.0000000000, propagate_forward_time: 121.3543167114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0253906250
validation[          , genome  5582] predictions:    6393/  10000 (63.93%), best:    6447/10000 (64.47%), error:     10663.38086, best error:     10642.19531 on epoch:    86, epoch:   89/100, mu: 0.9877145290, learning_rate: 0.0000009969, weight_decay: 0.0000063714

epoch time: 1801.1895751953s, input_fired_time: 23.7932243347, output_fired_time: 9.8284873962, propagate_forward_time: 556.8303833008, propagate_backward_time: 1198.2341308594, weight_update_time: 1.2670781612, other_time: 12.5034179688
epoch time: 124.6575164795s, input_fired_time: 0.8306068778, output_fired_time: 0.0000000000, propagate_forward_time: 120.6987228394, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1281890869
validation[          , genome  5582] predictions:    6434/  10000 (64.34%), best:    6447/10000 (64.47%), error:     10684.58691, best error:     10642.19531 on epoch:    86, epoch:   90/100, mu: 0.9878467917, learning_rate: 0.0000009353, weight_decay: 0.0000060662

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_186_5582_0_r505125911_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.529426
read mu: 0.987971
read mu_delta: 0.942125
read initial_learning_rate: 0.000291229
read learning_rate: 8.77485e-07
read learning_rate_delta: 0.938203
read initial_weight_decay: 0.000503154
read weight_decay: 5.77559e-06
read weight_decay_delta: 0.952095
read batch_size: 93
read epsilon: 1e-07
read alpha: 0.0776802
read input_dropout_probability: 0.00118188
read hidden_dropout_probability: 0.105026
read velocity_reset: 1916
read epoch: 91
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 86
read number_validation_images: 10000
read best_validation_predictions: 6447
read best_validation_error: 10642.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5582
read normal distribution: '1 0 0'
generator_str: '2016130362'
read generator: 2016130362
read generated_by_map:
1 crossover 1
reading 90 nodes.
reading 336 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1880.99523925781s, input_fired_time: 25.0385437011719, output_fired_time: 11.2662410736084, propagate_forward_time: 614.570007324219, propagate_backward_time: 1211.45275878906, weight_update_time: 0.230167955160141, other_time: 18.6676025390625
epoch time: 125.357307434082s, input_fired_time: 0.889391422271729, output_fired_time: 0, propagate_forward_time: 122.33521270752, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.13270568847656
validation[          , genome  5582] predictions:    6414/  10000 (64.14%), best:    6447/10000 (64.47%), error:     10670.82715, best error:     10642.19531 on epoch:    86, epoch:   91/100, mu: 0.9879714251, learning_rate: 0.0000008775, weight_decay: 0.0000057756

epoch time: 1674.8474121094s, input_fired_time: 26.6515769958, output_fired_time: 10.7536230087, propagate_forward_time: 514.4371337891, propagate_backward_time: 1101.7712402344, weight_update_time: 0.2092280537, other_time: 21.2337646484
epoch time: 99.4873352051s, input_fired_time: 1.7631998062, output_fired_time: 0.0000000000, propagate_forward_time: 95.7967071533, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9274291992
validation[          , genome  5582] predictions:    6407/  10000 (64.07%), best:    6447/10000 (64.47%), error:     10668.47363, best error:     10642.19531 on epoch:    86, epoch:   92/100, mu: 0.9880888462, learning_rate: 0.0000008233, weight_decay: 0.0000054989

epoch time: 1613.0854492188s, input_fired_time: 29.2324295044, output_fired_time: 11.2961778641, propagate_forward_time: 478.2733459473, propagate_backward_time: 1085.5894775391, weight_update_time: 0.2119550854, other_time: 8.6940917969
epoch time: 98.5449447632s, input_fired_time: 0.7603405714, output_fired_time: 0.0000000000, propagate_forward_time: 93.8216323853, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9629745483
validation[          , genome  5582] predictions:    6416/  10000 (64.16%), best:    6447/10000 (64.47%), error:     10687.85352, best error:     10642.19531 on epoch:    86, epoch:   93/100, mu: 0.9881994724, learning_rate: 0.0000007724, weight_decay: 0.0000052355

epoch time: 1612.6612548828s, input_fired_time: 22.0160751343, output_fired_time: 10.3194398880, propagate_forward_time: 490.2822570801, propagate_backward_time: 1084.2332763672, weight_update_time: 0.2006140202, other_time: 5.8101806641
epoch time: 101.9772720337s, input_fired_time: 0.7770644426, output_fired_time: 0.0000000000, propagate_forward_time: 98.2200393677, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9801712036
validation[          , genome  5582] predictions:    6375/  10000 (63.75%), best:    6447/10000 (64.47%), error:     10704.01465, best error:     10642.19531 on epoch:    86, epoch:   94/100, mu: 0.9883036613, learning_rate: 0.0000007247, weight_decay: 0.0000049847

epoch time: 1602.9163818359s, input_fired_time: 22.1846027374, output_fired_time: 9.3592681885, propagate_forward_time: 479.3079833984, propagate_backward_time: 1080.2370605469, weight_update_time: 0.1990473419, other_time: 11.8275146484
epoch time: 98.1620559692s, input_fired_time: 0.7618605494, output_fired_time: 0.0000000000, propagate_forward_time: 96.4717636108, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9284286499
validation[          , genome  5582] predictions:    6373/  10000 (63.73%), best:    6447/10000 (64.47%), error:     10691.64258, best error:     10642.19531 on epoch:    86, epoch:   95/100, mu: 0.9884018302, learning_rate: 0.0000006799, weight_decay: 0.0000047459

epoch time: 1571.7110595703s, input_fired_time: 27.0331192017, output_fired_time: 9.2229890823, propagate_forward_time: 467.8574523926, propagate_backward_time: 1056.8583984375, weight_update_time: 0.1958027333, other_time: 10.7391357422
epoch time: 98.2518157959s, input_fired_time: 1.7793490887, output_fired_time: 0.0000000000, propagate_forward_time: 94.5450820923, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9273834229
validation[          , genome  5582] predictions:    6414/  10000 (64.14%), best:    6447/10000 (64.47%), error:     10690.11523, best error:     10642.19531 on epoch:    86, epoch:   96/100, mu: 0.9884943366, learning_rate: 0.0000006379, weight_decay: 0.0000045185

epoch time: 1604.9359130859s, input_fired_time: 19.2384738922, output_fired_time: 7.3484091759, propagate_forward_time: 489.1446228027, propagate_backward_time: 1075.4324951172, weight_update_time: 1.2006360292, other_time: 13.7718505859
epoch time: 99.3623428345s, input_fired_time: 2.7678222656, output_fired_time: 0.0000000000, propagate_forward_time: 95.6810531616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9134674072
validation[          , genome  5582] predictions:    6389/  10000 (63.89%), best:    6447/10000 (64.47%), error:     10661.98145, best error:     10642.19531 on epoch:    86, epoch:   97/100, mu: 0.9885814786, learning_rate: 0.0000005984, weight_decay: 0.0000043021

epoch time: 1610.2377929688s, input_fired_time: 26.0252609253, output_fired_time: 9.3436746597, propagate_forward_time: 482.2001647949, propagate_backward_time: 1080.0839843750, weight_update_time: 1.2007439137, other_time: 12.5847167969
epoch time: 98.5212783813s, input_fired_time: 1.7672659159, output_fired_time: 0.0000000000, propagate_forward_time: 95.8361358643, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9178771973
validation[          , genome  5582] predictions:    6438/  10000 (64.38%), best:    6447/10000 (64.47%), error:     10662.66699, best error:     10642.19531 on epoch:    86, epoch:   98/100, mu: 0.9886635542, learning_rate: 0.0000005615, weight_decay: 0.0000040960

epoch time: 1604.9962158203s, input_fired_time: 24.3476638794, output_fired_time: 7.3170127869, propagate_forward_time: 473.6677551270, propagate_backward_time: 1088.8626708984, weight_update_time: 0.1997329593, other_time: 10.8011474609
epoch time: 100.0086517334s, input_fired_time: 0.7655693889, output_fired_time: 0.0000000000, propagate_forward_time: 98.2900543213, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9530258179
validation[          , genome  5582] predictions:    6448/  10000 (64.48%), best:    6448/10000 (64.48%), error:     10629.05371, best error:     10629.05371 on epoch:    99, epoch:   99/100, mu: 0.9887409210, learning_rate: 0.0000005268, weight_decay: 0.0000038998

epoch time: 1578.5782470703s, input_fired_time: 28.8387889862, output_fired_time: 8.2493629456, propagate_forward_time: 476.4588623047, propagate_backward_time: 1054.1690673828, weight_update_time: 0.2011039257, other_time: 10.8621826172
epoch time: 99.9727935791s, input_fired_time: 3.7790157795, output_fired_time: 0.0000000000, propagate_forward_time: 94.2422027588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9515762329
validation[          , genome  5582] predictions:    6388/  10000 (63.88%), best:    6448/10000 (64.48%), error:     10652.63184, best error:     10629.05371 on epoch:    99, epoch:  100/100, mu: 0.9888138175, learning_rate: 0.0000004942, weight_decay: 0.0000037129

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 491.6597595215s, input_fired_time: 9.8203611374, output_fired_time: 0.0000000000, propagate_forward_time: 471.1869201660, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.6524658203
best training[          , genome  5582] predictions:   33940/  50000 (67.88%), best:    6448/10000 (64.48%), error:     49797.33984, best error:     10629.05371 on epoch:    99, epoch:  101/100, mu: 0.9888824821, learning_rate: 0.0000004637, weight_decay: 0.0000035351
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 100.2442398071s, input_fired_time: 3.6788055897, output_fired_time: 0.0000000000, propagate_forward_time: 92.5858535767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9795837402
   testing[          , genome  5582] predictions:    6448/  10000 (64.48%), best:    6448/10000 (64.48%), error:     10629.05371, best error:     10629.05371 on epoch:    99, epoch:  101/100, mu: 0.9888824821, learning_rate: 0.0000004637, weight_decay: 0.0000035351
backpropagation finished successfully!
15:40:57 (12876): called boinc_finish(0)

</stderr_txt>
]]>