Nameexact_genome_1542069603_173_5731_1
Workunit4084827
Created13 Nov 2018, 6:29:36 UTC
Sent13 Nov 2018, 6:53:56 UTC
Report deadline14 Dec 2018, 16:04:17 UTC
Received14 Nov 2018, 1:14:43 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID60920
Run time14 hours 14 min 35 sec
CPU time14 hours 9 min 49 sec
Validate stateInvalid
Credit0.00
Device peak FLOPS3.88 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size577.97 MB
Peak swap size576.26 MB
Peak disk usage12.39 MB

Stderr output

<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731_1_r1728956668_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549152
read mu: 0.549152
read mu_delta: 0.973465
read initial_learning_rate: 0.0029105
read learning_rate: 0.0029105
read learning_rate_delta: 0.915135
read initial_weight_decay: 0.000320926
read weight_decay: 0.000320926
read weight_decay_delta: 0.9166
read batch_size: 120
read epsilon: 1e-07
read alpha: 0.122658
read input_dropout_probability: 0.0016865
read hidden_dropout_probability: 0.0613132
read velocity_reset: 1773
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5731
read normal distribution: '1 0 0'
generator_str: '449945435'
read generator: 449945435
read generated_by_map:
1 disable_edge 1
reading 93 nodes.
reading 379 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 943004958
post shuffle 1: 2070288070
epoch time: 1718.96508789063s, input_fired_time: 18.5510654449463, output_fired_time: 9.61565589904785, propagate_forward_time: 662.617065429688, propagate_backward_time: 1015.21893310547, weight_update_time: 0.360840886831284, other_time: 12.9624633789063
epoch time: 137.149978637695s, input_fired_time: 1.42167115211487, output_fired_time: 0, propagate_forward_time: 133.471954345703, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.25634765625
validation[          , genome  5731] predictions:    4935/  10000 (49.35%), best:    4935/10000 (49.35%), error:     14583.95898, best error:     14583.95898 on epoch:     0, epoch:    0/25, mu: 0.5491520166, learning_rate: 0.0029105004, weight_decay: 0.0003209260

epoch time: 1722.5675048828s, input_fired_time: 18.7622795105, output_fired_time: 9.5848035812, propagate_forward_time: 664.3488769531, propagate_backward_time: 1016.8788452148, weight_update_time: 0.3656370640, other_time: 12.9926147461
epoch time: 137.4648742676s, input_fired_time: 1.4236449003, output_fired_time: 0.0000000000, propagate_forward_time: 133.7451171875, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2961120605
validation[          , genome  5731] predictions:    5285/  10000 (52.85%), best:    5285/10000 (52.85%), error:     13482.91406, best error:     13482.91406 on epoch:     1, epoch:    1/25, mu: 0.5608497858, learning_rate: 0.0026635013, weight_decay: 0.0002941609

epoch time: 1721.9354248047s, input_fired_time: 18.8203773499, output_fired_time: 9.6094408035, propagate_forward_time: 664.1514892578, propagate_backward_time: 1016.4000854492, weight_update_time: 0.3617581129, other_time: 12.9539184570
epoch time: 137.2663574219s, input_fired_time: 1.4419324398, output_fired_time: 0.0000000000, propagate_forward_time: 133.5496520996, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2747802734
validation[          , genome  5731] predictions:    5304/  10000 (53.04%), best:    5285/10000 (52.85%), error:     13645.98926, best error:     13482.91406 on epoch:     1, epoch:    2/25, mu: 0.5722371936, learning_rate: 0.0024374637, weight_decay: 0.0002696280

epoch time: 1722.3647460938s, input_fired_time: 18.8507328033, output_fired_time: 9.6295452118, propagate_forward_time: 663.7766723633, propagate_backward_time: 1017.1473388672, weight_update_time: 0.3578991890, other_time: 12.9605712891
epoch time: 137.8728790283s, input_fired_time: 1.4500988722, output_fired_time: 0.0000000000, propagate_forward_time: 134.1452178955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2775573730
validation[          , genome  5731] predictions:    5710/  10000 (57.10%), best:    5710/10000 (57.10%), error:     12336.81250, best error:     12336.81250 on epoch:     3, epoch:    3/25, mu: 0.5833224058, learning_rate: 0.0022306088, weight_decay: 0.0002471411

epoch time: 1723.8892822266s, input_fired_time: 18.8962497711, output_fired_time: 9.6212997437, propagate_forward_time: 664.5148925781, propagate_backward_time: 1017.8789062500, weight_update_time: 0.3603321910, other_time: 12.9779052734
epoch time: 139.1230773926s, input_fired_time: 1.4802436829, output_fired_time: 0.0000000000, propagate_forward_time: 135.3635559082, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2792816162
validation[          , genome  5731] predictions:    5270/  10000 (52.70%), best:    5710/10000 (57.10%), error:     13660.82129, best error:     12336.81250 on epoch:     3, epoch:    4/25, mu: 0.5941134691, learning_rate: 0.0020413084, weight_decay: 0.0002265297

epoch time: 1723.3603515625s, input_fired_time: 18.9125938416, output_fired_time: 9.6516160965, propagate_forward_time: 664.8403930664, propagate_backward_time: 1016.9887695313, weight_update_time: 0.3545486927, other_time: 12.9670410156
epoch time: 137.4930267334s, input_fired_time: 1.4681719542, output_fired_time: 0.0000000000, propagate_forward_time: 133.7657623291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2590942383
validation[          , genome  5731] predictions:    5472/  10000 (54.72%), best:    5710/10000 (57.10%), error:     13238.85840, best error:     12336.81250 on epoch:     3, epoch:    5/25, mu: 0.6046181917, learning_rate: 0.0018680731, weight_decay: 0.0002076372

epoch time: 1730.0340576172s, input_fired_time: 18.9850139618, output_fired_time: 9.7578516006, propagate_forward_time: 666.7310180664, propagate_backward_time: 1021.5689697266, weight_update_time: 0.3601942658, other_time: 12.9913330078
epoch time: 137.9057006836s, input_fired_time: 1.4760615826, output_fired_time: 0.0000000000, propagate_forward_time: 134.1483764648, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2812652588
validation[          , genome  5731] predictions:    5878/  10000 (58.78%), best:    5878/10000 (58.78%), error:     11988.60840, best error:     11988.60840 on epoch:     6, epoch:    6/25, mu: 0.6148442030, learning_rate: 0.0017095393, weight_decay: 0.0001903203

epoch time: 1722.2484130859s, input_fired_time: 18.9426307678, output_fired_time: 9.6508893967, propagate_forward_time: 663.9788818359, propagate_backward_time: 1016.6940917969, weight_update_time: 0.3610572219, other_time: 12.9819335938
epoch time: 137.6163787842s, input_fired_time: 1.4819997549, output_fired_time: 0.0000000000, propagate_forward_time: 133.8502197266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2841644287
validation[          , genome  5731] predictions:    5828/  10000 (58.28%), best:    5828/10000 (58.28%), error:     11934.44336, best error:     11934.44336 on epoch:     7, epoch:    7/25, mu: 0.6247988343, learning_rate: 0.0015644595, weight_decay: 0.0001744477

epoch time: 1724.3409423828s, input_fired_time: 18.9654674530, output_fired_time: 9.6536617279, propagate_forward_time: 665.0634155273, propagate_backward_time: 1017.6366577148, weight_update_time: 0.3624021709, other_time: 13.0217895508
epoch time: 137.8259277344s, input_fired_time: 1.4854804277, output_fired_time: 0.0000000000, propagate_forward_time: 134.0623016357, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2781524658
validation[          , genome  5731] predictions:    5822/  10000 (58.22%), best:    5822/10000 (58.22%), error:     11919.47656, best error:     11919.47656 on epoch:     8, epoch:    8/25, mu: 0.6344893575, learning_rate: 0.0014316919, weight_decay: 0.0001598988

epoch time: 1723.1719970703s, input_fired_time: 18.9776782990, output_fired_time: 9.6408195496, propagate_forward_time: 664.2436523438, propagate_backward_time: 1017.3247680664, weight_update_time: 0.3629655242, other_time: 12.9850463867
epoch time: 137.6069335938s, input_fired_time: 1.4845947027, output_fired_time: 0.0000000000, propagate_forward_time: 133.8430938721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2792510986
validation[          , genome  5731] predictions:    5797/  10000 (57.97%), best:    5822/10000 (58.22%), error:     12226.59473, best error:     11919.47656 on epoch:     8, epoch:    9/25, mu: 0.6439227462, learning_rate: 0.0013101916, weight_decay: 0.0001465633

epoch time: 1722.4940185547s, input_fired_time: 19.0182018280, output_fired_time: 9.6305866241, propagate_forward_time: 664.1651611328, propagate_backward_time: 1016.6918334961, weight_update_time: 0.3590174913, other_time: 12.9882202148
epoch time: 137.8703308105s, input_fired_time: 1.4982377291, output_fired_time: 0.0000000000, propagate_forward_time: 134.1083221436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2637634277
validation[          , genome  5731] predictions:    5926/  10000 (59.26%), best:    5926/10000 (59.26%), error:     11650.62500, best error:     11650.62500 on epoch:    10, epoch:   10/25, mu: 0.6531057954, learning_rate: 0.0011990024, weight_decay: 0.0001343400

epoch time: 1723.4913330078s, input_fired_time: 19.0849227905, output_fired_time: 9.6259212494, propagate_forward_time: 664.7523193359, propagate_backward_time: 1017.1123657227, weight_update_time: 0.3585517406, other_time: 12.9157104492
epoch time: 137.1730346680s, input_fired_time: 1.4988837242, output_fired_time: 0.0000000000, propagate_forward_time: 133.3966674805, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2774810791
validation[          , genome  5731] predictions:    5931/  10000 (59.31%), best:    5926/10000 (59.26%), error:     11673.18555, best error:     11650.62500 on epoch:    10, epoch:   11/25, mu: 0.6620451808, learning_rate: 0.0010972492, weight_decay: 0.0001231361

epoch time: 1725.5557861328s, input_fired_time: 19.1260890961, output_fired_time: 9.6596746445, propagate_forward_time: 665.3563842773, propagate_backward_time: 1018.4313354492, weight_update_time: 0.3628251553, other_time: 12.9822387695
epoch time: 137.4873199463s, input_fired_time: 1.5026861429, output_fired_time: 0.0000000000, propagate_forward_time: 133.7339630127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2506713867
validation[          , genome  5731] predictions:    5946/  10000 (59.46%), best:    5926/10000 (59.26%), error:     11708.07520, best error:     11650.62500 on epoch:    10, epoch:   12/25, mu: 0.6707473993, learning_rate: 0.0010041313, weight_decay: 0.0001128666

epoch time: 1723.5604248047s, input_fired_time: 19.1567459106, output_fired_time: 9.6079139709, propagate_forward_time: 664.9421997070, propagate_backward_time: 1016.8737792969, weight_update_time: 0.3622004092, other_time: 12.9797363281
epoch time: 138.2304534912s, input_fired_time: 1.5183331966, output_fired_time: 0.0000000000, propagate_forward_time: 134.4467926025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2653350830
validation[          , genome  5731] predictions:    6093/  10000 (60.93%), best:    6093/10000 (60.93%), error:     11298.33203, best error:     11298.33203 on epoch:    13, epoch:   13/25, mu: 0.6792187095, learning_rate: 0.0009189158, weight_decay: 0.0001034536

epoch time: 1797.2545166016s, input_fired_time: 19.7937507629, output_fired_time: 10.0190095901, propagate_forward_time: 694.7012329102, propagate_backward_time: 1059.0468750000, weight_update_time: 0.3763089180, other_time: 13.6936035156
epoch time: 137.7517700195s, input_fired_time: 1.5153716803, output_fired_time: 0.0000000000, propagate_forward_time: 133.9609527588, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2754516602
validation[          , genome  5731] predictions:    6095/  10000 (60.95%), best:    6095/10000 (60.95%), error:     11247.96094, best error:     11247.96094 on epoch:    14, epoch:   14/25, mu: 0.6874651909, learning_rate: 0.0008409322, weight_decay: 0.0000948256

epoch time: 1725.5488281250s, input_fired_time: 19.1920890808, output_fired_time: 9.6337709427, propagate_forward_time: 665.3405761719, propagate_backward_time: 1018.3863525391, weight_update_time: 0.3609703481, other_time: 12.9959716797
epoch time: 137.7045135498s, input_fired_time: 1.5157773495, output_fired_time: 0.0000000000, propagate_forward_time: 133.9069976807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2817382813
validation[          , genome  5731] predictions:    6147/  10000 (61.47%), best:    6147/10000 (61.47%), error:     11157.90039, best error:     11157.90039 on epoch:    15, epoch:   15/25, mu: 0.6954928637, learning_rate: 0.0007695666, weight_decay: 0.0000869172

epoch time: 1738.3936767578s, input_fired_time: 19.2228279114, output_fired_time: 9.6129035950, propagate_forward_time: 676.2440795898, propagate_backward_time: 1020.2827758789, weight_update_time: 0.3625590503, other_time: 13.0311889648
epoch time: 137.1008605957s, input_fired_time: 1.5156053305, output_fired_time: 0.0000000000, propagate_forward_time: 133.3473205566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2379302979
validation[          , genome  5731] predictions:    5917/  10000 (59.17%), best:    6147/10000 (61.47%), error:     11632.46777, best error:     11157.90039 on epoch:    15, epoch:   16/25, mu: 0.7033075094, learning_rate: 0.0007042575, weight_decay: 0.0000796683

epoch time: 1725.1887207031s, input_fired_time: 19.2271385193, output_fired_time: 9.6247587204, propagate_forward_time: 665.2518310547, propagate_backward_time: 1018.0945434570, weight_update_time: 0.3610222340, other_time: 12.9904174805
epoch time: 137.6892852783s, input_fired_time: 1.5230119228, output_fired_time: 0.0000000000, propagate_forward_time: 133.8993988037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2668762207
validation[          , genome  5731] predictions:    6175/  10000 (61.75%), best:    6175/10000 (61.75%), error:     11030.43750, best error:     11030.43750 on epoch:    17, epoch:   17/25, mu: 0.7109147906, learning_rate: 0.0006444908, weight_decay: 0.0000730240

epoch time: 1730.0281982422s, input_fired_time: 19.2578639984, output_fired_time: 9.6587343216, propagate_forward_time: 666.5770263672, propagate_backward_time: 1021.4982910156, weight_update_time: 0.3592587113, other_time: 13.0363769531
epoch time: 144.5925292969s, input_fired_time: 1.6265531778, output_fired_time: 0.0000000000, propagate_forward_time: 140.5972747803, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3686981201
validation[          , genome  5731] predictions:    6166/  10000 (61.66%), best:    6166/10000 (61.66%), error:     10956.14551, best error:     10956.14551 on epoch:    18, epoch:   18/25, mu: 0.7183202505, learning_rate: 0.0005897962, weight_decay: 0.0000669338

epoch time: 1725.8977050781s, input_fired_time: 19.2294788361, output_fired_time: 9.6250486374, propagate_forward_time: 665.4916381836, propagate_backward_time: 1018.6098022461, weight_update_time: 0.3578872383, other_time: 12.9417114258
epoch time: 137.9850311279s, input_fired_time: 1.5327198505, output_fired_time: 0.0000000000, propagate_forward_time: 134.1692199707, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2830963135
validation[          , genome  5731] predictions:    6223/  10000 (62.23%), best:    6223/10000 (62.23%), error:     10824.51660, best error:     10824.51660 on epoch:    19, epoch:   19/25, mu: 0.7255291939, learning_rate: 0.0005397432, weight_decay: 0.0000613516

epoch time: 1728.7414550781s, input_fired_time: 19.2446727753, output_fired_time: 9.6616992950, propagate_forward_time: 667.2278442383, propagate_backward_time: 1019.6021728516, weight_update_time: 0.3631226420, other_time: 13.0050048828
epoch time: 138.2162017822s, input_fired_time: 1.5271301270, output_fired_time: 0.0000000000, propagate_forward_time: 134.4344024658, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2546691895
validation[          , genome  5731] predictions:    6147/  10000 (61.47%), best:    6223/10000 (62.23%), error:     11059.35156, best error:     10824.51660 on epoch:    19, epoch:   20/25, mu: 0.7325468659, learning_rate: 0.0004939380, weight_decay: 0.0000562349

epoch time: 1732.5528564453s, input_fired_time: 19.3029136658, output_fired_time: 9.6898717880, propagate_forward_time: 668.8699951172, propagate_backward_time: 1021.6909179688, weight_update_time: 0.3673492074, other_time: 12.9992675781
epoch time: 138.7223663330s, input_fired_time: 1.5536504984, output_fired_time: 0.0000000000, propagate_forward_time: 134.9053497314, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2633666992
validation[          , genome  5731] predictions:    6227/  10000 (62.27%), best:    6223/10000 (62.23%), error:     10861.30566, best error:     10824.51660 on epoch:    19, epoch:   21/25, mu: 0.7393783331, learning_rate: 0.0004520200, weight_decay: 0.0000515449

epoch time: 1734.1102294922s, input_fired_time: 19.3259468079, output_fired_time: 9.6662244797, propagate_forward_time: 668.9003295898, propagate_backward_time: 1023.2268676758, weight_update_time: 0.3637470603, other_time: 12.9909057617
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731_1_r1728956668_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549152
read mu: 0.746029
read mu_delta: 0.973465
read initial_learning_rate: 0.0029105
read learning_rate: 0.000413659
read learning_rate_delta: 0.915135
read initial_weight_decay: 0.000320926
read weight_decay: 4.72461e-05
read weight_decay_delta: 0.9166
read batch_size: 120
read epsilon: 1e-07
read alpha: 0.122658
read input_dropout_probability: 0.0016865
read hidden_dropout_probability: 0.0613132
read velocity_reset: 1773
read epoch: 22
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 19
read number_validation_images: 10000
read best_validation_predictions: 6223
read best_validation_error: 10824.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5731
read normal distribution: '1 0 0'
generator_str: '338603144'
read generator: 338603144
read generated_by_map:
1 disable_edge 1
reading 93 nodes.
reading 379 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1754.83190917969s, input_fired_time: 19.6486377716064, output_fired_time: 9.96653938293457, propagate_forward_time: 686.692993164063, propagate_backward_time: 1025.5146484375, weight_update_time: 0.377278089523315, other_time: 13.009033203125
epoch time: 141.981292724609s, input_fired_time: 1.55484163761139, output_fired_time: 0, propagate_forward_time: 138.184646606445, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.24180603027344
validation[          , genome  5731] predictions:    6313/  10000 (63.13%), best:    6313/10000 (63.13%), error:     10591.48242, best error:     10591.48242 on epoch:    22, epoch:   22/25, mu: 0.7460285425, learning_rate: 0.0004136594, weight_decay: 0.0000472461

epoch time: 1752.6456298828s, input_fired_time: 19.6362342834, output_fired_time: 9.9525165558, propagate_forward_time: 685.5220336914, propagate_backward_time: 1024.5703125000, weight_update_time: 0.3759480715, other_time: 12.9645996094
epoch time: 142.1503448486s, input_fired_time: 1.5658484697, output_fired_time: 0.0000000000, propagate_forward_time: 138.3430938721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2414093018
validation[          , genome  5731] predictions:    6266/  10000 (62.66%), best:    6313/10000 (63.13%), error:     10724.62109, best error:     10591.48242 on epoch:    22, epoch:   23/25, mu: 0.7525022626, learning_rate: 0.0003785542, weight_decay: 0.0000433058

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731_1_r1728956668_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549152
read mu: 0.758804
read mu_delta: 0.973465
read initial_learning_rate: 0.0029105
read learning_rate: 0.000346428
read learning_rate_delta: 0.915135
read initial_weight_decay: 0.000320926
read weight_decay: 3.96941e-05
read weight_decay_delta: 0.9166
read batch_size: 120
read epsilon: 1e-07
read alpha: 0.122658
read input_dropout_probability: 0.0016865
read hidden_dropout_probability: 0.0613132
read velocity_reset: 1773
read epoch: 24
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 22
read number_validation_images: 10000
read best_validation_predictions: 6313
read best_validation_error: 10591.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5731
read normal distribution: '1 0 0'
generator_str: '1596230269'
read generator: 1596230269
read generated_by_map:
1 disable_edge 1
reading 93 nodes.
reading 379 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1743.24572753906s, input_fired_time: 19.490837097168, output_fired_time: 9.82638454437256, propagate_forward_time: 677.780517578125, propagate_backward_time: 1023.14013671875, weight_update_time: 0.372320741415024, other_time: 13.0078125
epoch time: 139.682418823242s, input_fired_time: 1.55141222476959, output_fired_time: 0, propagate_forward_time: 135.875885009766, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.255126953125
validation[          , genome  5731] predictions:    6295/  10000 (62.95%), best:    6313/10000 (63.13%), error:     10630.28613, best error:     10591.48242 on epoch:    22, epoch:   24/25, mu: 0.7588042021, learning_rate: 0.0003464283, weight_decay: 0.0000396941

epoch time: 1739.7095947266s, input_fired_time: 19.4699535370, output_fired_time: 9.7862586975, propagate_forward_time: 674.9891967773, propagate_backward_time: 1022.4674682617, weight_update_time: 0.3680824041, other_time: 12.9966430664
epoch time: 138.9492340088s, input_fired_time: 1.5499873161, output_fired_time: 0.0000000000, propagate_forward_time: 135.1357269287, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2635192871
validation[          , genome  5731] predictions:    6262/  10000 (62.62%), best:    6313/10000 (63.13%), error:     10629.71875, best error:     10591.48242 on epoch:    22, epoch:   25/25, mu: 0.7649389505, learning_rate: 0.0003170287, weight_decay: 0.0000363836

evaluating best weights on full training data.
evaluting training set with running mean/variance:
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731_1_r1728956668_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549152
read mu: 0.770911
read mu_delta: 0.973465
read initial_learning_rate: 0.0029105
read learning_rate: 0.000290124
read learning_rate_delta: 0.915135
read initial_weight_decay: 0.000320926
read weight_decay: 3.33493e-05
read weight_decay_delta: 0.9166
read batch_size: 120
read epsilon: 1e-07
read alpha: 0.122658
read input_dropout_probability: 0.0016865
read hidden_dropout_probability: 0.0613132
read velocity_reset: 1773
read epoch: 26
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 22
read number_validation_images: 10000
read best_validation_predictions: 6313
read best_validation_error: 10591.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5731
read normal distribution: '1 0 0'
generator_str: '474579697'
read generator: 474579697
read generated_by_map:
1 disable_edge 1
reading 93 nodes.
reading 379 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542069603_173_5731_1_r1728956668_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.549152
read mu: 0.770911
read mu_delta: 0.973465
read initial_learning_rate: 0.0029105
read learning_rate: 0.000290124
read learning_rate_delta: 0.915135
read initial_weight_decay: 0.000320926
read weight_decay: 3.33493e-05
read weight_decay_delta: 0.9166
read batch_size: 120
read epsilon: 1e-07
read alpha: 0.122658
read input_dropout_probability: 0.0016865
read hidden_dropout_probability: 0.0613132
read velocity_reset: 1773
read epoch: 26
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 22
read number_validation_images: 10000
read best_validation_predictions: 6313
read best_validation_error: 10591.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5731
read normal distribution: '1 0 0'
generator_str: '474579697'
read generator: 474579697
read generated_by_map:
1 disable_edge 1
reading 93 nodes.
reading 379 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1686.42260742188s, input_fired_time: 19.0120315551758, output_fired_time: 9.26051998138428, propagate_forward_time: 649.527893066406, propagate_backward_time: 995.993408203125, weight_update_time: 0.346357852220535, other_time: 12.6287231445313
epoch time: 134.187683105469s, input_fired_time: 1.5092431306839, output_fired_time: 0, propagate_forward_time: 130.463958740234, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.2144775390625
validation[          , genome  5731] predictions:    6393/  10000 (63.93%), best:    6393/10000 (63.93%), error:     10439.60742, best error:     10439.60742 on epoch:    26, epoch:   26/25, mu: 0.7709108591, learning_rate: 0.0002901241, weight_decay: 0.0000333493

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 667.0861816406s, input_fired_time: 7.4883661270, output_fired_time: 0.0000000000, propagate_forward_time: 648.6106567383, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.9871826172
best training[          , genome  5731] predictions:   35483/  50000 (70.97%), best:    6393/10000 (63.93%), error:     43077.76172, best error:     10439.60742 on epoch:    26, epoch:   27/25, mu: 0.7767243385, learning_rate: 0.0002655027, weight_decay: 0.0000305679
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 134.4207916260s, input_fired_time: 1.5119621754, output_fired_time: 0.0000000000, propagate_forward_time: 130.7027893066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2060394287
   testing[          , genome  5731] predictions:    6393/  10000 (63.93%), best:    6393/10000 (63.93%), error:     10439.60742, best error:     10439.60742 on epoch:    26, epoch:   27/25, mu: 0.7767243385, learning_rate: 0.0002655027, weight_decay: 0.0000305679
backpropagation finished successfully!
18:14:29 (19012): called boinc_finish(0)

</stderr_txt>
]]>