Nameexact_genome_1542727803_174_6407_0
Workunit4101152
Created20 Nov 2018, 18:11:51 UTC
Sent20 Nov 2018, 18:13:36 UTC
Report deadline5 Jan 2019, 10:12:00 UTC
Received21 Nov 2018, 16:47:22 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time18 hours 6 min 26 sec
CPU time17 hours 47 min 37 sec
Validate stateValid
Credit11,416.39
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size526.48 MB
Peak swap size546.37 MB
Peak disk usage9.57 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407_0_r374941125_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.5
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.0125
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.0005
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 0
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6407
read normal distribution: '1 0.447710812091827 0.988357841968536'
generator_str: '391499432'
read generator: 391499432
read generated_by_map:
1 split_edge 1
reading 92 nodes.
reading 317 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 41059216
post shuffle 1: 1245048767
epoch time: 1299.93725585938s, input_fired_time: 10.2685394287109, output_fired_time: 8.78853511810303, propagate_forward_time: 510.911437988281, propagate_backward_time: 755.053405761719, weight_update_time: 0.68784111738205, other_time: 14.915283203125
epoch time: 105.841781616211s, input_fired_time: 1.23347353935242, output_fired_time: 0, propagate_forward_time: 101.931343078613, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.67696380615234
validation[          , genome  6407] predictions:    4764/  10000 (47.64%), best:    4764/10000 (47.64%), error:     17872.25195, best error:     17872.25195 on epoch:     0, epoch:    0/50, mu: 0.5000000000, learning_rate: 0.0125000002, weight_decay: 0.0005000000

epoch time: 1337.6000976563s, input_fired_time: 10.1967878342, output_fired_time: 8.9794263840, propagate_forward_time: 525.2269897461, propagate_backward_time: 777.4956054688, weight_update_time: 0.6829281449, other_time: 15.7013549805
epoch time: 111.2440872192s, input_fired_time: 1.2143073082, output_fired_time: 0.0000000000, propagate_forward_time: 106.9597473145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0700302124
validation[          , genome  6407] predictions:    4167/  10000 (41.67%), best:    4764/10000 (47.64%), error:     21448.99414, best error:     17872.25195 on epoch:     0, epoch:    1/50, mu: 0.5245000124, learning_rate: 0.0118749999, weight_decay: 0.0004750000

epoch time: 1359.0058593750s, input_fired_time: 10.1632595062, output_fired_time: 9.0883054733, propagate_forward_time: 534.2603149414, propagate_backward_time: 789.3486328125, weight_update_time: 0.6827595830, other_time: 16.1454467773
epoch time: 111.2293472290s, input_fired_time: 1.2086153030, output_fired_time: 0.0000000000, propagate_forward_time: 107.0815505981, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9391784668
validation[          , genome  6407] predictions:    5088/  10000 (50.88%), best:    5088/10000 (50.88%), error:     16173.96777, best error:     16173.96777 on epoch:     2, epoch:    2/50, mu: 0.5477750301, learning_rate: 0.0112812500, weight_decay: 0.0004512500

epoch time: 1364.0649414063s, input_fired_time: 10.1095867157, output_fired_time: 9.1054449081, propagate_forward_time: 535.8059692383, propagate_backward_time: 792.5769653320, weight_update_time: 0.6901378632, other_time: 16.4669189453
epoch time: 111.0244140625s, input_fired_time: 1.2063287497, output_fired_time: 0.0000000000, propagate_forward_time: 106.8304214478, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9876632690
validation[          , genome  6407] predictions:    4999/  10000 (49.99%), best:    5088/10000 (50.88%), error:     16812.63672, best error:     16173.96777 on epoch:     2, epoch:    3/50, mu: 0.5698862672, learning_rate: 0.0107171871, weight_decay: 0.0004286875

epoch time: 1366.3218994141s, input_fired_time: 10.0500001907, output_fired_time: 9.0855188370, propagate_forward_time: 536.6401367188, propagate_backward_time: 794.0452270508, weight_update_time: 0.6978202462, other_time: 16.5009155273
epoch time: 111.9800643921s, input_fired_time: 1.2002201080, output_fired_time: 0.0000000000, propagate_forward_time: 107.5705795288, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2092666626
validation[          , genome  6407] predictions:    5101/  10000 (51.01%), best:    5101/10000 (51.01%), error:     15372.19043, best error:     15372.19043 on epoch:     4, epoch:    4/50, mu: 0.5908919573, learning_rate: 0.0101813274, weight_decay: 0.0004072531

epoch time: 1369.0964355469s, input_fired_time: 9.9689455032, output_fired_time: 9.1321849823, propagate_forward_time: 538.3307495117, propagate_backward_time: 795.0154418945, weight_update_time: 0.6833428741, other_time: 16.6490478516
epoch time: 111.7139282227s, input_fired_time: 1.1810244322, output_fired_time: 0.0000000000, propagate_forward_time: 107.5056076050, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0272979736
validation[          , genome  6407] predictions:    5147/  10000 (51.47%), best:    5101/10000 (51.01%), error:     15576.45508, best error:     15372.19043 on epoch:     4, epoch:    5/50, mu: 0.6108473539, learning_rate: 0.0096722608, weight_decay: 0.0003868904

epoch time: 1368.6837158203s, input_fired_time: 9.9535360336, output_fired_time: 9.1403236389, propagate_forward_time: 537.9115600586, propagate_backward_time: 795.1196289063, weight_update_time: 0.6886695027, other_time: 16.5586547852
epoch time: 111.9461746216s, input_fired_time: 1.2068282366, output_fired_time: 0.0000000000, propagate_forward_time: 107.4762420654, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2631072998
validation[          , genome  6407] predictions:    5291/  10000 (52.91%), best:    5291/10000 (52.91%), error:     14472.81641, best error:     14472.81641 on epoch:     6, epoch:    6/50, mu: 0.6298049688, learning_rate: 0.0091886474, weight_decay: 0.0003675459

epoch time: 1369.4842529297s, input_fired_time: 9.9545621872, output_fired_time: 9.1090536118, propagate_forward_time: 538.0928344727, propagate_backward_time: 795.9254760742, weight_update_time: 0.6939830780, other_time: 16.4023437500
epoch time: 111.7952804565s, input_fired_time: 1.1852962971, output_fired_time: 0.0000000000, propagate_forward_time: 107.5316696167, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0783157349
validation[          , genome  6407] predictions:    5214/  10000 (52.14%), best:    5291/10000 (52.91%), error:     14971.28418, best error:     14472.81641 on epoch:     6, epoch:    7/50, mu: 0.6478147507, learning_rate: 0.0087292148, weight_decay: 0.0003491686

epoch time: 1367.7105712891s, input_fired_time: 9.9825744629, output_fired_time: 9.1414976120, propagate_forward_time: 537.3557128906, propagate_backward_time: 794.8605957031, weight_update_time: 0.6824535728, other_time: 16.3702392578
epoch time: 111.6465759277s, input_fired_time: 1.2069334984, output_fired_time: 0.0000000000, propagate_forward_time: 107.3935241699, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0461196899
validation[          , genome  6407] predictions:    5380/  10000 (53.80%), best:    5380/10000 (53.80%), error:     14290.75000, best error:     14290.75000 on epoch:     8, epoch:    8/50, mu: 0.6649240255, learning_rate: 0.0082927542, weight_decay: 0.0003317102

epoch time: 1369.8127441406s, input_fired_time: 9.9834384918, output_fired_time: 9.1613626480, propagate_forward_time: 538.2304077148, propagate_backward_time: 796.0722656250, weight_update_time: 0.6904708743, other_time: 16.3652954102
epoch time: 111.8868560791s, input_fired_time: 1.1877764463, output_fired_time: 0.0000000000, propagate_forward_time: 107.5374450684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1616363525
validation[          , genome  6407] predictions:    5427/  10000 (54.27%), best:    5380/10000 (53.80%), error:     14503.59180, best error:     14290.75000 on epoch:     8, epoch:    9/50, mu: 0.6811778545, learning_rate: 0.0078781163, weight_decay: 0.0003151247

epoch time: 1370.6340332031s, input_fired_time: 9.9892292023, output_fired_time: 9.1329784393, propagate_forward_time: 538.3114624023, propagate_backward_time: 796.4698486328, weight_update_time: 0.6801801920, other_time: 16.7305297852
epoch time: 112.2648239136s, input_fired_time: 1.2064858675, output_fired_time: 0.0000000000, propagate_forward_time: 107.9864578247, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0718765259
validation[          , genome  6407] predictions:    5454/  10000 (54.54%), best:    5454/10000 (54.54%), error:     13868.39258, best error:     13868.39258 on epoch:    10, epoch:   10/50, mu: 0.6966189742, learning_rate: 0.0074842102, weight_decay: 0.0002993684

epoch time: 1369.2066650391s, input_fired_time: 10.0158290863, output_fired_time: 9.1379079819, propagate_forward_time: 538.4088745117, propagate_backward_time: 795.2604370117, weight_update_time: 0.6947018504, other_time: 16.3835449219
epoch time: 112.0579605103s, input_fired_time: 1.2003216743, output_fired_time: 0.0000000000, propagate_forward_time: 107.7431030273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1145324707
validation[          , genome  6407] predictions:    5220/  10000 (52.20%), best:    5454/10000 (54.54%), error:     14960.14551, best error:     13868.39258 on epoch:    10, epoch:   11/50, mu: 0.7112880349, learning_rate: 0.0071099997, weight_decay: 0.0002844000

epoch time: 1368.1883544922s, input_fired_time: 9.9967813492, output_fired_time: 9.1489114761, propagate_forward_time: 538.0324707031, propagate_backward_time: 794.4980468750, weight_update_time: 0.6853216887, other_time: 16.5120849609
epoch time: 111.8934707642s, input_fired_time: 1.2112975121, output_fired_time: 0.0000000000, propagate_forward_time: 107.5719833374, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1101913452
validation[          , genome  6407] predictions:    5146/  10000 (51.46%), best:    5454/10000 (54.54%), error:     15374.54883, best error:     13868.39258 on epoch:    10, epoch:   12/50, mu: 0.7252236605, learning_rate: 0.0067544994, weight_decay: 0.0002701800

epoch time: 1316.7020263672s, input_fired_time: 9.7303009033, output_fired_time: 8.8171300888, propagate_forward_time: 513.8782958984, propagate_backward_time: 767.9065551758, weight_update_time: 0.6730788946, other_time: 16.3696899414
epoch time: 100.8480072021s, input_fired_time: 1.1320204735, output_fired_time: 0.0000000000, propagate_forward_time: 96.6476745605, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0683135986
validation[          , genome  6407] predictions:    5115/  10000 (51.15%), best:    5454/10000 (54.54%), error:     16694.89453, best error:     13868.39258 on epoch:    10, epoch:   13/50, mu: 0.7384625077, learning_rate: 0.0064167744, weight_decay: 0.0002566710

epoch time: 1258.0104980469s, input_fired_time: 9.4614810944, output_fired_time: 8.3270940781, propagate_forward_time: 487.9188537598, propagate_backward_time: 736.0035400391, weight_update_time: 0.6441724300, other_time: 16.2995605469
epoch time: 101.7854995728s, input_fired_time: 1.1392457485, output_fired_time: 0.0000000000, propagate_forward_time: 97.5558624268, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0903930664
validation[          , genome  6407] predictions:    5553/  10000 (55.53%), best:    5454/10000 (54.54%), error:     14019.46973, best error:     13868.39258 on epoch:    10, epoch:   14/50, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 1256.5391845703s, input_fired_time: 9.5672988892, output_fired_time: 8.3242864609, propagate_forward_time: 486.9205627441, propagate_backward_time: 735.3517456055, weight_update_time: 0.6517801881, other_time: 16.3753051758
epoch time: 101.4516220093s, input_fired_time: 1.1442656517, output_fired_time: 0.0000000000, propagate_forward_time: 97.2481079102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0592498779
validation[          , genome  6407] predictions:    5517/  10000 (55.17%), best:    5517/10000 (55.17%), error:     13616.92871, best error:     13616.92871 on epoch:    15, epoch:   15/50, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 1253.7454833984s, input_fired_time: 9.4395761490, output_fired_time: 8.3518114090, propagate_forward_time: 485.0673828125, propagate_backward_time: 734.7233886719, weight_update_time: 0.6484212279, other_time: 16.1633300781
epoch time: 101.8820724487s, input_fired_time: 1.1358029842, output_fired_time: 0.0000000000, propagate_forward_time: 97.7322540283, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0140151978
validation[          , genome  6407] predictions:    5291/  10000 (52.91%), best:    5517/10000 (55.17%), error:     14373.65723, best error:     13616.92871 on epoch:    15, epoch:   16/50, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 1256.0455322266s, input_fired_time: 9.4922170639, output_fired_time: 8.3417997360, propagate_forward_time: 487.1585083008, propagate_backward_time: 734.7199096680, weight_update_time: 0.6512992978, other_time: 16.3331298828
epoch time: 101.0615768433s, input_fired_time: 1.1184668541, output_fired_time: 0.0000000000, propagate_forward_time: 96.8416976929, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.1014099121
validation[          , genome  6407] predictions:    5423/  10000 (54.23%), best:    5517/10000 (55.17%), error:     14466.34473, best error:     13616.92871 on epoch:    15, epoch:   17/50, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 1217.3953857422s, input_fired_time: 9.2934303284, output_fired_time: 8.1178264618, propagate_forward_time: 468.2164916992, propagate_backward_time: 715.4503784180, weight_update_time: 0.6304163933, other_time: 16.3172607422
epoch time: 94.1580581665s, input_fired_time: 1.0750111341, output_fired_time: 0.0000000000, propagate_forward_time: 90.0221099854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0609359741
validation[          , genome  6407] predictions:    5322/  10000 (53.22%), best:    5517/10000 (55.17%), error:     14244.38867, best error:     13616.92871 on epoch:    15, epoch:   18/50, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 1173.9924316406s, input_fired_time: 9.0948743820, output_fired_time: 7.8433194160, propagate_forward_time: 449.6143798828, propagate_backward_time: 691.2092285156, weight_update_time: 0.6151556373, other_time: 16.2307128906
epoch time: 94.6252746582s, input_fired_time: 1.0855755806, output_fired_time: 0.0000000000, propagate_forward_time: 90.4664154053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0732803345
validation[          , genome  6407] predictions:    5573/  10000 (55.73%), best:    5517/10000 (55.17%), error:     13789.69336, best error:     13616.92871 on epoch:    15, epoch:   19/50, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407_0_r374941125_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.814342
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00448107
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.000179243
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 20
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 15
read number_validation_images: 10000
read best_validation_predictions: 5517
read best_validation_error: 13616.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6407
read normal distribution: '1 0.447710812091827 0.988357841968536'
generator_str: '401569755'
read generator: 401569755
read generated_by_map:
1 split_edge 1
reading 92 nodes.
reading 317 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1092.50769042969s, input_fired_time: 8.62121391296387, output_fired_time: 7.03221416473389, propagate_forward_time: 403.301849365234, propagate_backward_time: 635.586303710938, weight_update_time: 0.592524290084839, other_time: 37.966064453125
epoch time: 84.2345123291016s, input_fired_time: 1.01556766033173, output_fired_time: 0, propagate_forward_time: 80.4906921386719, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.72825622558594
validation[          , genome  6407] predictions:    5663/  10000 (56.63%), best:    5663/10000 (56.63%), error:     13393.95215, best error:     13393.95215 on epoch:    20, epoch:   20/50, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 1072.3956298828s, input_fired_time: 8.6455059052, output_fired_time: 7.0779595375, propagate_forward_time: 406.6779174805, propagate_backward_time: 635.6702880859, weight_update_time: 0.5865218043, other_time: 14.3239135742
epoch time: 86.5919113159s, input_fired_time: 1.0394858122, output_fired_time: 0.0000000000, propagate_forward_time: 82.8969039917, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6555252075
validation[          , genome  6407] predictions:    5524/  10000 (55.24%), best:    5663/10000 (56.63%), error:     13566.96777, best error:     13393.95215 on epoch:    20, epoch:   21/50, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 1116.6860351563s, input_fired_time: 8.8873691559, output_fired_time: 7.4188027382, propagate_forward_time: 435.5881958008, propagate_backward_time: 650.0440673828, weight_update_time: 0.5999756455, other_time: 14.7476196289
epoch time: 90.7486724854s, input_fired_time: 1.0587357283, output_fired_time: 0.0000000000, propagate_forward_time: 86.9497604370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7401733398
validation[          , genome  6407] predictions:    5715/  10000 (57.15%), best:    5715/10000 (57.15%), error:     13132.80664, best error:     13132.80664 on epoch:    22, epoch:   22/50, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 1114.2033691406s, input_fired_time: 8.8837957382, output_fired_time: 7.3816213608, propagate_forward_time: 435.7421264648, propagate_backward_time: 647.5589599609, weight_update_time: 0.5978998542, other_time: 14.6369018555
epoch time: 89.7088546753s, input_fired_time: 1.0455498695, output_fired_time: 0.0000000000, propagate_forward_time: 85.8895721436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7737350464
validation[          , genome  6407] predictions:    5627/  10000 (56.27%), best:    5715/10000 (57.15%), error:     13339.43945, best error:     13132.80664 on epoch:    22, epoch:   23/50, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 1141.8935546875s, input_fired_time: 9.0623044968, output_fired_time: 7.6369619370, propagate_forward_time: 439.9281311035, propagate_backward_time: 671.1002807617, weight_update_time: 0.6286678910, other_time: 14.1659545898
epoch time: 91.6486511230s, input_fired_time: 1.0808435678, output_fired_time: 0.0000000000, propagate_forward_time: 87.9598846436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6079254150
validation[          , genome  6407] predictions:    5470/  10000 (54.70%), best:    5715/10000 (57.15%), error:     14064.53125, best error:     13132.80664 on epoch:    22, epoch:   24/50, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 1222.4664306641s, input_fired_time: 9.4968156815, output_fired_time: 8.1634273529, propagate_forward_time: 476.4650573730, propagate_backward_time: 714.6135864258, weight_update_time: 0.6607899070, other_time: 13.7274780273
epoch time: 104.9335708618s, input_fired_time: 1.1723897457, output_fired_time: 0.0000000000, propagate_forward_time: 101.2059173584, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5552673340
validation[          , genome  6407] predictions:    5342/  10000 (53.42%), best:    5715/10000 (57.15%), error:     15053.14355, best error:     13132.80664 on epoch:    22, epoch:   25/50, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 1133.7225341797s, input_fired_time: 9.0347270966, output_fired_time: 7.5312581062, propagate_forward_time: 434.8377380371, propagate_backward_time: 667.9518432617, weight_update_time: 0.6162308455, other_time: 14.3670043945
epoch time: 87.4467468262s, input_fired_time: 1.0463331938, output_fired_time: 0.0000000000, propagate_forward_time: 83.6305313110, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7698822021
validation[          , genome  6407] predictions:    5520/  10000 (55.20%), best:    5715/10000 (57.15%), error:     13921.60254, best error:     13132.80664 on epoch:    22, epoch:   26/50, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 1075.6784667969s, input_fired_time: 8.7315778732, output_fired_time: 7.1551561356, propagate_forward_time: 405.7491760254, propagate_backward_time: 639.5800170898, weight_update_time: 0.5889338851, other_time: 14.4625854492
epoch time: 84.3069152832s, input_fired_time: 1.0258977413, output_fired_time: 0.0000000000, propagate_forward_time: 80.5279846191, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7530364990
validation[          , genome  6407] predictions:    5695/  10000 (56.95%), best:    5695/10000 (56.95%), error:     12750.81934, best error:     12750.81934 on epoch:    27, epoch:   27/50, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 1081.4335937500s, input_fired_time: 8.7576303482, output_fired_time: 7.2272324562, propagate_forward_time: 409.5244445801, propagate_backward_time: 641.2896118164, weight_update_time: 0.5927346945, other_time: 14.6347045898
epoch time: 83.7569046021s, input_fired_time: 1.0293408632, output_fired_time: 0.0000000000, propagate_forward_time: 80.0084915161, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7190704346
validation[          , genome  6407] predictions:    5490/  10000 (54.90%), best:    5695/10000 (56.95%), error:     13997.91211, best error:     12750.81934 on epoch:    27, epoch:   28/50, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 1138.6777343750s, input_fired_time: 9.1297464371, output_fired_time: 7.5779848099, propagate_forward_time: 440.2017517090, propagate_backward_time: 667.1183471680, weight_update_time: 0.6232302189, other_time: 14.6498413086
epoch time: 92.3726272583s, input_fired_time: 1.0818976164, output_fired_time: 0.0000000000, propagate_forward_time: 88.5418701172, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7488632202
validation[          , genome  6407] predictions:    5724/  10000 (57.24%), best:    5695/10000 (56.95%), error:     13083.98145, best error:     12750.81934 on epoch:    27, epoch:   29/50, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 1153.3669433594s, input_fired_time: 9.0571298599, output_fired_time: 7.7086944580, propagate_forward_time: 447.1209106445, propagate_backward_time: 674.6143798828, weight_update_time: 0.6356533766, other_time: 14.8657836914
epoch time: 95.2595825195s, input_fired_time: 1.0865237713, output_fired_time: 0.0000000000, propagate_forward_time: 91.4089126587, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7641448975
validation[          , genome  6407] predictions:    5470/  10000 (54.70%), best:    5695/10000 (56.95%), error:     13676.64551, best error:     12750.81934 on epoch:    27, epoch:   30/50, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407_0_r374941125_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.890086
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00254883
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 0.000101953
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 31
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 27
read number_validation_images: 10000
read best_validation_predictions: 5695
read best_validation_error: 12750.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6407
read normal distribution: '1 0.447710812091827 0.988357841968536'
generator_str: '259413395'
read generator: 259413395
read generated_by_map:
1 split_edge 1
reading 92 nodes.
reading 317 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1138.42907714844s, input_fired_time: 8.95372009277344, output_fired_time: 7.61567306518555, propagate_forward_time: 434.436584472656, propagate_backward_time: 668.423645019531, weight_update_time: 0.627620697021484, other_time: 18.9993896484375
epoch time: 101.222770690918s, input_fired_time: 1.13250315189362, output_fired_time: 0, propagate_forward_time: 97.0385208129883, propagate_backward_time: 0, weight_update_time: 0, other_time: 3.05175018310547
validation[          , genome  6407] predictions:    5544/  10000 (55.44%), best:    5695/10000 (56.95%), error:     13725.65332, best error:     12750.81934 on epoch:    27, epoch:   31/50, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 1153.3256835938s, input_fired_time: 9.0179719925, output_fired_time: 7.7159156799, propagate_forward_time: 441.5554809570, propagate_backward_time: 680.6905517578, weight_update_time: 0.6224638820, other_time: 14.3457641602
epoch time: 86.7390594482s, input_fired_time: 1.0263646841, output_fired_time: 0.0000000000, propagate_forward_time: 82.8997268677, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8129653931
validation[          , genome  6407] predictions:    5639/  10000 (56.39%), best:    5695/10000 (56.95%), error:     13430.53320, best error:     12750.81934 on epoch:    27, epoch:   32/50, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 1112.7905273438s, input_fired_time: 8.8677043915, output_fired_time: 7.4550733566, propagate_forward_time: 423.7110595703, propagate_backward_time: 657.8348388672, weight_update_time: 0.6110654473, other_time: 14.9218750000
epoch time: 86.4292755127s, input_fired_time: 1.0176237822, output_fired_time: 0.0000000000, propagate_forward_time: 82.5193862915, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8922653198
validation[          , genome  6407] predictions:    5626/  10000 (56.26%), best:    5695/10000 (56.95%), error:     12992.79688, best error:     12750.81934 on epoch:    27, epoch:   33/50, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 1179.3789062500s, input_fired_time: 9.2005014420, output_fired_time: 7.8886075020, propagate_forward_time: 454.9967651367, propagate_backward_time: 692.9464111328, weight_update_time: 0.6315415502, other_time: 14.3466186523
epoch time: 86.3824081421s, input_fired_time: 1.0318666697, output_fired_time: 0.0000000000, propagate_forward_time: 82.6147613525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7357788086
validation[          , genome  6407] predictions:    5635/  10000 (56.35%), best:    5695/10000 (56.95%), error:     13237.12012, best error:     12750.81934 on epoch:    27, epoch:   34/50, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 935.0997924805s, input_fired_time: 8.0756025314, output_fired_time: 6.3027544022, propagate_forward_time: 346.2424926758, propagate_backward_time: 559.2883911133, weight_update_time: 0.5313201547, other_time: 15.1905517578
epoch time: 66.4007110596s, input_fired_time: 0.8739458323, output_fired_time: 0.0000000000, propagate_forward_time: 62.6297264099, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8970375061
validation[          , genome  6407] predictions:    5544/  10000 (55.44%), best:    5695/10000 (56.95%), error:     13727.08594, best error:     12750.81934 on epoch:    27, epoch:   35/50, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 951.3392944336s, input_fired_time: 8.1984815598, output_fired_time: 6.2852697372, propagate_forward_time: 356.0866088867, propagate_backward_time: 565.6966552734, weight_update_time: 0.5569068193, other_time: 15.0722656250
epoch time: 76.9966125488s, input_fired_time: 0.9442228079, output_fired_time: 0.0000000000, propagate_forward_time: 73.1727905273, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8796005249
validation[          , genome  6407] predictions:    5573/  10000 (55.73%), best:    5695/10000 (56.95%), error:     13754.29883, best error:     12750.81934 on epoch:    27, epoch:   36/50, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 875.5207519531s, input_fired_time: 7.8206090927, output_fired_time: 5.7662968636, propagate_forward_time: 330.7518005371, propagate_backward_time: 516.0944213867, weight_update_time: 0.5156646967, other_time: 15.0875854492
epoch time: 67.3428649902s, input_fired_time: 0.8966255188, output_fired_time: 0.0000000000, propagate_forward_time: 63.5271720886, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9190711975
validation[          , genome  6407] predictions:    5655/  10000 (56.55%), best:    5695/10000 (56.95%), error:     12803.01367, best error:     12750.81934 on epoch:    27, epoch:   37/50, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 954.3056030273s, input_fired_time: 8.2317609787, output_fired_time: 6.4073410034, propagate_forward_time: 359.8227844238, propagate_backward_time: 564.4328613281, weight_update_time: 0.5529975891, other_time: 15.4108886719
epoch time: 73.4556274414s, input_fired_time: 0.9178470969, output_fired_time: 0.0000000000, propagate_forward_time: 69.5898742676, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9479064941
validation[          , genome  6407] predictions:    5565/  10000 (55.65%), best:    5695/10000 (56.95%), error:     13533.68359, best error:     12750.81934 on epoch:    27, epoch:   38/50, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 943.4938964844s, input_fired_time: 8.3264837265, output_fired_time: 6.1446805000, propagate_forward_time: 349.0599060059, propagate_backward_time: 564.0404052734, weight_update_time: 0.5379127860, other_time: 15.9224853516
epoch time: 68.5088043213s, input_fired_time: 0.9010657668, output_fired_time: 0.0000000000, propagate_forward_time: 64.6323089600, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9754333496
validation[          , genome  6407] predictions:    5672/  10000 (56.72%), best:    5695/10000 (56.95%), error:     13233.02832, best error:     12750.81934 on epoch:    27, epoch:   39/50, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 1680.3371582031s, input_fired_time: 13.1844720840, output_fired_time: 11.3718442917, propagate_forward_time: 657.5223388672, propagate_backward_time: 976.1111450195, weight_update_time: 0.8923337460, other_time: 22.1473999023
epoch time: 68.0532684326s, input_fired_time: 0.8810616136, output_fired_time: 0.0000000000, propagate_forward_time: 64.1514892578, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0207138062
validation[          , genome  6407] predictions:    5598/  10000 (55.98%), best:    5695/10000 (56.95%), error:     13147.18848, best error:     12750.81934 on epoch:    27, epoch:   40/50, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 901.9630737305s, input_fired_time: 7.9231538773, output_fired_time: 5.9393758774, propagate_forward_time: 342.0848083496, propagate_backward_time: 529.9869384766, weight_update_time: 0.5033416748, other_time: 16.0288085938
epoch time: 71.1968536377s, input_fired_time: 0.9036418200, output_fired_time: 0.0000000000, propagate_forward_time: 67.2836074829, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0096054077
validation[          , genome  6407] predictions:    5586/  10000 (55.86%), best:    5695/10000 (56.95%), error:     13533.60840, best error:     12750.81934 on epoch:    27, epoch:   41/50, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 890.3036499023s, input_fired_time: 7.8928585052, output_fired_time: 5.9046816826, propagate_forward_time: 340.9221191406, propagate_backward_time: 519.4505004883, weight_update_time: 0.5050917864, other_time: 16.1334838867
epoch time: 73.8932189941s, input_fired_time: 0.9321703315, output_fired_time: 0.0000000000, propagate_forward_time: 69.9547195435, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0063323975
validation[          , genome  6407] predictions:    5660/  10000 (56.60%), best:    5695/10000 (56.95%), error:     13186.28027, best error:     12750.81934 on epoch:    27, epoch:   42/50, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 879.2984619141s, input_fired_time: 7.8958396912, output_fired_time: 5.7025032043, propagate_forward_time: 335.7988281250, propagate_backward_time: 513.2032470703, weight_update_time: 0.4899801314, other_time: 16.6980590820
epoch time: 67.3190689087s, input_fired_time: 0.8758580089, output_fired_time: 0.0000000000, propagate_forward_time: 63.2105636597, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2326507568
validation[          , genome  6407] predictions:    5829/  10000 (58.29%), best:    5829/10000 (58.29%), error:     12568.31445, best error:     12568.31445 on epoch:    43, epoch:   43/50, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 864.4040527344s, input_fired_time: 7.8224434853, output_fired_time: 5.5012083054, propagate_forward_time: 327.4510803223, propagate_backward_time: 506.8179016113, weight_update_time: 0.4886938930, other_time: 16.8113708496
epoch time: 73.1578521729s, input_fired_time: 0.9439026713, output_fired_time: 0.0000000000, propagate_forward_time: 68.9951934814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.2187576294
validation[          , genome  6407] predictions:    5877/  10000 (58.77%), best:    5877/10000 (58.77%), error:     12389.94336, best error:     12389.94336 on epoch:    44, epoch:   44/50, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 789.6193237305s, input_fired_time: 7.5713462830, output_fired_time: 5.1050167084, propagate_forward_time: 298.2161865234, propagate_backward_time: 461.4909362793, weight_update_time: 0.4488327801, other_time: 17.2358093262
epoch time: 60.6291885376s, input_fired_time: 0.8513271809, output_fired_time: 0.0000000000, propagate_forward_time: 56.3835411072, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.3943214417
validation[          , genome  6407] predictions:    5911/  10000 (59.11%), best:    5877/10000 (58.77%), error:     12453.46191, best error:     12389.94336 on epoch:    44, epoch:   45/50, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 753.9733276367s, input_fired_time: 7.4206385612, output_fired_time: 4.9365200996, propagate_forward_time: 279.3088989258, propagate_backward_time: 445.2856750488, weight_update_time: 0.4435239434, other_time: 17.0215759277
epoch time: 61.9851264954s, input_fired_time: 0.8946464062, output_fired_time: 0.0000000000, propagate_forward_time: 58.0315475464, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.0589332581
validation[          , genome  6407] predictions:    5884/  10000 (58.84%), best:    5877/10000 (58.77%), error:     12443.44043, best error:     12389.94336 on epoch:    44, epoch:   46/50, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 1299.1125488281s, input_fired_time: 10.4881410599, output_fired_time: 9.0285196304, propagate_forward_time: 512.0294189453, propagate_backward_time: 753.6370849609, weight_update_time: 0.7362582684, other_time: 13.9293212891
epoch time: 109.2566986084s, input_fired_time: 1.2942045927, output_fired_time: 0.0000000000, propagate_forward_time: 105.4740982056, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4883956909
validation[          , genome  6407] predictions:    6029/  10000 (60.29%), best:    6029/10000 (60.29%), error:     11949.79004, best error:     11949.79004 on epoch:    47, epoch:   47/50, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 1335.2220458984s, input_fired_time: 10.5347642899, output_fired_time: 9.1158533096, propagate_forward_time: 528.3671264648, propagate_backward_time: 773.1212768555, weight_update_time: 0.7342783809, other_time: 14.0830078125
epoch time: 107.8521194458s, input_fired_time: 1.2839756012, output_fired_time: 0.0000000000, propagate_forward_time: 104.0682983398, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4998474121
validation[          , genome  6407] predictions:    5981/  10000 (59.81%), best:    6029/10000 (60.29%), error:     12426.42676, best error:     11949.79004 on epoch:    47, epoch:   48/50, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 1318.1298828125s, input_fired_time: 10.4839076996, output_fired_time: 8.9939279556, propagate_forward_time: 525.1418457031, propagate_backward_time: 759.3836669922, weight_update_time: 0.7159909010, other_time: 14.1265869141
epoch time: 101.6452331543s, input_fired_time: 1.2027589083, output_fired_time: 0.0000000000, propagate_forward_time: 97.7856750488, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6567993164
validation[          , genome  6407] predictions:    6000/  10000 (60.00%), best:    6029/10000 (60.29%), error:     12327.85449, best error:     11949.79004 on epoch:    47, epoch:   49/50, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 1224.3651123047s, input_fired_time: 9.8249893188, output_fired_time: 8.2349109650, propagate_forward_time: 486.9334411621, propagate_backward_time: 705.0640869141, weight_update_time: 0.6566043496, other_time: 14.3077392578
epoch time: 97.4002838135s, input_fired_time: 1.1579364538, output_fired_time: 0.0000000000, propagate_forward_time: 93.5745391846, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_174_6407_0_r374941125_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.952297
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000961811
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 3.84725e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 50
read max_epochs: 50
read reset_weights: 0
read padding: 2
read best_epoch: 47
read number_validation_images: 10000
read best_validation_predictions: 6029
read best_validation_error: 11949.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6407
read normal distribution: '1 0.447710812091827 0.988357841968536'
generator_str: '1355626797'
read generator: 1355626797
read generated_by_map:
1 split_edge 1
reading 92 nodes.
reading 317 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1177.8974609375s, input_fired_time: 9.61798667907715, output_fired_time: 7.9939112663269, propagate_forward_time: 465.396759033203, propagate_backward_time: 680.018005371094, weight_update_time: 0.677420854568481, other_time: 14.8707885742188
epoch time: 96.5155639648438s, input_fired_time: 1.17418932914734, output_fired_time: 0, propagate_forward_time: 92.5452499389648, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.79612731933594
validation[          , genome  6407] predictions:    5984/  10000 (59.84%), best:    6029/10000 (60.29%), error:     12103.50488, best error:     11949.79004 on epoch:    47, epoch:   50/50, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 466.7770080566s, input_fired_time: 5.7266569138, output_fired_time: 0.0000000000, propagate_forward_time: 446.7022094727, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 14.3481445313
best training[          , genome  6407] predictions:   35779/  50000 (71.56%), best:    6029/10000 (60.29%), error:     41249.08594, best error:     11949.79004 on epoch:    47, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 89.1439285278s, input_fired_time: 1.1075810194, output_fired_time: 0.0000000000, propagate_forward_time: 85.1820678711, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8542785645
   testing[          , genome  6407] predictions:    6029/  10000 (60.29%), best:    6029/10000 (60.29%), error:     11949.79004, best error:     11949.79004 on epoch:    47, epoch:   51/50, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488
backpropagation finished successfully!
20:45:08 (16912): called boinc_finish(0)

</stderr_txt>
]]>