Nameexact_genome_1542031203_188_5598_0
Workunit4081783
Created12 Nov 2018, 19:30:54 UTC
Sent12 Nov 2018, 19:33:57 UTC
Report deadline2 Feb 2019, 9:21:05 UTC
Received22 Nov 2018, 17:35:14 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID23692
Run time1 days 11 hours 36 min 38 sec
CPU time1 days 8 hours 53 min 54 sec
Validate stateValid
Credit20,393.60
Device peak FLOPS4.15 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size537.07 MB
Peak swap size556.92 MB
Peak disk usage10.59 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
ch:   41/100, mu: 0.9431895018, learning_rate: 0.0000533767, weight_decay: 0.0000410379

epoch time: 1081.4539794922s, input_fired_time: 13.3687763214, output_fired_time: 6.8040833473, propagate_forward_time: 422.7062988281, propagate_backward_time: 626.4338378906, weight_update_time: 0.4495550990, other_time: 12.1409912109
epoch time: 90.2294311523s, input_fired_time: 1.0652827024, output_fired_time: 0.0000000000, propagate_forward_time: 87.0578536987, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1062927246
validation[          , genome  5598] predictions:    6262/  10000 (62.62%), best:    6262/10000 (62.62%), error:     10766.89551, best error:     10766.89551 on epoch:    42, epoch:   42/100, mu: 0.9457818866, learning_rate: 0.0000497546, weight_decay: 0.0000386032

epoch time: 1084.8291015625s, input_fired_time: 13.4414377213, output_fired_time: 6.8442435265, propagate_forward_time: 424.2023010254, propagate_backward_time: 628.0445556641, weight_update_time: 0.4552267790, other_time: 12.2966308594
epoch time: 87.1208419800s, input_fired_time: 1.0440311432, output_fired_time: 0.0000000000, propagate_forward_time: 83.9990615845, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0777511597
validation[          , genome  5598] predictions:    6359/  10000 (63.59%), best:    6359/10000 (63.59%), error:     10687.36328, best error:     10687.36328 on epoch:    43, epoch:   43/100, mu: 0.9482306838, learning_rate: 0.0000463782, weight_decay: 0.0000363130

epoch time: 1063.0399169922s, input_fired_time: 13.0662040710, output_fired_time: 6.6518917084, propagate_forward_time: 415.4154968262, propagate_backward_time: 615.9519042969, weight_update_time: 0.4431337416, other_time: 11.9544677734
epoch time: 85.3689193726s, input_fired_time: 1.0116221905, output_fired_time: 0.0000000000, propagate_forward_time: 82.3851776123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9721221924
validation[          , genome  5598] predictions:    6270/  10000 (62.70%), best:    6359/10000 (63.59%), error:     10750.02344, best error:     10687.36328 on epoch:    43, epoch:   44/100, mu: 0.9505438805, learning_rate: 0.0000432310, weight_decay: 0.0000341586

epoch time: 1057.3811035156s, input_fired_time: 13.0580444336, output_fired_time: 6.6660532951, propagate_forward_time: 413.4514160156, propagate_backward_time: 612.3021240234, weight_update_time: 0.4387520850, other_time: 11.9034423828
epoch time: 85.6275253296s, input_fired_time: 1.0411270857, output_fired_time: 0.0000000000, propagate_forward_time: 82.5883255005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9980697632
validation[          , genome  5598] predictions:    6261/  10000 (62.61%), best:    6359/10000 (63.59%), error:     10776.91406, best error:     10687.36328 on epoch:    43, epoch:   45/100, mu: 0.9527289867, learning_rate: 0.0000402974, weight_decay: 0.0000321321

epoch time: 1062.9147949219s, input_fired_time: 13.0538578033, output_fired_time: 6.6380515099, propagate_forward_time: 414.9270629883, propagate_backward_time: 616.5787963867, weight_update_time: 0.4293096066, other_time: 11.7170410156
epoch time: 85.7634353638s, input_fired_time: 1.0180549622, output_fired_time: 0.0000000000, propagate_forward_time: 82.7511444092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9942321777
validation[          , genome  5598] predictions:    6284/  10000 (62.84%), best:    6359/10000 (63.59%), error:     10712.33691, best error:     10687.36328 on epoch:    43, epoch:   46/100, mu: 0.9547930360, learning_rate: 0.0000375628, weight_decay: 0.0000302258

epoch time: 1055.9567871094s, input_fired_time: 12.9601287842, output_fired_time: 6.6412382126, propagate_forward_time: 412.5762634277, propagate_backward_time: 611.9287719727, weight_update_time: 0.4293617904, other_time: 11.8504028320
epoch time: 85.6168212891s, input_fired_time: 1.0160832405, output_fired_time: 0.0000000000, propagate_forward_time: 82.5825195313, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0182189941
validation[          , genome  5598] predictions:    6305/  10000 (63.05%), best:    6359/10000 (63.59%), error:     10724.83984, best error:     10687.36328 on epoch:    43, epoch:   47/100, mu: 0.9567428231, learning_rate: 0.0000350138, weight_decay: 0.0000284326

epoch time: 1072.9183349609s, input_fired_time: 13.1707859039, output_fired_time: 6.7471575737, propagate_forward_time: 418.6381225586, propagate_backward_time: 622.3687744141, weight_update_time: 0.4459252357, other_time: 11.9934692383
epoch time: 86.5186691284s, input_fired_time: 1.0212843418, output_fired_time: 0.0000000000, propagate_forward_time: 83.5640487671, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9333343506
validation[          , genome  5598] predictions:    6311/  10000 (63.11%), best:    6359/10000 (63.59%), error:     10700.60156, best error:     10687.36328 on epoch:    43, epoch:   48/100, mu: 0.9585846066, learning_rate: 0.0000326378, weight_decay: 0.0000267458

epoch time: 1058.2058105469s, input_fired_time: 13.0290441513, output_fired_time: 6.7046856880, propagate_forward_time: 413.9225463867, propagate_backward_time: 612.7011718750, weight_update_time: 0.4316952527, other_time: 11.8483276367
epoch time: 86.5025863647s, input_fired_time: 1.0232636929, output_fired_time: 0.0000000000, propagate_forward_time: 83.4927825928, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9865417480
validation[          , genome  5598] predictions:    6392/  10000 (63.92%), best:    6392/10000 (63.92%), error:     10490.33398, best error:     10490.33398 on epoch:    49, epoch:   49/100, mu: 0.9603244066, learning_rate: 0.0000304230, weight_decay: 0.0000251590

epoch time: 1057.4677734375s, input_fired_time: 13.0953226089, output_fired_time: 6.6306958199, propagate_forward_time: 413.2734680176, propagate_backward_time: 612.6005249023, weight_update_time: 0.4328759909, other_time: 11.8677368164
epoch time: 86.6461868286s, input_fired_time: 1.0251134634, output_fired_time: 0.0000000000, propagate_forward_time: 83.6042861938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0167846680
validation[          , genome  5598] predictions:    6242/  10000 (62.42%), best:    6392/10000 (63.92%), error:     10752.16699, best error:     10490.33398 on epoch:    49, epoch:   50/100, mu: 0.9619678259, learning_rate: 0.0000283585, weight_decay: 0.0000236664

epoch time: 1074.2458496094s, input_fired_time: 13.2189922333, output_fired_time: 6.7832036018, propagate_forward_time: 420.4964294434, propagate_backward_time: 621.8507690430, weight_update_time: 0.4357429743, other_time: 11.8964233398
epoch time: 87.0498962402s, input_fired_time: 1.0479986668, output_fired_time: 0.0000000000, propagate_forward_time: 83.9794464111, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0224533081
validation[          , genome  5598] predictions:    6320/  10000 (63.20%), best:    6392/10000 (63.92%), error:     10591.28320, best error:     10490.33398 on epoch:    49, epoch:   51/100, mu: 0.9635202289, learning_rate: 0.0000264341, weight_decay: 0.0000222623

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598_0_r294846561_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50605
read mu: 0.964987
read mu_delta: 0.94462
read initial_learning_rate: 0.000951961
read learning_rate: 2.46403e-05
read learning_rate_delta: 0.93214
read initial_weight_decay: 0.000503734
read weight_decay: 2.09416e-05
read weight_decay_delta: 0.940673
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0868122
read input_dropout_probability: 0.0011958
read hidden_dropout_probability: 0.0988721
read velocity_reset: 1656
read epoch: 52
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 49
read number_validation_images: 10000
read best_validation_predictions: 6392
read best_validation_error: 10490.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5598
read normal distribution: '0 -2.47144103050232 -0.697616279125214'
generator_str: '966256949'
read generator: 966256949
read generated_by_map:
1 change_size 1
reading 70 nodes.
reading 293 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1072.66735839844s, input_fired_time: 13.0623722076416, output_fired_time: 6.82428979873657, propagate_forward_time: 416.767028808594, propagate_backward_time: 624.002075195313, weight_update_time: 0.435981303453445, other_time: 12.0115356445313
epoch time: 86.567268371582s, input_fired_time: 1.02756869792938, output_fired_time: 0, propagate_forward_time: 83.6166763305664, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.92302703857422
validation[          , genome  5598] predictions:    6315/  10000 (63.15%), best:    6392/10000 (63.92%), error:     10599.98730, best error:     10490.33398 on epoch:    49, epoch:   52/100, mu: 0.9649866819, learning_rate: 0.0000246403, weight_decay: 0.0000209416

epoch time: 1081.2785644531s, input_fired_time: 13.1415138245, output_fired_time: 6.7955999374, propagate_forward_time: 419.3996887207, propagate_backward_time: 630.0402832031, weight_update_time: 0.4367635250, other_time: 11.9014892578
epoch time: 86.9942550659s, input_fired_time: 1.0622924566, output_fired_time: 0.0000000000, propagate_forward_time: 83.8480834961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0838775635
validation[          , genome  5598] predictions:    6253/  10000 (62.53%), best:    6392/10000 (63.92%), error:     10619.68457, best error:     10490.33398 on epoch:    49, epoch:   53/100, mu: 0.9663718939, learning_rate: 0.0000229682, weight_decay: 0.0000196992

epoch time: 1076.9077148438s, input_fired_time: 13.1069822311, output_fired_time: 6.8715538979, propagate_forward_time: 418.5854492188, propagate_backward_time: 626.2914428711, weight_update_time: 0.4386954606, other_time: 12.0523071289
epoch time: 88.6133956909s, input_fired_time: 1.0727417469, output_fired_time: 0.0000000000, propagate_forward_time: 85.4385223389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1021347046
validation[          , genome  5598] predictions:    6256/  10000 (62.56%), best:    6392/10000 (63.92%), error:     10670.70508, best error:     10490.33398 on epoch:    49, epoch:   54/100, mu: 0.9676803946, learning_rate: 0.0000214096, weight_decay: 0.0000185305

epoch time: 1067.2375488281s, input_fired_time: 13.0681972504, output_fired_time: 6.7501287460, propagate_forward_time: 414.0668945313, propagate_backward_time: 621.4179687500, weight_update_time: 0.4488570392, other_time: 11.9343261719
epoch time: 86.4769134521s, input_fired_time: 1.0400910378, output_fired_time: 0.0000000000, propagate_forward_time: 83.4595870972, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9772338867
validation[          , genome  5598] predictions:    6256/  10000 (62.56%), best:    6392/10000 (63.92%), error:     10643.88281, best error:     10490.33398 on epoch:    49, epoch:   55/100, mu: 0.9689164758, learning_rate: 0.0000199567, weight_decay: 0.0000174311

epoch time: 1072.4096679688s, input_fired_time: 13.0696620941, output_fired_time: 6.7680859566, propagate_forward_time: 417.1969299316, propagate_backward_time: 623.5408325195, weight_update_time: 0.4356066883, other_time: 11.8341674805
epoch time: 85.9156112671s, input_fired_time: 1.0466849804, output_fired_time: 0.0000000000, propagate_forward_time: 82.8198242188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0491027832
validation[          , genome  5598] predictions:    6378/  10000 (63.78%), best:    6378/10000 (63.78%), error:     10457.87109, best error:     10457.87109 on epoch:    56, epoch:   56/100, mu: 0.9700840712, learning_rate: 0.0000186025, weight_decay: 0.0000163970

epoch time: 1080.6207275391s, input_fired_time: 13.0732011795, output_fired_time: 6.8491854668, propagate_forward_time: 420.2774963379, propagate_backward_time: 628.5323486328, weight_update_time: 0.4381311834, other_time: 11.8884277344
epoch time: 87.5399398804s, input_fired_time: 1.0399731398, output_fired_time: 0.0000000000, propagate_forward_time: 84.4884872437, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0114822388
validation[          , genome  5598] predictions:    6320/  10000 (63.20%), best:    6378/10000 (63.78%), error:     10533.43457, best error:     10457.87109 on epoch:    56, epoch:   57/100, mu: 0.9711869955, learning_rate: 0.0000173401, weight_decay: 0.0000154242

epoch time: 1085.5532226563s, input_fired_time: 13.1421537399, output_fired_time: 6.7750859261, propagate_forward_time: 422.0696411133, propagate_backward_time: 631.8817749023, weight_update_time: 0.4322243631, other_time: 11.6844482422
epoch time: 87.8501586914s, input_fired_time: 1.0373641253, output_fired_time: 0.0000000000, propagate_forward_time: 84.9163894653, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8964080811
validation[          , genome  5598] predictions:    6399/  10000 (63.99%), best:    6399/10000 (63.99%), error:     10404.49512, best error:     10404.49512 on epoch:    58, epoch:   58/100, mu: 0.9722288847, learning_rate: 0.0000161634, weight_decay: 0.0000145091

epoch time: 1093.2285156250s, input_fired_time: 13.1088314056, output_fired_time: 6.8050599098, propagate_forward_time: 425.1105651855, propagate_backward_time: 636.6551513672, weight_update_time: 0.4288911521, other_time: 11.5488281250
epoch time: 88.5330886841s, input_fired_time: 1.0698901415, output_fired_time: 0.0000000000, propagate_forward_time: 85.4789276123, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9842681885
validation[          , genome  5598] predictions:    6399/  10000 (63.99%), best:    6399/10000 (63.99%), error:     10413.47168, best error:     10404.49512 on epoch:    58, epoch:   59/100, mu: 0.9732130766, learning_rate: 0.0000150666, weight_decay: 0.0000136483

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598_0_r294846561_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50605
read mu: 0.974143
read mu_delta: 0.94462
read initial_learning_rate: 0.000951961
read learning_rate: 1.40442e-05
read learning_rate_delta: 0.93214
read initial_weight_decay: 0.000503734
read weight_decay: 1.28386e-05
read weight_decay_delta: 0.940673
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0868122
read input_dropout_probability: 0.0011958
read hidden_dropout_probability: 0.0988721
read velocity_reset: 1656
read epoch: 60
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 58
read number_validation_images: 10000
read best_validation_predictions: 6399
read best_validation_error: 10404.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5598
read normal distribution: '0 -2.47144103050232 -0.697616279125214'
generator_str: '832509110'
read generator: 832509110
read generated_by_map:
1 change_size 1
reading 70 nodes.
reading 293 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1130.681640625s, input_fired_time: 13.1993417739868, output_fired_time: 6.77110624313354, propagate_forward_time: 449.923126220703, propagate_backward_time: 650.419799804688, weight_update_time: 0.424100428819656, other_time: 10.3682861328125
epoch time: 95.6379623413086s, input_fired_time: 1.06283986568451, output_fired_time: 0, propagate_forward_time: 92.7800903320313, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.79502868652344
validation[          , genome  5598] predictions:    6362/  10000 (63.62%), best:    6399/10000 (63.99%), error:     10508.01758, best error:     10404.49512 on epoch:    58, epoch:   60/100, mu: 0.9741427302, learning_rate: 0.0000140442, weight_decay: 0.0000128386

epoch time: 1161.0496826172s, input_fired_time: 13.8142175674, output_fired_time: 7.0223522186, propagate_forward_time: 463.0985107422, propagate_backward_time: 666.4182739258, weight_update_time: 0.4682103992, other_time: 10.6963500977
epoch time: 94.7405014038s, input_fired_time: 1.0559288263, output_fired_time: 0.0000000000, propagate_forward_time: 91.9361801147, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7483901978
validation[          , genome  5598] predictions:    6400/  10000 (64.00%), best:    6400/10000 (64.00%), error:     10357.04199, best error:     10357.04199 on epoch:    61, epoch:   61/100, mu: 0.9750208855, learning_rate: 0.0000130911, weight_decay: 0.0000120769

epoch time: 1151.3499755859s, input_fired_time: 13.3062105179, output_fired_time: 6.8961591721, propagate_forward_time: 461.4159545898, propagate_backward_time: 659.1719360352, weight_update_time: 0.4370233417, other_time: 10.5598144531
epoch time: 100.3293914795s, input_fired_time: 1.1040925980, output_fired_time: 0.0000000000, propagate_forward_time: 97.3515396118, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8737564087
validation[          , genome  5598] predictions:    6376/  10000 (63.76%), best:    6400/10000 (64.00%), error:     10393.25195, best error:     10357.04199 on epoch:    61, epoch:   62/100, mu: 0.9758504033, learning_rate: 0.0000122028, weight_decay: 0.0000113605

epoch time: 32792.4921875000s, input_fired_time: 13.4720344543, output_fired_time: 7.0766530037, propagate_forward_time: 32115.7285156250, propagate_backward_time: 641.4503173828, weight_update_time: 0.4771129191, other_time: 14.7625732422
epoch time: 90.5183868408s, input_fired_time: 1.1136186123, output_fired_time: 0.0000000000, propagate_forward_time: 86.7840423584, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6207275391
validation[          , genome  5598] predictions:    6403/  10000 (64.03%), best:    6400/10000 (64.00%), error:     10455.34961, best error:     10357.04199 on epoch:    61, epoch:   63/100, mu: 0.9766340256, learning_rate: 0.0000113747, weight_decay: 0.0000106865

epoch time: 1102.3980712891s, input_fired_time: 13.4835348129, output_fired_time: 7.0352892876, propagate_forward_time: 429.1718444824, propagate_backward_time: 639.8338623047, weight_update_time: 0.4583767951, other_time: 12.8735351563
epoch time: 89.6933517456s, input_fired_time: 1.1117340326, output_fired_time: 0.0000000000, propagate_forward_time: 86.3935623169, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1880569458
validation[          , genome  5598] predictions:    6371/  10000 (63.71%), best:    6400/10000 (64.00%), error:     10437.42969, best error:     10357.04199 on epoch:    61, epoch:   64/100, mu: 0.9773742557, learning_rate: 0.0000106028, weight_decay: 0.0000100525

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598_0_r294846561_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50605
read mu: 0.978073
read mu_delta: 0.94462
read initial_learning_rate: 0.000951961
read learning_rate: 9.88331e-06
read learning_rate_delta: 0.93214
read initial_weight_decay: 0.000503734
read weight_decay: 9.45609e-06
read weight_decay_delta: 0.940673
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0868122
read input_dropout_probability: 0.0011958
read hidden_dropout_probability: 0.0988721
read velocity_reset: 1656
read epoch: 65
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 61
read number_validation_images: 10000
read best_validation_predictions: 6400
read best_validation_error: 10357
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5598
read normal distribution: '0 -2.47144103050232 -0.697616279125214'
generator_str: '2061923098'
read generator: 2061923098
read generated_by_map:
1 change_size 1
reading 70 nodes.
reading 293 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1131.29504394531s, input_fired_time: 13.8356695175171, output_fired_time: 7.28529357910156, propagate_forward_time: 440.783569335938, propagate_backward_time: 656.104431152344, weight_update_time: 0.480954706668854, other_time: 13.2860717773438
epoch time: 90.0514221191406s, input_fired_time: 1.101597905159, output_fired_time: 0, propagate_forward_time: 86.6883697509766, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.26145172119141
validation[          , genome  5598] predictions:    6363/  10000 (63.63%), best:    6400/10000 (64.00%), error:     10410.93848, best error:     10357.04199 on epoch:    61, epoch:   65/100, mu: 0.9780734777, learning_rate: 0.0000098833, weight_decay: 0.0000094561

epoch time: 1109.5146484375s, input_fired_time: 13.5246906281, output_fired_time: 7.0393242836, propagate_forward_time: 431.9261474609, propagate_backward_time: 644.2068481445, weight_update_time: 0.4667226374, other_time: 12.8176879883
epoch time: 90.3939437866s, input_fired_time: 1.1167016029, output_fired_time: 0.0000000000, propagate_forward_time: 87.0502395630, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2270050049
validation[          , genome  5598] predictions:    6360/  10000 (63.60%), best:    6400/10000 (64.00%), error:     10395.11914, best error:     10357.04199 on epoch:    61, epoch:   66/100, mu: 0.9787339568, learning_rate: 0.0000092126, weight_decay: 0.0000088951

epoch time: 1118.6937255859s, input_fired_time: 13.2626857758, output_fired_time: 6.8822369576, propagate_forward_time: 443.0370483398, propagate_backward_time: 644.1283569336, weight_update_time: 0.4439277649, other_time: 11.3834228516
epoch time: 93.3381195068s, input_fired_time: 1.0538384914, output_fired_time: 0.0000000000, propagate_forward_time: 90.4947662354, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7895126343
validation[          , genome  5598] predictions:    6420/  10000 (64.20%), best:    6420/10000 (64.20%), error:     10314.27246, best error:     10314.27246 on epoch:    67, epoch:   67/100, mu: 0.9793578982, learning_rate: 0.0000085875, weight_decay: 0.0000083674

epoch time: 24674.4140625000s, input_fired_time: 12.9611196518, output_fired_time: 23577.6445312500, propagate_forward_time: 426.1069946289, propagate_backward_time: 647.4151000977, weight_update_time: 0.3942403793, other_time: 10.2864990234
epoch time: 88.1684036255s, input_fired_time: 1.0181854963, output_fired_time: 0.0000000000, propagate_forward_time: 85.4277343750, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7224807739
validation[          , genome  5598] predictions:    6367/  10000 (63.67%), best:    6420/10000 (64.20%), error:     10341.81445, best error:     10314.27246 on epoch:    67, epoch:   68/100, mu: 0.9799472690, learning_rate: 0.0000080047, weight_decay: 0.0000078710

epoch time: 1101.7415771484s, input_fired_time: 12.8290185928, output_fired_time: 6.4441466331, propagate_forward_time: 426.1540222168, propagate_backward_time: 646.1855468750, weight_update_time: 0.4022535682, other_time: 10.1289062500
epoch time: 88.0189285278s, input_fired_time: 1.0522195101, output_fired_time: 0.0000000000, propagate_forward_time: 85.3094024658, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6573028564
validation[          , genome  5598] predictions:    6406/  10000 (64.06%), best:    6420/10000 (64.20%), error:     10369.87012, best error:     10314.27246 on epoch:    67, epoch:   69/100, mu: 0.9805039763, learning_rate: 0.0000074615, weight_decay: 0.0000074040

epoch time: 1115.1539306641s, input_fired_time: 12.9347848892, output_fired_time: 6.5368561745, propagate_forward_time: 430.8559875488, propagate_backward_time: 654.5736083984, weight_update_time: 0.4012116492, other_time: 10.2526855469
epoch time: 98.8493347168s, input_fired_time: 1.1447107792, output_fired_time: 0.0000000000, propagate_forward_time: 95.8383255005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8662948608
validation[          , genome  5598] predictions:    6403/  10000 (64.03%), best:    6420/10000 (64.20%), error:     10324.24805, best error:     10314.27246 on epoch:    67, epoch:   70/100, mu: 0.9810298681, learning_rate: 0.0000069552, weight_decay: 0.0000069647

epoch time: 1454.6651611328s, input_fired_time: 13.2129364014, output_fired_time: 6.5956640244, propagate_forward_time: 434.0575256348, propagate_backward_time: 990.5000000000, weight_update_time: 0.4009157121, other_time: 10.2990722656
epoch time: 93.7829513550s, input_fired_time: 1.0679484606, output_fired_time: 0.0000000000, propagate_forward_time: 90.9018325806, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8131713867
validation[          , genome  5598] predictions:    6391/  10000 (63.91%), best:    6420/10000 (64.20%), error:     10340.22266, best error:     10314.27246 on epoch:    67, epoch:   71/100, mu: 0.9815266132, learning_rate: 0.0000064832, weight_decay: 0.0000065515

epoch time: 1155.9483642578s, input_fired_time: 13.6345720291, output_fired_time: 6.8094844818, propagate_forward_time: 447.0145263672, propagate_backward_time: 677.4045410156, weight_update_time: 0.4167008400, other_time: 11.0853271484
epoch time: 93.7708663940s, input_fired_time: 1.1102226973, output_fired_time: 0.0000000000, propagate_forward_time: 90.7665786743, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8940658569
validation[          , genome  5598] predictions:    6413/  10000 (64.13%), best:    6420/10000 (64.20%), error:     10327.88379, best error:     10314.27246 on epoch:    67, epoch:   72/100, mu: 0.9819958806, learning_rate: 0.0000060433, weight_decay: 0.0000061629

epoch time: 1103.3051757813s, input_fired_time: 12.9518680573, output_fired_time: 6.5223879814, propagate_forward_time: 426.3008728027, propagate_backward_time: 647.0015869141, weight_update_time: 0.4046438038, other_time: 10.5284423828
epoch time: 87.8905258179s, input_fired_time: 1.0283951759, output_fired_time: 0.0000000000, propagate_forward_time: 85.1570968628, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7050323486
validation[          , genome  5598] predictions:    6440/  10000 (64.40%), best:    6440/10000 (64.40%), error:     10279.44043, best error:     10279.44043 on epoch:    73, epoch:   73/100, mu: 0.9824391603, learning_rate: 0.0000056332, weight_decay: 0.0000057972

epoch time: 1099.1674804688s, input_fired_time: 12.8781480789, output_fired_time: 6.4716148376, propagate_forward_time: 424.6618041992, propagate_backward_time: 644.6945190430, weight_update_time: 0.4009059370, other_time: 10.4614257813
epoch time: 89.1452331543s, input_fired_time: 1.0283353329, output_fired_time: 0.0000000000, propagate_forward_time: 86.4382553101, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6786422729
validation[          , genome  5598] predictions:    6389/  10000 (63.89%), best:    6440/10000 (64.40%), error:     10331.12891, best error:     10279.44043 on epoch:    73, epoch:   74/100, mu: 0.9828578830, learning_rate: 0.0000052509, weight_decay: 0.0000054533

epoch time: 1094.5909423828s, input_fired_time: 12.8964328766, output_fired_time: 6.4211778641, propagate_forward_time: 422.6840209961, propagate_backward_time: 642.3638916016, weight_update_time: 0.4012902975, other_time: 10.2254028320
epoch time: 87.3384094238s, input_fired_time: 1.0186690092, output_fired_time: 0.0000000000, propagate_forward_time: 84.6201858521, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6995544434
validation[          , genome  5598] predictions:    6409/  10000 (64.09%), best:    6440/10000 (64.40%), error:     10327.83398, best error:     10279.44043 on epoch:    73, epoch:   75/100, mu: 0.9832534194, learning_rate: 0.0000048946, weight_decay: 0.0000051298

epoch time: 1097.7663574219s, input_fired_time: 12.9114122391, output_fired_time: 6.5093903542, propagate_forward_time: 424.2829589844, propagate_backward_time: 643.4719848633, weight_update_time: 0.4093961716, other_time: 10.5906372070
epoch time: 88.4309234619s, input_fired_time: 1.0422041416, output_fired_time: 0.0000000000, propagate_forward_time: 85.6118469238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7768707275
validation[          , genome  5598] predictions:    6416/  10000 (64.16%), best:    6440/10000 (64.40%), error:     10314.39160, best error:     10279.44043 on epoch:    73, epoch:   76/100, mu: 0.9836270213, learning_rate: 0.0000045624, weight_decay: 0.0000048254

epoch time: 1096.5079345703s, input_fired_time: 12.9756135941, output_fired_time: 6.5529842377, propagate_forward_time: 425.5310363770, propagate_backward_time: 641.1113891602, weight_update_time: 0.4054839611, other_time: 10.3369750977
epoch time: 87.7702484131s, input_fired_time: 1.0275229216, output_fired_time: 0.0000000000, propagate_forward_time: 85.0252456665, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7174835205
validation[          , genome  5598] predictions:    6440/  10000 (64.40%), best:    6440/10000 (64.40%), error:     10283.80078, best error:     10279.44043 on epoch:    73, epoch:   77/100, mu: 0.9839799404, learning_rate: 0.0000042528, weight_decay: 0.0000045392

epoch time: 1092.4816894531s, input_fired_time: 13.0365180969, output_fired_time: 6.5141887665, propagate_forward_time: 424.2832031250, propagate_backward_time: 638.2954101563, weight_update_time: 0.4060534537, other_time: 10.3524169922
epoch time: 89.6750030518s, input_fired_time: 1.0755991936, output_fired_time: 0.0000000000, propagate_forward_time: 86.8577194214, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7416839600
validation[          , genome  5598] predictions:    6405/  10000 (64.05%), best:    6440/10000 (64.40%), error:     10283.27246, best error:     10279.44043 on epoch:    73, epoch:   78/100, mu: 0.9843133092, learning_rate: 0.0000039642, weight_decay: 0.0000042699

epoch time: 1119.0010986328s, input_fired_time: 13.3000030518, output_fired_time: 6.7034778595, propagate_forward_time: 433.8534240723, propagate_backward_time: 654.4567871094, weight_update_time: 0.4183694422, other_time: 10.6873779297
epoch time: 90.0041275024s, input_fired_time: 1.0687061548, output_fired_time: 0.0000000000, propagate_forward_time: 87.1120376587, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8233871460
validation[          , genome  5598] predictions:    6442/  10000 (64.42%), best:    6442/10000 (64.42%), error:     10263.34766, best error:     10263.34766 on epoch:    79, epoch:   79/100, mu: 0.9846282601, learning_rate: 0.0000036952, weight_decay: 0.0000040165

epoch time: 1135.1346435547s, input_fired_time: 13.4347467422, output_fired_time: 6.8132295609, propagate_forward_time: 441.1183776855, propagate_backward_time: 662.8511962891, weight_update_time: 0.4239116311, other_time: 10.9171142578
epoch time: 90.0723495483s, input_fired_time: 1.0577276945, output_fired_time: 0.0000000000, propagate_forward_time: 87.2265090942, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7881164551
validation[          , genome  5598] predictions:    6423/  10000 (64.23%), best:    6442/10000 (64.42%), error:     10264.73340, best error:     10263.34766 on epoch:    79, epoch:   80/100, mu: 0.9849257469, learning_rate: 0.0000034445, weight_decay: 0.0000037783

epoch time: 1119.9077148438s, input_fired_time: 13.2786884308, output_fired_time: 6.6315488815, propagate_forward_time: 434.1204223633, propagate_backward_time: 655.1746215820, weight_update_time: 0.4144611359, other_time: 10.7023925781
epoch time: 89.2948150635s, input_fired_time: 1.0583214760, output_fired_time: 0.0000000000, propagate_forward_time: 86.4748535156, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7616424561
validation[          , genome  5598] predictions:    6454/  10000 (64.54%), best:    6454/10000 (64.54%), error:     10218.34375, best error:     10218.34375 on epoch:    81, epoch:   81/100, mu: 0.9852067828, learning_rate: 0.0000032107, weight_decay: 0.0000035541

epoch time: 1112.3525390625s, input_fired_time: 13.2903928757, output_fired_time: 6.6244955063, propagate_forward_time: 432.1481323242, propagate_backward_time: 649.7344970703, weight_update_time: 0.4166854918, other_time: 10.5549926758
epoch time: 88.5038604736s, input_fired_time: 1.0605337620, output_fired_time: 0.0000000000, propagate_forward_time: 85.7043533325, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7389755249
validation[          , genome  5598] predictions:    6468/  10000 (64.68%), best:    6454/10000 (64.54%), error:     10232.86426, best error:     10218.34375 on epoch:    81, epoch:   82/100, mu: 0.9854722023, learning_rate: 0.0000029928, weight_decay: 0.0000033432

epoch time: 1106.7106933594s, input_fired_time: 13.1789865494, output_fired_time: 6.6008915901, propagate_forward_time: 429.8297424316, propagate_backward_time: 646.5761108398, weight_update_time: 0.4058932960, other_time: 10.5249633789
epoch time: 88.6520156860s, input_fired_time: 1.0414242744, output_fired_time: 0.0000000000, propagate_forward_time: 85.8692855835, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7413024902
validation[          , genome  5598] predictions:    6438/  10000 (64.38%), best:    6454/10000 (64.54%), error:     10255.46191, best error:     10218.34375 on epoch:    81, epoch:   83/100, mu: 0.9857229590, learning_rate: 0.0000027897, weight_decay: 0.0000031449

epoch time: 1106.7728271484s, input_fired_time: 13.2051868439, output_fired_time: 6.6292061806, propagate_forward_time: 430.0818481445, propagate_backward_time: 646.3815307617, weight_update_time: 0.4080636203, other_time: 10.4750976563
epoch time: 89.1533279419s, input_fired_time: 1.0452549458, output_fired_time: 0.0000000000, propagate_forward_time: 86.3301620483, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7779083252
validation[          , genome  5598] predictions:    6411/  10000 (64.11%), best:    6454/10000 (64.54%), error:     10296.79980, best error:     10218.34375 on epoch:    81, epoch:   84/100, mu: 0.9859598279, learning_rate: 0.0000026004, weight_decay: 0.0000029583

epoch time: 1107.8345947266s, input_fired_time: 13.1533918381, output_fired_time: 6.5863237381, propagate_forward_time: 429.6980285645, propagate_backward_time: 647.8946533203, weight_update_time: 0.4078098536, other_time: 10.5021972656
epoch time: 88.8924713135s, input_fired_time: 1.0685956478, output_fired_time: 0.0000000000, propagate_forward_time: 86.0706863403, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7531890869
validation[          , genome  5598] predictions:    6415/  10000 (64.15%), best:    6454/10000 (64.54%), error:     10248.38672, best error:     10218.34375 on epoch:    81, epoch:   85/100, mu: 0.9861835837, learning_rate: 0.0000024240, weight_decay: 0.0000027828

epoch time: 1114.0524902344s, input_fired_time: 13.1731424332, output_fired_time: 6.5609750748, propagate_forward_time: 430.8739013672, propagate_backward_time: 652.8882446289, weight_update_time: 0.4107882380, other_time: 10.5562133789
epoch time: 89.0398635864s, input_fired_time: 1.0491672754, output_fired_time: 0.0000000000, propagate_forward_time: 86.2457046509, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7449951172
validation[          , genome  5598] predictions:    6426/  10000 (64.26%), best:    6454/10000 (64.54%), error:     10241.08398, best error:     10218.34375 on epoch:    81, epoch:   86/100, mu: 0.9863949418, learning_rate: 0.0000022595, weight_decay: 0.0000026177

epoch time: 1111.5856933594s, input_fired_time: 13.2007837296, output_fired_time: 6.6422281265, propagate_forward_time: 432.2489013672, propagate_backward_time: 648.9044189453, weight_update_time: 0.4120279849, other_time: 10.5893554688
epoch time: 89.3063659668s, input_fired_time: 1.0474979877, output_fired_time: 0.0000000000, propagate_forward_time: 86.4778823853, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7809829712
validation[          , genome  5598] predictions:    6454/  10000 (64.54%), best:    6454/10000 (64.54%), error:     10231.97168, best error:     10218.34375 on epoch:    81, epoch:   87/100, mu: 0.9865946174, learning_rate: 0.0000021062, weight_decay: 0.0000024624

epoch time: 1110.7869873047s, input_fired_time: 13.1952695847, output_fired_time: 6.6442995071, propagate_forward_time: 430.3996276855, propagate_backward_time: 649.9930419922, weight_update_time: 0.4133282006, other_time: 10.5546875000
epoch time: 88.9288177490s, input_fired_time: 1.0421282053, output_fired_time: 0.0000000000, propagate_forward_time: 86.1416473389, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7450408936
validation[          , genome  5598] predictions:    6446/  10000 (64.46%), best:    6454/10000 (64.54%), error:     10256.01563, best error:     10218.34375 on epoch:    81, epoch:   88/100, mu: 0.9867832065, learning_rate: 0.0000019632, weight_decay: 0.0000023163

epoch time: 24760.5156250000s, input_fired_time: 13.1463766098, output_fired_time: 6.5360383987, propagate_forward_time: 429.8228759766, propagate_backward_time: 24300.6113281250, weight_update_time: 0.3987777829, other_time: 10.4003906250
epoch time: 86.3410339355s, input_fired_time: 1.0286821127, output_fired_time: 0.0000000000, propagate_forward_time: 83.7080459595, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6043090820
validation[          , genome  5598] predictions:    6465/  10000 (64.65%), best:    6465/10000 (64.65%), error:     10201.87402, best error:     10201.87402 on epoch:    89, epoch:   89/100, mu: 0.9869613647, learning_rate: 0.0000018300, weight_decay: 0.0000021789

epoch time: 1089.7542724609s, input_fired_time: 12.7568702698, output_fired_time: 6.3136625290, propagate_forward_time: 418.4100341797, propagate_backward_time: 642.5751953125, weight_update_time: 0.3846137822, other_time: 9.6984863281
epoch time: 87.2717132568s, input_fired_time: 1.0347237587, output_fired_time: 0.0000000000, propagate_forward_time: 84.5789642334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6580276489
validation[          , genome  5598] predictions:    6449/  10000 (64.49%), best:    6465/10000 (64.65%), error:     10202.53516, best error:     10201.87402 on epoch:    89, epoch:   90/100, mu: 0.9871296287, learning_rate: 0.0000017058, weight_decay: 0.0000020496

epoch time: 1094.4624023438s, input_fired_time: 12.9278526306, output_fired_time: 6.3710041046, propagate_forward_time: 421.3514099121, propagate_backward_time: 643.9262084961, weight_update_time: 0.3884100318, other_time: 9.8859252930
epoch time: 87.0506973267s, input_fired_time: 1.0364403725, output_fired_time: 0.0000000000, propagate_forward_time: 84.3574371338, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6568222046
validation[          , genome  5598] predictions:    6465/  10000 (64.65%), best:    6465/10000 (64.65%), error:     10209.49805, best error:     10201.87402 on epoch:    89, epoch:   91/100, mu: 0.9872885942, learning_rate: 0.0000015901, weight_decay: 0.0000019280

epoch time: 1099.5812988281s, input_fired_time: 13.0213212967, output_fired_time: 6.4373326302, propagate_forward_time: 423.6569824219, propagate_backward_time: 646.4270019531, weight_update_time: 0.3909956813, other_time: 10.0385742188
epoch time: 87.2796325684s, input_fired_time: 1.0214828253, output_fired_time: 0.0000000000, propagate_forward_time: 84.6039962769, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6541519165
validation[          , genome  5598] predictions:    6444/  10000 (64.44%), best:    6465/10000 (64.65%), error:     10204.27344, best error:     10201.87402 on epoch:    89, epoch:   92/100, mu: 0.9874387383, learning_rate: 0.0000014822, weight_decay: 0.0000018137

epoch time: 1088.7536621094s, input_fired_time: 12.8019590378, output_fired_time: 6.3882565498, propagate_forward_time: 420.5006103516, propagate_backward_time: 639.2239379883, weight_update_time: 0.3919349909, other_time: 9.8388061523
epoch time: 86.9428253174s, input_fired_time: 1.0233130455, output_fired_time: 0.0000000000, propagate_forward_time: 84.2998962402, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6196136475
validation[          , genome  5598] predictions:    6443/  10000 (64.43%), best:    6465/10000 (64.65%), error:     10202.86523, best error:     10201.87402 on epoch:    89, epoch:   93/100, mu: 0.9875805974, learning_rate: 0.0000013816, weight_decay: 0.0000017061

epoch time: 1091.4702148438s, input_fired_time: 12.8432025909, output_fired_time: 6.4081773758, propagate_forward_time: 421.3174438477, propagate_backward_time: 641.0494384766, weight_update_time: 0.3891811967, other_time: 9.8518676758
epoch time: 86.7469635010s, input_fired_time: 1.0284291506, output_fired_time: 0.0000000000, propagate_forward_time: 84.0897369385, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6287994385
validation[          , genome  5598] predictions:    6447/  10000 (64.47%), best:    6447/10000 (64.47%), error:     10175.16113, best error:     10175.16113 on epoch:    94, epoch:   94/100, mu: 0.9877145886, learning_rate: 0.0000012878, weight_decay: 0.0000016048

epoch time: 1088.6740722656s, input_fired_time: 12.8124122620, output_fired_time: 6.3312368393, propagate_forward_time: 419.9007263184, propagate_backward_time: 639.7971801758, weight_update_time: 0.3891813159, other_time: 9.8325805664
epoch time: 87.3952102661s, input_fired_time: 1.0268409252, output_fired_time: 0.0000000000, propagate_forward_time: 84.6949844360, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6733856201
validation[          , genome  5598] predictions:    6479/  10000 (64.79%), best:    6479/10000 (64.79%), error:     10171.25879, best error:     10171.25879 on epoch:    95, epoch:   95/100, mu: 0.9878411293, learning_rate: 0.0000012004, weight_decay: 0.0000015096

epoch time: 1092.2802734375s, input_fired_time: 12.8857755661, output_fired_time: 6.3804154396, propagate_forward_time: 421.4030761719, propagate_backward_time: 641.7678833008, weight_update_time: 0.3975532949, other_time: 9.8432006836
epoch time: 86.7386627197s, input_fired_time: 1.0223134756, output_fired_time: 0.0000000000, propagate_forward_time: 84.0544586182, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6618881226
validation[          , genome  5598] predictions:    6443/  10000 (64.43%), best:    6479/10000 (64.79%), error:     10176.11523, best error:     10171.25879 on epoch:    95, epoch:   96/100, mu: 0.9879606962, learning_rate: 0.0000011190, weight_decay: 0.0000014201

epoch time: 1089.8555908203s, input_fired_time: 12.7925539017, output_fired_time: 6.3329472542, propagate_forward_time: 420.5283508301, propagate_backward_time: 640.2972412109, weight_update_time: 0.3976125419, other_time: 9.9044189453
epoch time: 87.1108169556s, input_fired_time: 1.0208188295, output_fired_time: 0.0000000000, propagate_forward_time: 84.4491348267, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6408615112
validation[          , genome  5598] predictions:    6456/  10000 (64.56%), best:    6479/10000 (64.79%), error:     10183.05078, best error:     10171.25879 on epoch:    95, epoch:   97/100, mu: 0.9880736470, learning_rate: 0.0000010430, weight_decay: 0.0000013358

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542031203_188_5598_0_r294846561_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.50605
read mu: 0.98818
read mu_delta: 0.94462
read initial_learning_rate: 0.000951961
read learning_rate: 9.72265e-07
read learning_rate_delta: 0.93214
read initial_weight_decay: 0.000503734
read weight_decay: 1.25657e-06
read weight_decay_delta: 0.940673
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0868122
read input_dropout_probability: 0.0011958
read hidden_dropout_probability: 0.0988721
read velocity_reset: 1656
read epoch: 98
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 95
read number_validation_images: 10000
read best_validation_predictions: 6479
read best_validation_error: 10171.3
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5598
read normal distribution: '0 -2.47144103050232 -0.697616279125214'
generator_str: '802907408'
read generator: 802907408
read generated_by_map:
1 change_size 1
reading 70 nodes.
reading 293 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1119.57824707031s, input_fired_time: 13.2974586486816, output_fired_time: 6.49473094940186, propagate_forward_time: 431.681762695313, propagate_backward_time: 658.036010742188, weight_update_time: 0.399000316858292, other_time: 10.0682373046875
epoch time: 88.2776184082031s, input_fired_time: 1.05261957645416, output_fired_time: 0, propagate_forward_time: 85.5564651489258, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.66853332519531
validation[          , genome  5598] predictions:    6469/  10000 (64.69%), best:    6469/10000 (64.69%), error:     10166.44336, best error:     10166.44336 on epoch:    98, epoch:   98/100, mu: 0.9881803393, learning_rate: 0.0000009723, weight_decay: 0.0000012566

epoch time: 1094.0217285156s, input_fired_time: 12.8414182663, output_fired_time: 6.3706607819, propagate_forward_time: 421.5832824707, propagate_backward_time: 643.3693237305, weight_update_time: 0.3895564377, other_time: 9.8571166992
epoch time: 87.0644226074s, input_fired_time: 1.0157086849, output_fired_time: 0.0000000000, propagate_forward_time: 84.4206237793, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.6280899048
validation[          , genome  5598] predictions:    6479/  10000 (64.79%), best:    6469/10000 (64.69%), error:     10181.82813, best error:     10166.44336 on epoch:    98, epoch:   99/100, mu: 0.9882811308, learning_rate: 0.0000009063, weight_decay: 0.0000011820

epoch time: 1091.1558837891s, input_fired_time: 12.9217824936, output_fired_time: 6.4785466194, propagate_forward_time: 422.6730651855, propagate_backward_time: 638.9594116211, weight_update_time: 0.3977816403, other_time: 10.1231079102
epoch time: 87.9642944336s, input_fired_time: 1.0467314720, output_fired_time: 0.0000000000, propagate_forward_time: 85.1642227173, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7533416748
validation[          , genome  5598] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10158.84180, best error:     10158.84180 on epoch:   100, epoch:  100/100, mu: 0.9883763194, learning_rate: 0.0000008448, weight_decay: 0.0000011119

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 438.9204406738s, input_fired_time: 5.1829128265, output_fired_time: 0.0000000000, propagate_forward_time: 424.7771911621, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 8.9603271484
best training[          , genome  5598] predictions:   35365/  50000 (70.73%), best:    6485/10000 (64.85%), error:     44470.33984, best error:     10158.84180 on epoch:   100, epoch:  101/100, mu: 0.9884662628, learning_rate: 0.0000007875, weight_decay: 0.0000010459
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 87.7727432251s, input_fired_time: 1.0337049961, output_fired_time: 0.0000000000, propagate_forward_time: 84.9685440063, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.7704925537
   testing[          , genome  5598] predictions:    6485/  10000 (64.85%), best:    6485/10000 (64.85%), error:     10158.84180, best error:     10158.84180 on epoch:   100, epoch:  101/100, mu: 0.9884662628, learning_rate: 0.0000007875, weight_decay: 0.0000010459
backpropagation finished successfully!
12:34:20 (5836): called boinc_finish(0)

</stderr_txt>
]]>