Nameexact_genome_1542727803_183_6414_2
Workunit4101254
Created21 Nov 2018, 19:56:32 UTC
Sent21 Nov 2018, 19:56:42 UTC
Report deadline1 Feb 2019, 6:55:24 UTC
Received23 Nov 2018, 12:13:29 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID87483
Run time23 hours 26 min 50 sec
CPU time22 hours 16 min 10 sec
Validate stateValid
Credit17,864.36
Device peak FLOPS4.23 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size518.99 MB
Peak swap size538.73 MB
Peak disk usage9.51 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
orward_time: 40.1947555542, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6562538147
validation[          , genome  6414] predictions:    6009/  10000 (60.09%), best:    6046/10000 (60.46%), error:     12092.13867, best error:     12038.13281 on epoch:    45, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 555.6315917969s, input_fired_time: 5.1363649368, output_fired_time: 3.1012334824, propagate_forward_time: 203.1726379395, propagate_backward_time: 330.5666809082, weight_update_time: 0.3885818124, other_time: 13.6546630859
epoch time: 44.3719863892s, input_fired_time: 0.5934486389, output_fired_time: 0.0000000000, propagate_forward_time: 41.1447639465, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6337738037
validation[          , genome  6414] predictions:    6077/  10000 (60.77%), best:    6077/10000 (60.77%), error:     11779.84473, best error:     11779.84473 on epoch:    55, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 556.4187011719s, input_fired_time: 5.1420636177, output_fired_time: 3.1414337158, propagate_forward_time: 203.5606536865, propagate_backward_time: 330.6506652832, weight_update_time: 0.4139635563, other_time: 13.9238586426
epoch time: 43.8093795776s, input_fired_time: 0.5926430225, output_fired_time: 0.0000000000, propagate_forward_time: 40.5365371704, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6801986694
validation[          , genome  6414] predictions:    5965/  10000 (59.65%), best:    6077/10000 (60.77%), error:     12131.58008, best error:     11779.84473 on epoch:    55, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 557.1276855469s, input_fired_time: 5.1494078636, output_fired_time: 3.1411488056, propagate_forward_time: 204.2649230957, propagate_backward_time: 330.5930480957, weight_update_time: 0.4123617411, other_time: 13.9791259766
epoch time: 42.4600067139s, input_fired_time: 0.5894284844, output_fired_time: 0.0000000000, propagate_forward_time: 39.1476135254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7229652405
validation[          , genome  6414] predictions:    6054/  10000 (60.54%), best:    6077/10000 (60.77%), error:     12036.79199, best error:     11779.84473 on epoch:    55, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 560.1774902344s, input_fired_time: 5.1670293808, output_fired_time: 3.1410892010, propagate_forward_time: 207.0014343262, propagate_backward_time: 330.9883117676, weight_update_time: 0.4133447707, other_time: 13.8795776367
epoch time: 43.5149688721s, input_fired_time: 0.5938608050, output_fired_time: 0.0000000000, propagate_forward_time: 40.2510185242, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6700897217
validation[          , genome  6414] predictions:    5931/  10000 (59.31%), best:    6077/10000 (60.77%), error:     12345.07910, best error:     11779.84473 on epoch:    55, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 998.8513793945s, input_fired_time: 7.6174950600, output_fired_time: 5.9535250664, propagate_forward_time: 393.1456604004, propagate_backward_time: 578.4531250000, weight_update_time: 0.7706750631, other_time: 13.6815185547
epoch time: 99.4598999023s, input_fired_time: 1.1490347385, output_fired_time: 0.0000000000, propagate_forward_time: 95.4598464966, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8510208130
validation[          , genome  6414] predictions:    6041/  10000 (60.41%), best:    6077/10000 (60.77%), error:     12068.33887, best error:     11779.84473 on epoch:    55, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 1139.9687500000s, input_fired_time: 8.4232044220, output_fired_time: 7.0768623352, propagate_forward_time: 451.1665954590, propagate_backward_time: 658.2164916992, weight_update_time: 0.9642380476, other_time: 15.0855102539
epoch time: 92.6426544189s, input_fired_time: 1.0798889399, output_fired_time: 0.0000000000, propagate_forward_time: 89.0262603760, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5365066528
validation[          , genome  6414] predictions:    6042/  10000 (60.42%), best:    6077/10000 (60.77%), error:     11965.03223, best error:     11779.84473 on epoch:    55, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.968555
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000547078
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 2.18831e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 61
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 55
read number_validation_images: 10000
read best_validation_predictions: 6077
read best_validation_error: 11779.8
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '903196373'
read generator: 903196373
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 552.299377441406s, input_fired_time: 5.00065517425537, output_fired_time: 3.10409450531006, propagate_forward_time: 201.281204223633, propagate_backward_time: 332.827728271484, weight_update_time: 0.384581357240677, other_time: 10.0857238769531
epoch time: 42.6841583251953s, input_fired_time: 0.581959307193756, output_fired_time: 0, propagate_forward_time: 40.0918159484863, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.01038360595703
validation[          , genome  6414] predictions:    5984/  10000 (59.84%), best:    6077/10000 (60.77%), error:     12116.28320, best error:     11779.84473 on epoch:    55, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 550.6403808594s, input_fired_time: 5.0023880005, output_fired_time: 3.1276443005, propagate_forward_time: 200.2255706787, propagate_backward_time: 331.7652587891, weight_update_time: 0.4029545486, other_time: 10.5195312500
epoch time: 42.8738632202s, input_fired_time: 0.5973896384, output_fired_time: 0.0000000000, propagate_forward_time: 40.2377853394, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0386886597
validation[          , genome  6414] predictions:    6068/  10000 (60.68%), best:    6077/10000 (60.77%), error:     11885.12598, best error:     11779.84473 on epoch:    55, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 879.1758422852s, input_fired_time: 6.6710963249, output_fired_time: 5.2153944969, propagate_forward_time: 345.3998718262, propagate_backward_time: 510.1060791016, weight_update_time: 0.6732526422, other_time: 11.7834472656
epoch time: 90.8615646362s, input_fired_time: 1.0745692253, output_fired_time: 0.0000000000, propagate_forward_time: 87.6551513672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1318435669
validation[          , genome  6414] predictions:    6137/  10000 (61.37%), best:    6077/10000 (60.77%), error:     11827.30859, best error:     11779.84473 on epoch:    55, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 1042.4373779297s, input_fired_time: 7.4785580635, output_fired_time: 6.3505101204, propagate_forward_time: 411.3463439941, propagate_backward_time: 604.7226562500, weight_update_time: 0.7915684581, other_time: 12.5394287109
epoch time: 84.7899246216s, input_fired_time: 0.9312018752, output_fired_time: 0.0000000000, propagate_forward_time: 81.8046493530, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0540771484
validation[          , genome  6414] predictions:    6093/  10000 (60.93%), best:    6077/10000 (60.77%), error:     11811.41504, best error:     11779.84473 on epoch:    55, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 1041.2215576172s, input_fired_time: 7.4805374146, output_fired_time: 6.4105324745, propagate_forward_time: 410.2823791504, propagate_backward_time: 604.4899902344, weight_update_time: 0.7923586965, other_time: 12.5581054688
epoch time: 85.4491043091s, input_fired_time: 0.9485431910, output_fired_time: 0.0000000000, propagate_forward_time: 82.2918243408, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2087402344
validation[          , genome  6414] predictions:    6026/  10000 (60.26%), best:    6077/10000 (60.77%), error:     11854.10547, best error:     11779.84473 on epoch:    55, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 1088.2886962891s, input_fired_time: 7.8704204559, output_fired_time: 6.6769709587, propagate_forward_time: 432.5829467773, propagate_backward_time: 627.7739257813, weight_update_time: 0.8676097989, other_time: 13.3844604492
epoch time: 92.7656173706s, input_fired_time: 1.0349140167, output_fired_time: 0.0000000000, propagate_forward_time: 89.3146209717, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4160842896
validation[          , genome  6414] predictions:    6089/  10000 (60.89%), best:    6077/10000 (60.77%), error:     12042.89844, best error:     11779.84473 on epoch:    55, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 1137.4819335938s, input_fired_time: 8.3745784760, output_fired_time: 7.0481028557, propagate_forward_time: 457.8752441406, propagate_backward_time: 649.4663085938, weight_update_time: 0.9538780451, other_time: 14.7176513672
epoch time: 94.6727218628s, input_fired_time: 1.0494836569, output_fired_time: 0.0000000000, propagate_forward_time: 91.0347137451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.5885238647
validation[          , genome  6414] predictions:    6141/  10000 (61.41%), best:    6141/10000 (61.41%), error:     11772.51563, best error:     11772.51563 on epoch:    67, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 1092.9998779297s, input_fired_time: 7.8777103424, output_fired_time: 6.6948227882, propagate_forward_time: 438.9888000488, propagate_backward_time: 626.2967529297, weight_update_time: 0.8327822685, other_time: 13.1418457031
epoch time: 89.8444290161s, input_fired_time: 1.0021450520, output_fired_time: 0.0000000000, propagate_forward_time: 86.6861801147, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1561050415
validation[          , genome  6414] predictions:    6061/  10000 (60.61%), best:    6141/10000 (61.41%), error:     11839.74121, best error:     11772.51563 on epoch:    67, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.975773
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000362943
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 1.45177e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 69
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 67
read number_validation_images: 10000
read best_validation_predictions: 6141
read best_validation_error: 11772.5
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '325040578'
read generator: 325040578
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1048.42407226563s, input_fired_time: 7.54979944229126, output_fired_time: 6.11197566986084, propagate_forward_time: 424.515991210938, propagate_backward_time: 598.790649414063, weight_update_time: 0.733471512794495, other_time: 11.4556884765625
epoch time: 89.2808990478516s, input_fired_time: 0.93938022851944, output_fired_time: 0, propagate_forward_time: 86.3037719726563, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.03775024414063
validation[          , genome  6414] predictions:    5938/  10000 (59.38%), best:    6141/10000 (61.41%), error:     12311.28809, best error:     11772.51563 on epoch:    67, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 1042.0812988281s, input_fired_time: 7.4615764618, output_fired_time: 6.1207203865, propagate_forward_time: 422.9098205566, propagate_backward_time: 594.1179199219, weight_update_time: 0.7391432524, other_time: 11.4713134766
epoch time: 85.1121063232s, input_fired_time: 0.9232785106, output_fired_time: 0.0000000000, propagate_forward_time: 82.1624298096, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0263977051
validation[          , genome  6414] predictions:    6018/  10000 (60.18%), best:    6141/10000 (61.41%), error:     11929.99023, best error:     11772.51563 on epoch:    67, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 1052.9250488281s, input_fired_time: 7.5374598503, output_fired_time: 6.1495180130, propagate_forward_time: 425.2008056641, propagate_backward_time: 601.8128662109, weight_update_time: 0.7839334607, other_time: 12.2243652344
epoch time: 94.5193786621s, input_fired_time: 1.0850697756, output_fired_time: 0.0000000000, propagate_forward_time: 90.9933395386, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4409713745
validation[          , genome  6414] predictions:    6039/  10000 (60.39%), best:    6141/10000 (61.41%), error:     12192.29980, best error:     11772.51563 on epoch:    67, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 1032.1767578125s, input_fired_time: 7.4681677818, output_fired_time: 6.1281027794, propagate_forward_time: 412.8966979980, propagate_backward_time: 593.9844970703, weight_update_time: 0.7399486899, other_time: 11.6993408203
epoch time: 85.2163162231s, input_fired_time: 0.9210812449, output_fired_time: 0.0000000000, propagate_forward_time: 82.2054595947, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0897750854
validation[          , genome  6414] predictions:    6110/  10000 (61.10%), best:    6110/10000 (61.10%), error:     11755.49414, best error:     11755.49414 on epoch:    72, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 1042.2197265625s, input_fired_time: 7.5048999786, output_fired_time: 6.1368618011, propagate_forward_time: 417.6972351074, propagate_backward_time: 598.9831542969, weight_update_time: 0.7442529798, other_time: 11.8975830078
epoch time: 87.0969085693s, input_fired_time: 0.9658022523, output_fired_time: 0.0000000000, propagate_forward_time: 83.9305038452, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2005996704
validation[          , genome  6414] predictions:    6114/  10000 (61.14%), best:    6114/10000 (61.14%), error:     11717.15039, best error:     11717.15039 on epoch:    73, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 1047.7755126953s, input_fired_time: 7.7016324997, output_fired_time: 6.2452883720, propagate_forward_time: 419.7421569824, propagate_backward_time: 601.8823852539, weight_update_time: 0.7616111636, other_time: 12.2040405273
epoch time: 89.3035964966s, input_fired_time: 0.9849411845, output_fired_time: 0.0000000000, propagate_forward_time: 86.0497360229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2689208984
validation[          , genome  6414] predictions:    6147/  10000 (61.47%), best:    6114/10000 (61.14%), error:     11774.44824, best error:     11717.15039 on epoch:    73, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 1045.9954833984s, input_fired_time: 7.5164942741, output_fired_time: 6.1937365532, propagate_forward_time: 424.1962890625, propagate_backward_time: 595.8796997070, weight_update_time: 0.7423400879, other_time: 12.2092895508
epoch time: 88.9329528809s, input_fired_time: 0.9458478093, output_fired_time: 0.0000000000, propagate_forward_time: 85.8013610840, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1857452393
validation[          , genome  6414] predictions:    6123/  10000 (61.23%), best:    6114/10000 (61.14%), error:     11799.79004, best error:     11717.15039 on epoch:    73, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.980065
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000253457
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 1.01383e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 76
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 73
read number_validation_images: 10000
read best_validation_predictions: 6114
read best_validation_error: 11717.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '266931584'
read generator: 266931584
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1015.07189941406s, input_fired_time: 7.39478254318237, output_fired_time: 6.12333106994629, propagate_forward_time: 401.976898193359, propagate_backward_time: 587.447875976563, weight_update_time: 0.747655391693115, other_time: 12.1290283203125
epoch time: 80.3175048828125s, input_fired_time: 0.890129625797272, output_fired_time: 0, propagate_forward_time: 77.3891983032227, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.03817749023438
validation[          , genome  6414] predictions:    6091/  10000 (60.91%), best:    6114/10000 (61.14%), error:     11787.98340, best error:     11717.15039 on epoch:    73, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 888.9704589844s, input_fired_time: 6.6579508781, output_fired_time: 5.2483386993, propagate_forward_time: 346.4881286621, propagate_backward_time: 519.6321411133, weight_update_time: 0.6061117649, other_time: 10.9439086914
epoch time: 68.8234863281s, input_fired_time: 0.8016931415, output_fired_time: 0.0000000000, propagate_forward_time: 66.0838012695, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9379882813
validation[          , genome  6414] predictions:    6073/  10000 (60.73%), best:    6114/10000 (61.14%), error:     11902.30176, best error:     11717.15039 on epoch:    73, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 1042.5670166016s, input_fired_time: 7.6015629768, output_fired_time: 6.2275490761, propagate_forward_time: 413.9655456543, propagate_backward_time: 603.2288208008, weight_update_time: 0.7335998416, other_time: 11.5435180664
epoch time: 88.5923843384s, input_fired_time: 0.9881253839, output_fired_time: 0.0000000000, propagate_forward_time: 85.4378204346, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1664352417
validation[          , genome  6414] predictions:    6106/  10000 (61.06%), best:    6106/10000 (61.06%), error:     11703.81738, best error:     11703.81738 on epoch:    78, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 1055.5754394531s, input_fired_time: 7.5060162544, output_fired_time: 6.2775931358, propagate_forward_time: 419.8356018066, propagate_backward_time: 610.4758911133, weight_update_time: 0.7288828492, other_time: 11.4804077148
epoch time: 84.9259338379s, input_fired_time: 0.9316350818, output_fired_time: 0.0000000000, propagate_forward_time: 81.9676818848, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0266189575
validation[          , genome  6414] predictions:    6156/  10000 (61.56%), best:    6106/10000 (61.06%), error:     11801.23340, best error:     11703.81738 on epoch:    78, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 1027.4451904297s, input_fired_time: 7.4255962372, output_fired_time: 6.0691246986, propagate_forward_time: 407.1575012207, propagate_backward_time: 595.8355102539, weight_update_time: 0.7007703185, other_time: 10.9574584961
epoch time: 84.3944931030s, input_fired_time: 0.9285954237, output_fired_time: 0.0000000000, propagate_forward_time: 81.4260864258, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0398101807
validation[          , genome  6414] predictions:    6183/  10000 (61.83%), best:    6183/10000 (61.83%), error:     11690.59570, best error:     11690.59570 on epoch:    80, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 1031.7806396484s, input_fired_time: 7.3696732521, output_fired_time: 6.1497516632, propagate_forward_time: 409.7307434082, propagate_backward_time: 597.5257568359, weight_update_time: 0.6930961609, other_time: 11.0047607422
epoch time: 84.5937652588s, input_fired_time: 0.9181671739, output_fired_time: 0.0000000000, propagate_forward_time: 81.7510910034, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9245071411
validation[          , genome  6414] predictions:    6114/  10000 (61.14%), best:    6183/10000 (61.83%), error:     11921.50781, best error:     11690.59570 on epoch:    80, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 1026.1649169922s, input_fired_time: 7.3353052139, output_fired_time: 6.1158008575, propagate_forward_time: 406.9021911621, propagate_backward_time: 594.8910522461, weight_update_time: 0.6953637004, other_time: 10.9205932617
epoch time: 84.2611923218s, input_fired_time: 0.9328561425, output_fired_time: 0.0000000000, propagate_forward_time: 81.3686981201, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9596405029
validation[          , genome  6414] predictions:    6095/  10000 (60.95%), best:    6183/10000 (61.83%), error:     11849.91113, best error:     11690.59570 on epoch:    80, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.983062
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000176998
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 7.07993e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 83
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 80
read number_validation_images: 10000
read best_validation_predictions: 6183
read best_validation_error: 11690.6
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '796448690'
read generator: 796448690
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1131.9482421875s, input_fired_time: 8.48940181732178, output_fired_time: 6.8589768409729, propagate_forward_time: 445.359222412109, propagate_backward_time: 658.4921875, weight_update_time: 0.852699398994446, other_time: 12.7484130859375
epoch time: 84.4804611206055s, input_fired_time: 0.966488659381866, output_fired_time: 0, propagate_forward_time: 81.470947265625, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.04302215576172
validation[          , genome  6414] predictions:    6178/  10000 (61.78%), best:    6178/10000 (61.78%), error:     11662.91504, best error:     11662.91504 on epoch:    83, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 1066.4644775391s, input_fired_time: 7.8144345284, output_fired_time: 6.4705195427, propagate_forward_time: 419.4047241211, propagate_backward_time: 620.7444458008, weight_update_time: 0.7743398547, other_time: 12.0303955078
epoch time: 88.6644744873s, input_fired_time: 1.0653944016, output_fired_time: 0.0000000000, propagate_forward_time: 85.4140396118, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1850433350
validation[          , genome  6414] predictions:    6216/  10000 (62.16%), best:    6216/10000 (62.16%), error:     11597.18457, best error:     11597.18457 on epoch:    84, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 1059.2292480469s, input_fired_time: 7.8545284271, output_fired_time: 6.4150605202, propagate_forward_time: 416.0076293945, propagate_backward_time: 616.9684448242, weight_update_time: 0.8105885386, other_time: 11.9836425781
epoch time: 83.0034866333s, input_fired_time: 0.9301581383, output_fired_time: 0.0000000000, propagate_forward_time: 80.1006164551, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.9727096558
validation[          , genome  6414] predictions:    6152/  10000 (61.52%), best:    6216/10000 (62.16%), error:     11637.32910, best error:     11597.18457 on epoch:    84, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 1102.0531005859s, input_fired_time: 8.2149200439, output_fired_time: 6.7383546829, propagate_forward_time: 434.9583129883, propagate_backward_time: 640.0459594727, weight_update_time: 0.8097658753, other_time: 12.0954589844
epoch time: 87.8502349854s, input_fired_time: 1.0308647156, output_fired_time: 0.0000000000, propagate_forward_time: 84.6954574585, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1239089966
validation[          , genome  6414] predictions:    6180/  10000 (61.80%), best:    6216/10000 (62.16%), error:     11614.37793, best error:     11597.18457 on epoch:    84, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 1073.0179443359s, input_fired_time: 8.0510129929, output_fired_time: 6.4556484222, propagate_forward_time: 423.6043395996, propagate_backward_time: 623.1101074219, weight_update_time: 0.8036319017, other_time: 11.7967529297
epoch time: 86.6057662964s, input_fired_time: 1.0116764307, output_fired_time: 0.0000000000, propagate_forward_time: 83.4509277344, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1431655884
validation[          , genome  6414] predictions:    6178/  10000 (61.78%), best:    6178/10000 (61.78%), error:     11596.40234, best error:     11596.40234 on epoch:    87, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 1045.7254638672s, input_fired_time: 7.7570381165, output_fired_time: 6.3358483315, propagate_forward_time: 411.3560485840, propagate_backward_time: 609.0461425781, weight_update_time: 0.7623312473, other_time: 11.2303466797
epoch time: 86.4375076294s, input_fired_time: 1.0127345324, output_fired_time: 0.0000000000, propagate_forward_time: 83.2197570801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2050170898
validation[          , genome  6414] predictions:    6188/  10000 (61.88%), best:    6178/10000 (61.78%), error:     11608.28711, best error:     11596.40234 on epoch:    87, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 1101.1202392578s, input_fired_time: 8.0886030197, output_fired_time: 6.6970782280, propagate_forward_time: 433.0552673340, propagate_backward_time: 640.6064453125, weight_update_time: 0.8765441179, other_time: 12.6728515625
epoch time: 89.2246932983s, input_fired_time: 1.0282000303, output_fired_time: 0.0000000000, propagate_forward_time: 85.8220443726, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3744506836
validation[          , genome  6414] predictions:    6076/  10000 (60.76%), best:    6178/10000 (61.78%), error:     11783.38477, best error:     11596.40234 on epoch:    87, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.985155
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000123604
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 4.94418e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 90
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 87
read number_validation_images: 10000
read best_validation_predictions: 6178
read best_validation_error: 11596.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '393465010'
read generator: 393465010
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1215.41442871094s, input_fired_time: 9.12772369384766, output_fired_time: 7.50314140319824, propagate_forward_time: 478.324157714844, propagate_backward_time: 705.819519042969, weight_update_time: 0.995249927043915, other_time: 14.639892578125
epoch time: 111.751922607422s, input_fired_time: 1.42313539981842, output_fired_time: 0, propagate_forward_time: 107.497596740723, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.83119201660156
validation[          , genome  6414] predictions:    6148/  10000 (61.48%), best:    6178/10000 (61.78%), error:     11614.10547, best error:     11596.40234 on epoch:    87, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 1124.0195312500s, input_fired_time: 8.4310150146, output_fired_time: 6.8685083389, propagate_forward_time: 442.2763366699, propagate_backward_time: 653.2752685547, weight_update_time: 0.8541094065, other_time: 13.1683349609
epoch time: 95.0682754517s, input_fired_time: 1.1054379940, output_fired_time: 0.0000000000, propagate_forward_time: 91.4731979370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4896392822
validation[          , genome  6414] predictions:    6204/  10000 (62.04%), best:    6204/10000 (62.04%), error:     11557.70996, best error:     11557.70996 on epoch:    91, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 1156.5965576172s, input_fired_time: 8.6175403595, output_fired_time: 7.0649724007, propagate_forward_time: 453.3906250000, propagate_backward_time: 673.9353027344, weight_update_time: 0.8536589146, other_time: 13.5881347656
epoch time: 91.5692214966s, input_fired_time: 1.0711338520, output_fired_time: 0.0000000000, propagate_forward_time: 88.1324996948, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3655853271
validation[          , genome  6414] predictions:    6157/  10000 (61.57%), best:    6204/10000 (62.04%), error:     11613.81445, best error:     11557.70996 on epoch:    91, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 1220.4572753906s, input_fired_time: 9.4754400253, output_fired_time: 7.5822577477, propagate_forward_time: 479.7377929688, propagate_backward_time: 708.4892578125, weight_update_time: 0.9618151784, other_time: 15.1724853516
epoch time: 98.3819732666s, input_fired_time: 1.1214362383, output_fired_time: 0.0000000000, propagate_forward_time: 94.6304855347, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6300506592
validation[          , genome  6414] predictions:    6127/  10000 (61.27%), best:    6204/10000 (62.04%), error:     11636.11230, best error:     11557.70996 on epoch:    91, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 1156.2482910156s, input_fired_time: 8.6633386612, output_fired_time: 7.0845241547, propagate_forward_time: 453.4056396484, propagate_backward_time: 672.7708129883, weight_update_time: 0.9431582093, other_time: 14.3240356445
epoch time: 101.8198547363s, input_fired_time: 1.1758477688, output_fired_time: 0.0000000000, propagate_forward_time: 97.6878204346, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9561843872
validation[          , genome  6414] predictions:    6156/  10000 (61.56%), best:    6204/10000 (62.04%), error:     11692.85254, best error:     11557.70996 on epoch:    91, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 1091.3973388672s, input_fired_time: 8.1855115891, output_fired_time: 6.4609131813, propagate_forward_time: 428.5263671875, propagate_backward_time: 635.0516967773, weight_update_time: 0.8368814588, other_time: 13.1727905273
epoch time: 85.9736480713s, input_fired_time: 1.0159648657, output_fired_time: 0.0000000000, propagate_forward_time: 82.8067703247, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1509094238
validation[          , genome  6414] predictions:    6204/  10000 (62.04%), best:    6204/10000 (62.04%), error:     11622.62793, best error:     11557.70996 on epoch:    91, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542727803_183_6414_2_r302139260_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.986438
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 9.08606e-05
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 3.63442e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 96
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 91
read number_validation_images: 10000
read best_validation_predictions: 6204
read best_validation_error: 11557.7
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6414
read normal distribution: '1 0 0'
generator_str: '2102487698'
read generator: 2102487698
read generated_by_map:
1 disable_edge 1
reading 71 nodes.
reading 286 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1066.75061035156s, input_fired_time: 7.87252569198608, output_fired_time: 6.49714946746826, propagate_forward_time: 420.3935546875, propagate_backward_time: 619.597534179688, weight_update_time: 0.770044088363647, other_time: 12.3897705078125
epoch time: 86.9936676025391s, input_fired_time: 1.00068700313568, output_fired_time: 0, propagate_forward_time: 83.8203582763672, propagate_backward_time: 0, weight_update_time: 0, other_time: 2.17262268066406
validation[          , genome  6414] predictions:    6200/  10000 (62.00%), best:    6204/10000 (62.04%), error:     11630.13965, best error:     11557.70996 on epoch:    91, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 1046.4440917969s, input_fired_time: 7.6777386665, output_fired_time: 6.2919754982, propagate_forward_time: 411.2891235352, propagate_backward_time: 609.1123046875, weight_update_time: 0.7730965018, other_time: 12.0729370117
epoch time: 86.6463928223s, input_fired_time: 0.9855231643, output_fired_time: 0.0000000000, propagate_forward_time: 83.5457916260, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1150817871
validation[          , genome  6414] predictions:    6182/  10000 (61.82%), best:    6204/10000 (62.04%), error:     11700.40820, best error:     11557.70996 on epoch:    91, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 1172.9654541016s, input_fired_time: 8.9381513596, output_fired_time: 7.1128201485, propagate_forward_time: 460.1244506836, propagate_backward_time: 682.6676635742, weight_update_time: 0.8508834839, other_time: 14.1224365234
epoch time: 92.6305923462s, input_fired_time: 1.0902566910, output_fired_time: 0.0000000000, propagate_forward_time: 89.2151336670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3252029419
validation[          , genome  6414] predictions:    6235/  10000 (62.35%), best:    6204/10000 (62.04%), error:     11629.19531, best error:     11557.70996 on epoch:    91, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 1089.1634521484s, input_fired_time: 8.2110710144, output_fired_time: 6.4995799065, propagate_forward_time: 429.3138427734, propagate_backward_time: 632.3036499023, weight_update_time: 0.8243089914, other_time: 12.8352661133
epoch time: 88.3416900635s, input_fired_time: 1.0279942751, output_fired_time: 0.0000000000, propagate_forward_time: 85.0282516479, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.2854461670
validation[          , genome  6414] predictions:    6143/  10000 (61.43%), best:    6204/10000 (62.04%), error:     11697.33008, best error:     11557.70996 on epoch:    91, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 1036.1525878906s, input_fired_time: 7.5942139626, output_fired_time: 6.2300806046, propagate_forward_time: 407.3430175781, propagate_backward_time: 602.8664550781, weight_update_time: 0.7568769455, other_time: 12.1187744141
epoch time: 84.6313171387s, input_fired_time: 0.9744228721, output_fired_time: 0.0000000000, propagate_forward_time: 81.4957504272, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1611404419
validation[          , genome  6414] predictions:    6168/  10000 (61.68%), best:    6204/10000 (62.04%), error:     11607.17480, best error:     11557.70996 on epoch:    91, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 416.3359069824s, input_fired_time: 4.7309694290, output_fired_time: 0.0000000000, propagate_forward_time: 400.8469543457, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 10.7579956055
best training[          , genome  6414] predictions:   40165/  50000 (80.33%), best:    6204/10000 (62.04%), error:     29966.90430, best error:     11557.70996 on epoch:    91, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 83.2054290771s, input_fired_time: 0.9344080687, output_fired_time: 0.0000000000, propagate_forward_time: 80.1429519653, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1280670166
   testing[          , genome  6414] predictions:    6204/  10000 (62.04%), best:    6204/10000 (62.04%), error:     11557.70996, best error:     11557.70996 on epoch:    91, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
15:12:35 (16488): called boinc_finish(0)

</stderr_txt>
]]>