Nameexact_genome_1542531604_184_6276_2
Workunit4097799
Created1 Mar 2019, 17:54:49 UTC
Sent1 Mar 2019, 17:58:19 UTC
Report deadline28 May 2019, 8:22:01 UTC
Received6 Mar 2019, 10:30:28 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID92383
Run time1 days 1 hours 40 min 15 sec
CPU time1 days 1 hours 40 min 15 sec
Validate stateValid
Credit21,899.95
Device peak FLOPS3.77 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size524.05 MB
Peak swap size543.77 MB
Peak disk usage9.27 MB

Stderr output

<core_client_version>7.6.33</core_client_version>
<![CDATA[
<stderr_txt>
redictions:    5652/  10000 (56.52%), best:    5752/10000 (57.52%), error:     12998.52832, best error:     12569.43750 on epoch:    33, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 2106.6950683594s, input_fired_time: 21.3434696198, output_fired_time: 12.6779584885, propagate_forward_time: 584.7244873047, propagate_backward_time: 1473.1184082031, weight_update_time: 0.8097735643, other_time: 14.8306884766
epoch time: 125.4078598022s, input_fired_time: 0.7861403227, output_fired_time: 0.0000000000, propagate_forward_time: 121.8544082642, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7673110962
validation[          , genome  6276] predictions:    5777/  10000 (57.77%), best:    5752/10000 (57.52%), error:     12690.41504, best error:     12569.43750 on epoch:    33, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 2096.4875488281s, input_fired_time: 16.7659893036, output_fired_time: 12.7370691299, propagate_forward_time: 581.2877807617, propagate_backward_time: 1469.5098876953, weight_update_time: 3.8895094395, other_time: 16.1868896484
epoch time: 123.0366744995s, input_fired_time: 3.8293318748, output_fired_time: 0.0000000000, propagate_forward_time: 117.9595184326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2478256226
validation[          , genome  6276] predictions:    5806/  10000 (58.06%), best:    5806/10000 (58.06%), error:     12505.18066, best error:     12505.18066 on epoch:    37, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 2097.1206054688s, input_fired_time: 15.2931146622, output_fired_time: 22.5819854736, propagate_forward_time: 572.9328613281, propagate_backward_time: 1470.1273193359, weight_update_time: 0.8143721223, other_time: 16.1851806641
epoch time: 120.4066772461s, input_fired_time: 2.3502898216, output_fired_time: 0.0000000000, propagate_forward_time: 115.3630523682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6933364868
validation[          , genome  6276] predictions:    5483/  10000 (54.83%), best:    5806/10000 (58.06%), error:     13717.42969, best error:     12505.18066 on epoch:    37, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 2101.5856933594s, input_fired_time: 14.5903520584, output_fired_time: 12.7835407257, propagate_forward_time: 596.1793212891, propagate_backward_time: 1458.6088867188, weight_update_time: 0.8064019084, other_time: 19.4237060547
epoch time: 122.4935760498s, input_fired_time: 2.2625236511, output_fired_time: 0.0000000000, propagate_forward_time: 118.9916992188, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2393493652
validation[          , genome  6276] predictions:    5678/  10000 (56.78%), best:    5806/10000 (58.06%), error:     12975.49219, best error:     12505.18066 on epoch:    37, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 2210.5983886719s, input_fired_time: 21.5673789978, output_fired_time: 5.5019078255, propagate_forward_time: 621.0972900391, propagate_backward_time: 1541.2308349609, weight_update_time: 2.3481531143, other_time: 21.2009277344
epoch time: 127.6234817505s, input_fired_time: 0.8083409071, output_fired_time: 0.0000000000, propagate_forward_time: 122.5480194092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2671203613
validation[          , genome  6276] predictions:    5831/  10000 (58.31%), best:    5806/10000 (58.06%), error:     12543.43750, best error:     12505.18066 on epoch:    37, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 2097.4223632813s, input_fired_time: 21.0041179657, output_fired_time: 9.7226610184, propagate_forward_time: 609.9248657227, propagate_backward_time: 1439.0772705078, weight_update_time: 0.8203963637, other_time: 17.6934814453
epoch time: 129.6383209229s, input_fired_time: 2.3037707806, output_fired_time: 0.0000000000, propagate_forward_time: 121.3665466309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9680023193
validation[          , genome  6276] predictions:    5803/  10000 (58.03%), best:    5803/10000 (58.03%), error:     12235.16797, best error:     12235.16797 on epoch:    41, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 2143.2180175781s, input_fired_time: 14.4859981537, output_fired_time: 14.2668685913, propagate_forward_time: 601.2642211914, propagate_backward_time: 1498.2420654297, weight_update_time: 0.8366045952, other_time: 14.9588623047
epoch time: 122.4299926758s, input_fired_time: 0.8057306409, output_fired_time: 0.0000000000, propagate_forward_time: 120.3584213257, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2658386230
validation[          , genome  6276] predictions:    5760/  10000 (57.60%), best:    5803/10000 (58.03%), error:     12466.68457, best error:     12235.16797 on epoch:    41, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 2140.6818847656s, input_fired_time: 18.7527084351, output_fired_time: 12.9016685486, propagate_forward_time: 618.4606933594, propagate_backward_time: 1475.7625732422, weight_update_time: 0.8212331533, other_time: 14.8043212891
epoch time: 125.2934799194s, input_fired_time: 0.8135715127, output_fired_time: 0.0000000000, propagate_forward_time: 121.7379379272, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7419738770
validation[          , genome  6276] predictions:    5858/  10000 (58.58%), best:    5803/10000 (58.03%), error:     12397.50586, best error:     12235.16797 on epoch:    41, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 2147.5996093750s, input_fired_time: 13.8306140900, output_fired_time: 9.7871742249, propagate_forward_time: 609.1328125000, propagate_backward_time: 1498.5129394531, weight_update_time: 0.8182010055, other_time: 16.3361816406
epoch time: 127.2719039917s, input_fired_time: 2.2515771389, output_fired_time: 0.0000000000, propagate_forward_time: 120.6672134399, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3531112671
validation[          , genome  6276] predictions:    5779/  10000 (57.79%), best:    5803/10000 (58.03%), error:     12969.21582, best error:     12235.16797 on epoch:    41, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 2167.8356933594s, input_fired_time: 22.9045772552, output_fired_time: 12.7891120911, propagate_forward_time: 588.5997314453, propagate_backward_time: 1527.1910400391, weight_update_time: 2.2784385681, other_time: 16.3513183594
epoch time: 125.0837478638s, input_fired_time: 2.2201707363, output_fired_time: 0.0000000000, propagate_forward_time: 120.0946807861, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7688980103
validation[          , genome  6276] predictions:    5862/  10000 (58.62%), best:    5803/10000 (58.03%), error:     12414.92188, best error:     12235.16797 on epoch:    41, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 2156.7583007813s, input_fired_time: 12.2698326111, output_fired_time: 11.3312473297, propagate_forward_time: 610.7907714844, propagate_backward_time: 1507.4379882813, weight_update_time: 2.3310503960, other_time: 14.9284667969
epoch time: 122.1108093262s, input_fired_time: 2.8280129433, output_fired_time: 0.0000000000, propagate_forward_time: 115.0036926270, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2791061401
validation[          , genome  6276] predictions:    5789/  10000 (57.89%), best:    5803/10000 (58.03%), error:     12702.47656, best error:     12235.16797 on epoch:    41, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 2153.2360839844s, input_fired_time: 22.9956817627, output_fired_time: 11.3943634033, propagate_forward_time: 604.9565429688, propagate_backward_time: 1492.9710693359, weight_update_time: 3.8336482048, other_time: 20.9185791016
epoch time: 126.8814849854s, input_fired_time: 2.3216869831, output_fired_time: 0.0000000000, propagate_forward_time: 123.2974624634, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2623367310
validation[          , genome  6276] predictions:    5815/  10000 (58.15%), best:    5803/10000 (58.03%), error:     12366.67090, best error:     12235.16797 on epoch:    41, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 2144.2590332031s, input_fired_time: 21.3900661469, output_fired_time: 15.9018249512, propagate_forward_time: 592.3983764648, propagate_backward_time: 1498.1965332031, weight_update_time: 2.3268914223, other_time: 16.3720703125
epoch time: 124.9557342529s, input_fired_time: 0.8146268129, output_fired_time: 0.0000000000, propagate_forward_time: 119.8717727661, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2693328857
validation[          , genome  6276] predictions:    5875/  10000 (58.75%), best:    5803/10000 (58.03%), error:     12487.68555, best error:     12235.16797 on epoch:    41, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276_2_r2131080866_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.950313
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00101243
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 4.04973e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 49
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 41
read number_validation_images: 10000
read best_validation_predictions: 5803
read best_validation_error: 12235.2
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6276
read normal distribution: '1 1.30090987682343 -0.848073363304138'
generator_str: '1605163265'
read generator: 1605163265
read generated_by_map:
1 split_edge 1
reading 76 nodes.
reading 260 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2261.61645507813s, input_fired_time: 23.3900947570801, output_fired_time: 11.7348480224609, propagate_forward_time: 628.055419921875, propagate_backward_time: 1580.00915527344, weight_update_time: 0.887096464633942, other_time: 18.4268798828125
epoch time: 118.118560791016s, input_fired_time: 0.807886004447937, output_fired_time: 0, propagate_forward_time: 111.390953063965, propagate_backward_time: 0, weight_update_time: 0, other_time: 5.91972351074219
validation[          , genome  6276] predictions:    5852/  10000 (58.52%), best:    5803/10000 (58.03%), error:     12564.50098, best error:     12235.16797 on epoch:    41, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 2107.7131347656s, input_fired_time: 16.3798713684, output_fired_time: 15.9322309494, propagate_forward_time: 585.7920532227, propagate_backward_time: 1477.8079833984, weight_update_time: 0.8107150793, other_time: 11.8011474609
epoch time: 123.2119216919s, input_fired_time: 3.7993175983, output_fired_time: 0.0000000000, propagate_forward_time: 118.1642532349, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2483520508
validation[          , genome  6276] predictions:    5947/  10000 (59.47%), best:    5947/10000 (59.47%), error:     11977.11523, best error:     11977.11523 on epoch:    50, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 2111.1738281250s, input_fired_time: 19.8376541138, output_fired_time: 11.2883529663, propagate_forward_time: 589.5506591797, propagate_backward_time: 1477.0865478516, weight_update_time: 0.8097651601, other_time: 13.4106445313
epoch time: 125.4926910400s, input_fired_time: 2.2585079670, output_fired_time: 0.0000000000, propagate_forward_time: 121.9986572266, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2355270386
validation[          , genome  6276] predictions:    5973/  10000 (59.73%), best:    5947/10000 (59.47%), error:     11999.10938, best error:     11977.11523 on epoch:    50, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 2097.3452148438s, input_fired_time: 9.3559570313, output_fired_time: 12.5994234085, propagate_forward_time: 575.5098876953, propagate_backward_time: 1482.0003662109, weight_update_time: 0.8111453652, other_time: 17.8796386719
epoch time: 123.6705245972s, input_fired_time: 5.1787505150, output_fired_time: 0.0000000000, propagate_forward_time: 115.8072204590, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6845550537
validation[          , genome  6276] predictions:    5989/  10000 (59.89%), best:    5989/10000 (59.89%), error:     11871.27246, best error:     11871.27246 on epoch:    52, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 2117.4147949219s, input_fired_time: 15.3793277740, output_fired_time: 12.8618698120, propagate_forward_time: 575.2020263672, propagate_backward_time: 1496.4310302734, weight_update_time: 2.2230780125, other_time: 17.5405273438
epoch time: 126.2365036011s, input_fired_time: 0.8088598847, output_fired_time: 0.0000000000, propagate_forward_time: 122.6501770020, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7774658203
validation[          , genome  6276] predictions:    5788/  10000 (57.88%), best:    5989/10000 (59.89%), error:     12671.46973, best error:     11871.27246 on epoch:    52, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 2111.9221191406s, input_fired_time: 16.8614501953, output_fired_time: 14.4899749756, propagate_forward_time: 587.2949829102, propagate_backward_time: 1477.0814208984, weight_update_time: 0.8086317778, other_time: 16.1942138672
epoch time: 125.7706909180s, input_fired_time: 0.8193138242, output_fired_time: 0.0000000000, propagate_forward_time: 120.7030792236, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2482986450
validation[          , genome  6276] predictions:    5995/  10000 (59.95%), best:    5995/10000 (59.95%), error:     11779.53906, best error:     11779.53906 on epoch:    54, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 2099.3356933594s, input_fired_time: 16.8723793030, output_fired_time: 17.2909011841, propagate_forward_time: 612.2052001953, propagate_backward_time: 1429.2763671875, weight_update_time: 2.2881958485, other_time: 23.6907958984
epoch time: 125.8443527222s, input_fired_time: 2.2764751911, output_fired_time: 0.0000000000, propagate_forward_time: 120.7941818237, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7736968994
validation[          , genome  6276] predictions:    6000/  10000 (60.00%), best:    5995/10000 (59.95%), error:     11822.46680, best error:     11779.53906 on epoch:    54, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 2103.5883789063s, input_fired_time: 16.8643608093, output_fired_time: 6.7900800705, propagate_forward_time: 567.5015258789, propagate_backward_time: 1496.3033447266, weight_update_time: 0.8105971813, other_time: 16.1292724609
epoch time: 123.6601181030s, input_fired_time: 3.7203898430, output_fired_time: 0.0000000000, propagate_forward_time: 115.6768493652, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2628784180
validation[          , genome  6276] predictions:    5948/  10000 (59.48%), best:    5995/10000 (59.95%), error:     12174.62598, best error:     11779.53906 on epoch:    54, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 2100.4863281250s, input_fired_time: 21.2319660187, output_fired_time: 8.2892360687, propagate_forward_time: 571.2962646484, propagate_backward_time: 1492.4730224609, weight_update_time: 0.8126701117, other_time: 7.1958007813
epoch time: 122.9966125488s, input_fired_time: 0.8154034019, output_fired_time: 0.0000000000, propagate_forward_time: 119.4063873291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7748184204
validation[          , genome  6276] predictions:    5955/  10000 (59.55%), best:    5995/10000 (59.95%), error:     12286.85742, best error:     11779.53906 on epoch:    54, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 2108.3950195313s, input_fired_time: 17.0158176422, output_fired_time: 14.1835708618, propagate_forward_time: 586.3582763672, propagate_backward_time: 1468.7144775391, weight_update_time: 0.8121451139, other_time: 22.1228027344
epoch time: 123.7161788940s, input_fired_time: 0.8206018209, output_fired_time: 0.0000000000, propagate_forward_time: 118.6453628540, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2502136230
validation[          , genome  6276] predictions:    5892/  10000 (58.92%), best:    5995/10000 (59.95%), error:     12338.46484, best error:     11779.53906 on epoch:    54, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 2098.8405761719s, input_fired_time: 20.1212081909, output_fired_time: 12.8031148911, propagate_forward_time: 592.6461791992, propagate_backward_time: 1452.6372070313, weight_update_time: 5.2639856339, other_time: 20.6328125000
epoch time: 120.9680404663s, input_fired_time: 3.7785785198, output_fired_time: 0.0000000000, propagate_forward_time: 115.9562988281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2331619263
validation[          , genome  6276] predictions:    5902/  10000 (59.02%), best:    5995/10000 (59.95%), error:     12123.24805, best error:     11779.53906 on epoch:    54, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 2232.5722656250s, input_fired_time: 20.2966976166, output_fired_time: 14.7358407974, propagate_forward_time: 648.5000610352, propagate_backward_time: 1532.3194580078, weight_update_time: 2.3507115841, other_time: 16.7203369141
epoch time: 122.0824127197s, input_fired_time: 2.2713024616, output_fired_time: 0.0000000000, propagate_forward_time: 118.5265808105, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2845306396
validation[          , genome  6276] predictions:    6037/  10000 (60.37%), best:    6037/10000 (60.37%), error:     11741.98730, best error:     11741.98730 on epoch:    60, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 2129.4443359375s, input_fired_time: 18.0926666260, output_fired_time: 16.0770454407, propagate_forward_time: 593.7461547852, propagate_backward_time: 1482.3558349609, weight_update_time: 3.8873357773, other_time: 19.1724853516
epoch time: 116.4778213501s, input_fired_time: 0.8241497278, output_fired_time: 0.0000000000, propagate_forward_time: 114.4243469238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2293243408
validation[          , genome  6276] predictions:    5885/  10000 (58.85%), best:    6037/10000 (60.37%), error:     12185.91602, best error:     11741.98730 on epoch:    60, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 2102.5686035156s, input_fired_time: 21.4130935669, output_fired_time: 17.2107372284, propagate_forward_time: 599.5380859375, propagate_backward_time: 1452.8387451172, weight_update_time: 0.8126661181, other_time: 11.5679931641
epoch time: 125.1818847656s, input_fired_time: 0.8300809860, output_fired_time: 0.0000000000, propagate_forward_time: 123.1060256958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2457809448
validation[          , genome  6276] predictions:    6042/  10000 (60.42%), best:    6037/10000 (60.37%), error:     11915.84961, best error:     11741.98730 on epoch:    60, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 2117.8662109375s, input_fired_time: 24.4491786957, output_fired_time: 15.8922300339, propagate_forward_time: 611.7241821289, propagate_backward_time: 1451.0722656250, weight_update_time: 0.8184598684, other_time: 14.7282714844
epoch time: 122.8017730713s, input_fired_time: 5.4725975990, output_fired_time: 0.0000000000, propagate_forward_time: 114.5897750854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7394027710
validation[          , genome  6276] predictions:    5892/  10000 (58.92%), best:    6037/10000 (60.37%), error:     12371.70313, best error:     11741.98730 on epoch:    60, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 2110.1169433594s, input_fired_time: 18.5470619202, output_fired_time: 8.2911958694, propagate_forward_time: 603.8520507813, propagate_backward_time: 1461.8981933594, weight_update_time: 3.7910079956, other_time: 17.5283203125
epoch time: 125.7936096191s, input_fired_time: 0.8326858878, output_fired_time: 0.0000000000, propagate_forward_time: 123.7144241333, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2464981079
validation[          , genome  6276] predictions:    6047/  10000 (60.47%), best:    6037/10000 (60.37%), error:     11832.66504, best error:     11741.98730 on epoch:    60, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 2101.4416503906s, input_fired_time: 21.5031509399, output_fired_time: 16.0200443268, propagate_forward_time: 566.5494995117, propagate_backward_time: 1478.0817871094, weight_update_time: 2.2857303619, other_time: 19.2871093750
epoch time: 123.2380065918s, input_fired_time: 2.3776769638, output_fired_time: 0.0000000000, propagate_forward_time: 119.6306076050, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2297210693
validation[          , genome  6276] predictions:    6035/  10000 (60.35%), best:    6037/10000 (60.37%), error:     11952.61816, best error:     11741.98730 on epoch:    60, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 2097.9694824219s, input_fired_time: 11.0171623230, output_fired_time: 17.4106178284, propagate_forward_time: 600.7302856445, propagate_backward_time: 1447.5318603516, weight_update_time: 2.2587709427, other_time: 21.2796630859
epoch time: 121.1060028076s, input_fired_time: 0.8255367875, output_fired_time: 0.0000000000, propagate_forward_time: 119.0518951416, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2285690308
validation[          , genome  6276] predictions:    5989/  10000 (59.89%), best:    6037/10000 (60.37%), error:     12008.06836, best error:     11741.98730 on epoch:    60, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 2102.7500000000s, input_fired_time: 18.1500415802, output_fired_time: 11.4263219833, propagate_forward_time: 584.8218994141, propagate_backward_time: 1472.2662353516, weight_update_time: 0.8103284240, other_time: 16.0854492188
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276_2_r2131080866_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.974236
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000402153
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 1.60861e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 67
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 60
read number_validation_images: 10000
read best_validation_predictions: 6037
read best_validation_error: 11742
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6276
read normal distribution: '1 1.30090987682343 -0.848073363304138'
generator_str: '2062787606'
read generator: 2062787606
read generated_by_map:
1 split_edge 1
reading 76 nodes.
reading 260 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2211.39184570313s, input_fired_time: 12.6660022735596, output_fired_time: 16.0381736755371, propagate_forward_time: 606.276611328125, propagate_backward_time: 1552.24780273438, weight_update_time: 0.835154116153717, other_time: 24.163330078125
epoch time: 130.039886474609s, input_fired_time: 5.34431791305542, output_fired_time: 0, propagate_forward_time: 123.379524230957, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.31604766845703
validation[          , genome  6276] predictions:    5915/  10000 (59.15%), best:    6037/10000 (60.37%), error:     12140.86621, best error:     11741.98730 on epoch:    60, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 2194.3239746094s, input_fired_time: 15.7601041794, output_fired_time: 15.3276433945, propagate_forward_time: 621.1793212891, propagate_backward_time: 1524.0595703125, weight_update_time: 5.3634757996, other_time: 17.9974365234
epoch time: 129.1098937988s, input_fired_time: 2.3913164139, output_fired_time: 0.0000000000, propagate_forward_time: 123.9151306152, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8034439087
validation[          , genome  6276] predictions:    6051/  10000 (60.51%), best:    6037/10000 (60.37%), error:     12034.64941, best error:     11741.98730 on epoch:    60, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 2186.7434082031s, input_fired_time: 26.3174476624, output_fired_time: 13.0316495895, propagate_forward_time: 609.6004028320, propagate_backward_time: 1519.9333496094, weight_update_time: 2.3620593548, other_time: 17.8605957031
epoch time: 129.1955413818s, input_fired_time: 2.3614768982, output_fired_time: 0.0000000000, propagate_forward_time: 122.5353088379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2987518311
validation[          , genome  6276] predictions:    6000/  10000 (60.00%), best:    6037/10000 (60.37%), error:     11925.21484, best error:     11741.98730 on epoch:    60, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 2179.8779296875s, input_fired_time: 21.7739887238, output_fired_time: 8.4978294373, propagate_forward_time: 607.7216186523, propagate_backward_time: 1517.9534912109, weight_update_time: 0.8379551768, other_time: 23.9310302734
epoch time: 125.3470764160s, input_fired_time: 0.8700022697, output_fired_time: 0.0000000000, propagate_forward_time: 121.7078475952, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7692260742
validation[          , genome  6276] predictions:    6063/  10000 (60.63%), best:    6037/10000 (60.37%), error:     11759.29688, best error:     11741.98730 on epoch:    60, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 2178.2756347656s, input_fired_time: 18.9895439148, output_fired_time: 8.4898395538, propagate_forward_time: 612.4733276367, propagate_backward_time: 1527.8341064453, weight_update_time: 0.8444975019, other_time: 10.4888916016
epoch time: 127.6607208252s, input_fired_time: 2.3714840412, output_fired_time: 0.0000000000, propagate_forward_time: 124.0268554688, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2623825073
validation[          , genome  6276] predictions:    6065/  10000 (60.65%), best:    6037/10000 (60.37%), error:     11766.85645, best error:     11741.98730 on epoch:    60, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 2128.3261718750s, input_fired_time: 18.5812320709, output_fired_time: 14.4516916275, propagate_forward_time: 575.6959228516, propagate_backward_time: 1505.2313232422, weight_update_time: 0.8167265058, other_time: 14.3659667969
epoch time: 123.8007965088s, input_fired_time: 0.8405650854, output_fired_time: 0.0000000000, propagate_forward_time: 120.1356887817, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8245391846
validation[          , genome  6276] predictions:    6089/  10000 (60.89%), best:    6089/10000 (60.89%), error:     11638.79785, best error:     11638.79785 on epoch:    72, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 2101.2028808594s, input_fired_time: 20.0427742004, output_fired_time: 15.7980136871, propagate_forward_time: 582.8587646484, propagate_backward_time: 1469.4069824219, weight_update_time: 2.2759039402, other_time: 13.0963134766
epoch time: 118.1920776367s, input_fired_time: 2.3844690323, output_fired_time: 0.0000000000, propagate_forward_time: 111.5674362183, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2401733398
validation[          , genome  6276] predictions:    6050/  10000 (60.50%), best:    6050/10000 (60.50%), error:     11574.57324, best error:     11574.57324 on epoch:    73, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 2108.2680664063s, input_fired_time: 15.4262380600, output_fired_time: 14.4970817566, propagate_forward_time: 613.1110229492, propagate_backward_time: 1450.5764160156, weight_update_time: 0.8135334849, other_time: 14.6572265625
epoch time: 123.1314697266s, input_fired_time: 2.3709001541, output_fired_time: 0.0000000000, propagate_forward_time: 117.9682083130, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7923583984
validation[          , genome  6276] predictions:    6110/  10000 (61.10%), best:    6110/10000 (61.10%), error:     11555.11133, best error:     11555.11133 on epoch:    74, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 2102.3261718750s, input_fired_time: 18.5547752380, output_fired_time: 9.7504558563, propagate_forward_time: 582.7019653320, propagate_backward_time: 1476.5518798828, weight_update_time: 3.8314855099, other_time: 14.7672119141
epoch time: 123.4515075684s, input_fired_time: 2.2667362690, output_fired_time: 0.0000000000, propagate_forward_time: 118.4509963989, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7337722778
validation[          , genome  6276] predictions:    6047/  10000 (60.47%), best:    6110/10000 (61.10%), error:     11673.77344, best error:     11555.11133 on epoch:    74, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 2115.8569335938s, input_fired_time: 18.6408348083, output_fired_time: 9.8217325211, propagate_forward_time: 593.7985229492, propagate_backward_time: 1475.8796386719, weight_update_time: 2.2853701115, other_time: 17.7160644531
epoch time: 125.8212738037s, input_fired_time: 2.3159377575, output_fired_time: 0.0000000000, propagate_forward_time: 120.7793960571, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7259368896
validation[          , genome  6276] predictions:    6044/  10000 (60.44%), best:    6110/10000 (61.10%), error:     11786.74219, best error:     11555.11133 on epoch:    74, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 2113.1928710938s, input_fired_time: 17.2192935944, output_fired_time: 9.8473176956, propagate_forward_time: 589.4395751953, propagate_backward_time: 1482.0554199219, weight_update_time: 2.2858526707, other_time: 14.6312255859
epoch time: 120.8620376587s, input_fired_time: 2.2787125111, output_fired_time: 0.0000000000, propagate_forward_time: 117.3445816040, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2387466431
validation[          , genome  6276] predictions:    6002/  10000 (60.02%), best:    6110/10000 (61.10%), error:     11921.90625, best error:     11555.11133 on epoch:    74, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 2118.1223144531s, input_fired_time: 12.4656543732, output_fired_time: 8.3917179108, propagate_forward_time: 601.5355834961, propagate_backward_time: 1473.6270751953, weight_update_time: 3.6979591846, other_time: 22.1024169922
epoch time: 120.5095596313s, input_fired_time: 2.3089849949, output_fired_time: 0.0000000000, propagate_forward_time: 115.3856582642, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8149185181
validation[          , genome  6276] predictions:    6037/  10000 (60.37%), best:    6110/10000 (61.10%), error:     11793.10645, best error:     11555.11133 on epoch:    74, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 2127.6784667969s, input_fired_time: 24.5359382629, output_fired_time: 11.4520292282, propagate_forward_time: 579.3463745117, propagate_backward_time: 1500.5926513672, weight_update_time: 0.8269789815, other_time: 11.7513427734
epoch time: 121.1583404541s, input_fired_time: 0.8632647991, output_fired_time: 0.0000000000, propagate_forward_time: 119.0479278564, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2471466064
validation[          , genome  6276] predictions:    6068/  10000 (60.68%), best:    6110/10000 (61.10%), error:     11674.38281, best error:     11555.11133 on epoch:    74, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 2093.2370605469s, input_fired_time: 19.6788597107, output_fired_time: 14.0970163345, propagate_forward_time: 586.6807861328, propagate_backward_time: 1459.4587402344, weight_update_time: 0.8119489551, other_time: 13.3216552734
epoch time: 118.0905456543s, input_fired_time: 0.8455977440, output_fired_time: 0.0000000000, propagate_forward_time: 111.6865844727, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5583648682
validation[          , genome  6276] predictions:    6033/  10000 (60.33%), best:    6110/10000 (61.10%), error:     11811.79004, best error:     11555.11133 on epoch:    74, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 2090.1726074219s, input_fired_time: 16.9460144043, output_fired_time: 17.2389698029, propagate_forward_time: 587.7357788086, propagate_backward_time: 1441.6013183594, weight_update_time: 0.8111802340, other_time: 26.6503906250
epoch time: 121.3217620850s, input_fired_time: 3.7810823917, output_fired_time: 0.0000000000, propagate_forward_time: 116.3095626831, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2311172485
validation[          , genome  6276] predictions:    6040/  10000 (60.40%), best:    6110/10000 (61.10%), error:     11775.17383, best error:     11555.11133 on epoch:    74, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 2096.0144042969s, input_fired_time: 27.3485450745, output_fired_time: 8.2717895508, propagate_forward_time: 565.3918457031, propagate_backward_time: 1475.8762207031, weight_update_time: 2.2599680424, other_time: 19.1259765625
epoch time: 122.7698745728s, input_fired_time: 2.3780078888, output_fired_time: 0.0000000000, propagate_forward_time: 116.1247253418, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2671432495
validation[          , genome  6276] predictions:    6078/  10000 (60.78%), best:    6110/10000 (61.10%), error:     11640.87109, best error:     11555.11133 on epoch:    74, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 2106.5283203125s, input_fired_time: 18.7000350952, output_fired_time: 13.3650770187, propagate_forward_time: 597.4619140625, propagate_backward_time: 1453.0491943359, weight_update_time: 3.9413032532, other_time: 23.9522705078
epoch time: 123.0126037598s, input_fired_time: 0.8467148542, output_fired_time: 0.0000000000, propagate_forward_time: 116.7080001831, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4578857422
validation[          , genome  6276] predictions:    6028/  10000 (60.28%), best:    6110/10000 (61.10%), error:     11789.81348, best error:     11555.11133 on epoch:    74, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 2124.3254394531s, input_fired_time: 23.1014385223, output_fired_time: 7.3751497269, propagate_forward_time: 597.6417236328, propagate_backward_time: 1479.8321533203, weight_update_time: 3.8501312733, other_time: 16.3750000000
epoch time: 123.4493942261s, input_fired_time: 2.3870971203, output_fired_time: 0.0000000000, propagate_forward_time: 118.3563842773, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7059097290
validation[          , genome  6276] predictions:    6102/  10000 (61.02%), best:    6110/10000 (61.10%), error:     11644.30957, best error:     11555.11133 on epoch:    74, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 2097.6411132813s, input_fired_time: 18.8562602997, output_fired_time: 11.3292636871, propagate_forward_time: 586.8873901367, propagate_backward_time: 1459.7524414063, weight_update_time: 0.8140585423, other_time: 20.8156738281
epoch time: 120.3623123169s, input_fired_time: 0.8484025002, output_fired_time: 0.0000000000, propagate_forward_time: 116.7261199951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7877883911
validation[          , genome  6276] predictions:    6136/  10000 (61.36%), best:    6136/10000 (61.36%), error:     11432.37012, best error:     11432.37012 on epoch:    85, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 2204.5053710938s, input_fired_time: 26.5900096893, output_fired_time: 6.8745722771, propagate_forward_time: 644.9439086914, propagate_backward_time: 1512.7253417969, weight_update_time: 0.8263433576, other_time: 13.3715820313
epoch time: 130.5039978027s, input_fired_time: 2.4533443451, output_fired_time: 0.0000000000, propagate_forward_time: 123.7609252930, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2897338867
validation[          , genome  6276] predictions:    6157/  10000 (61.57%), best:    6136/10000 (61.36%), error:     11529.61328, best error:     11432.37012 on epoch:    85, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 2141.0417480469s, input_fired_time: 21.5148315430, output_fired_time: 15.8325643539, propagate_forward_time: 599.3954467773, propagate_backward_time: 1484.9969482422, weight_update_time: 3.7636339664, other_time: 19.3018798828
epoch time: 123.0534973145s, input_fired_time: 0.8666796684, output_fired_time: 0.0000000000, propagate_forward_time: 117.8772048950, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3096160889
validation[          , genome  6276] predictions:    6118/  10000 (61.18%), best:    6136/10000 (61.36%), error:     11531.93848, best error:     11432.37012 on epoch:    85, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 2152.1425781250s, input_fired_time: 20.2993755341, output_fired_time: 9.8932161331, propagate_forward_time: 603.4463500977, propagate_backward_time: 1500.3530273438, weight_update_time: 0.8251763582, other_time: 18.1506347656
epoch time: 123.1080474854s, input_fired_time: 2.3670032024, output_fired_time: 0.0000000000, propagate_forward_time: 114.9588699341, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.7821731567
validation[          , genome  6276] predictions:    6134/  10000 (61.34%), best:    6136/10000 (61.36%), error:     11482.68945, best error:     11432.37012 on epoch:    85, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 2125.3339843750s, input_fired_time: 18.9975433350, output_fired_time: 16.8286876678, propagate_forward_time: 593.1069335938, propagate_backward_time: 1470.9610595703, weight_update_time: 6.8250951767, other_time: 25.4398193359
epoch time: 125.7486114502s, input_fired_time: 0.8582293987, output_fired_time: 0.0000000000, propagate_forward_time: 122.1392135620, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7511672974
validation[          , genome  6276] predictions:    6153/  10000 (61.53%), best:    6136/10000 (61.36%), error:     11462.34082, best error:     11432.37012 on epoch:    85, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 2131.8774414063s, input_fired_time: 24.5739707947, output_fired_time: 11.5073862076, propagate_forward_time: 602.9943847656, propagate_backward_time: 1476.4483642578, weight_update_time: 2.3353037834, other_time: 16.3533935547
epoch time: 125.5809783936s, input_fired_time: 0.8610019088, output_fired_time: 0.0000000000, propagate_forward_time: 121.9504241943, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7695541382
validation[          , genome  6276] predictions:    6125/  10000 (61.25%), best:    6136/10000 (61.36%), error:     11564.91309, best error:     11432.37012 on epoch:    85, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 2136.6076660156s, input_fired_time: 21.5683097839, output_fired_time: 11.0191316605, propagate_forward_time: 599.8703002930, propagate_backward_time: 1484.7043457031, weight_update_time: 0.8212533593, other_time: 19.4455566406
epoch time: 125.9430770874s, input_fired_time: 0.8615992665, output_fired_time: 0.0000000000, propagate_forward_time: 120.8147583008, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.2667160034
validation[          , genome  6276] predictions:    6132/  10000 (61.32%), best:    6136/10000 (61.36%), error:     11512.08887, best error:     11432.37012 on epoch:    85, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 2132.7189941406s, input_fired_time: 15.7757310867, output_fired_time: 11.4248313904, propagate_forward_time: 611.3900756836, propagate_backward_time: 1477.5158691406, weight_update_time: 0.8178873062, other_time: 16.6125488281
epoch time: 120.6619644165s, input_fired_time: 0.8637244701, output_fired_time: 0.0000000000, propagate_forward_time: 113.8815917969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9166488647
validation[          , genome  6276] predictions:    6103/  10000 (61.03%), best:    6136/10000 (61.36%), error:     11644.64453, best error:     11432.37012 on epoch:    85, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542531604_184_6276_2_r2131080866_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.985846
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000105975
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 4.23901e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 93
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 85
read number_validation_images: 10000
read best_validation_predictions: 6136
read best_validation_error: 11432.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6276
read normal distribution: '1 1.30090987682343 -0.848073363304138'
generator_str: '1498160268'
read generator: 1498160268
read generated_by_map:
1 split_edge 1
reading 76 nodes.
reading 260 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 2202.11401367188s, input_fired_time: 21.8973178863525, output_fired_time: 19.1351222991943, propagate_forward_time: 610.611083984375, propagate_backward_time: 1535.65063476563, weight_update_time: 0.838279485702515, other_time: 14.820068359375
epoch time: 140.121917724609s, input_fired_time: 3.96764183044434, output_fired_time: 0, propagate_forward_time: 128.687957763672, propagate_backward_time: 0, weight_update_time: 0, other_time: 7.46632385253906
validation[          , genome  6276] predictions:    6113/  10000 (61.13%), best:    6136/10000 (61.36%), error:     11549.69727, best error:     11432.37012 on epoch:    85, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 2308.4975585938s, input_fired_time: 28.4182090759, output_fired_time: 16.1649227142, propagate_forward_time: 630.1605224609, propagate_backward_time: 1618.1630859375, weight_update_time: 0.8689688444, other_time: 15.5906982422
epoch time: 126.0097961426s, input_fired_time: 0.8719288111, output_fired_time: 0.0000000000, propagate_forward_time: 122.3108596802, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.8270111084
validation[          , genome  6276] predictions:    6116/  10000 (61.16%), best:    6136/10000 (61.36%), error:     11536.08887, best error:     11432.37012 on epoch:    85, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 2162.7353515625s, input_fired_time: 18.8386135101, output_fired_time: 14.5373525620, propagate_forward_time: 618.2429809570, propagate_backward_time: 1497.6771240234, weight_update_time: 0.8282300234, other_time: 13.4393310547
epoch time: 120.7129821777s, input_fired_time: 3.9298322201, output_fired_time: 0.0000000000, propagate_forward_time: 112.4677505493, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3153991699
validation[          , genome  6276] predictions:    6149/  10000 (61.49%), best:    6136/10000 (61.36%), error:     11570.43945, best error:     11432.37012 on epoch:    85, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 2116.0161132813s, input_fired_time: 21.8396415710, output_fired_time: 13.8084688187, propagate_forward_time: 607.9653930664, propagate_backward_time: 1450.0874023438, weight_update_time: 2.3436136246, other_time: 22.3154296875
epoch time: 120.3845443726s, input_fired_time: 0.8641571403, output_fired_time: 0.0000000000, propagate_forward_time: 118.2900466919, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2303390503
validation[          , genome  6276] predictions:    6169/  10000 (61.69%), best:    6169/10000 (61.69%), error:     11415.99512, best error:     11415.99512 on epoch:    96, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 2116.1950683594s, input_fired_time: 24.5684909821, output_fired_time: 6.7388648987, propagate_forward_time: 596.2580566406, propagate_backward_time: 1475.3955078125, weight_update_time: 2.2865161896, other_time: 13.2341308594
epoch time: 119.8415069580s, input_fired_time: 0.8615067601, output_fired_time: 0.0000000000, propagate_forward_time: 114.6336364746, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.3463668823
validation[          , genome  6276] predictions:    6159/  10000 (61.59%), best:    6169/10000 (61.69%), error:     11508.01953, best error:     11415.99512 on epoch:    96, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 2108.4326171875s, input_fired_time: 18.9058265686, output_fired_time: 11.2626848221, propagate_forward_time: 583.9490356445, propagate_backward_time: 1479.7973632813, weight_update_time: 0.8092319369, other_time: 14.5178222656
epoch time: 122.7779922485s, input_fired_time: 2.3361313343, output_fired_time: 0.0000000000, propagate_forward_time: 114.7967987061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.6450653076
validation[          , genome  6276] predictions:    6151/  10000 (61.51%), best:    6169/10000 (61.69%), error:     11485.38672, best error:     11415.99512 on epoch:    96, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 2087.9604492188s, input_fired_time: 15.8657617569, output_fired_time: 9.9551143646, propagate_forward_time: 592.6636352539, propagate_backward_time: 1448.8453369141, weight_update_time: 0.8083441854, other_time: 20.6307373047
epoch time: 124.8827972412s, input_fired_time: 0.8535690308, output_fired_time: 0.0000000000, propagate_forward_time: 121.3343811035, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.6948471069
validation[          , genome  6276] predictions:    6166/  10000 (61.66%), best:    6169/10000 (61.69%), error:     11452.00977, best error:     11415.99512 on epoch:    96, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 2088.6640625000s, input_fired_time: 20.0742187500, output_fired_time: 11.1889219284, propagate_forward_time: 578.3942871094, propagate_backward_time: 1462.8344726563, weight_update_time: 3.8136858940, other_time: 16.1721191406
epoch time: 122.1881332397s, input_fired_time: 6.9056682587, output_fired_time: 0.0000000000, propagate_forward_time: 112.5003356934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.7821273804
validation[          , genome  6276] predictions:    6190/  10000 (61.90%), best:    6169/10000 (61.69%), error:     11445.25684, best error:     11415.99512 on epoch:    96, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 618.3290405273s, input_fired_time: 10.1398038864, output_fired_time: 0.0000000000, propagate_forward_time: 595.9016113281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.2875976563
best training[          , genome  6276] predictions:   40110/  50000 (80.22%), best:    6169/10000 (61.69%), error:     30656.26758, best error:     11415.99512 on epoch:    96, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 123.1313018799s, input_fired_time: 0.8685335517, output_fired_time: 0.0000000000, propagate_forward_time: 120.9716110229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2911605835
   testing[          , genome  6276] predictions:    6169/  10000 (61.69%), best:    6169/10000 (61.69%), error:     11415.99512, best error:     11415.99512 on epoch:    96, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
12:29:58 (6636): called boinc_finish(0)

</stderr_txt>
]]>