Nameexact_genome_1542277804_189_5971_2
Workunit4090622
Created1 Feb 2019, 20:36:51 UTC
Sent1 Feb 2019, 20:37:05 UTC
Report deadline21 Apr 2019, 2:32:54 UTC
Received3 Feb 2019, 2:11:22 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID10687
Run time21 hours 42 min 8 sec
CPU time21 hours 40 min 39 sec
Validate stateValid
Credit19,561.78
Device peak FLOPS4.74 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size520.68 MB
Peak swap size518.98 MB
Peak disk usage10.77 MB

Stderr output

<core_client_version>7.4.42</core_client_version>
<![CDATA[
<stderr_txt>
 34/100, mu: 0.8916946650, learning_rate: 0.0000414106, weight_decay: 0.0000887385

epoch time: 808.9621582031s, input_fired_time: 11.8529405594, output_fired_time: 4.2073979378, propagate_forward_time: 298.3703308105, propagate_backward_time: 485.5067138672, weight_update_time: 0.3562977612, other_time: 9.0247497559
epoch time: 62.1483116150s, input_fired_time: 0.7050946951, output_fired_time: 0.0000000000, propagate_forward_time: 60.0169410706, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4262771606
validation[          , genome  5971] predictions:    6125/  10000 (61.25%), best:    6125/10000 (61.25%), error:     11351.87012, best error:     11351.87012 on epoch:    35, epoch:   35/100, mu: 0.8961471319, learning_rate: 0.0000384834, weight_decay: 0.0000841919

epoch time: 806.5740966797s, input_fired_time: 11.8279275894, output_fired_time: 4.1648683548, propagate_forward_time: 297.9835510254, propagate_backward_time: 483.6712646484, weight_update_time: 0.3515843153, other_time: 8.9264831543
epoch time: 62.5744400024s, input_fired_time: 0.7105749249, output_fired_time: 0.0000000000, propagate_forward_time: 60.4506111145, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4132537842
validation[          , genome  5971] predictions:    6000/  10000 (60.00%), best:    6125/10000 (61.25%), error:     11636.12988, best error:     11351.87012 on epoch:    35, epoch:   36/100, mu: 0.9003979564, learning_rate: 0.0000357630, weight_decay: 0.0000798782

epoch time: 807.9747314453s, input_fired_time: 11.8650236130, output_fired_time: 4.1837973595, propagate_forward_time: 298.8002929688, propagate_backward_time: 484.1459655762, weight_update_time: 0.3539564610, other_time: 8.9796447754
epoch time: 62.9654006958s, input_fired_time: 0.7184389830, output_fired_time: 0.0000000000, propagate_forward_time: 60.8076133728, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4393501282
validation[          , genome  5971] predictions:    6071/  10000 (60.71%), best:    6125/10000 (61.25%), error:     11415.30371, best error:     11351.87012 on epoch:    35, epoch:   37/100, mu: 0.9044562578, learning_rate: 0.0000332350, weight_decay: 0.0000757855

epoch time: 803.1351928711s, input_fired_time: 11.7598190308, output_fired_time: 4.1530003548, propagate_forward_time: 296.0722961426, propagate_backward_time: 482.2327880859, weight_update_time: 0.3486329615, other_time: 8.9172668457
epoch time: 61.3487472534s, input_fired_time: 0.6983966231, output_fired_time: 0.0000000000, propagate_forward_time: 59.2465858459, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4037666321
validation[          , genome  5971] predictions:    5988/  10000 (59.88%), best:    6125/10000 (61.25%), error:     11575.29980, best error:     11351.87012 on epoch:    35, epoch:   38/100, mu: 0.9083307385, learning_rate: 0.0000308856, weight_decay: 0.0000719025

epoch time: 802.6219482422s, input_fired_time: 11.8257389069, output_fired_time: 4.1904568672, propagate_forward_time: 296.7629089355, propagate_backward_time: 480.8497009277, weight_update_time: 0.3530205786, other_time: 8.9931640625
epoch time: 60.7541275024s, input_fired_time: 0.6892074943, output_fired_time: 0.0000000000, propagate_forward_time: 58.6790466309, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3858718872
validation[          , genome  5971] predictions:    6048/  10000 (60.48%), best:    6125/10000 (61.25%), error:     11466.17871, best error:     11351.87012 on epoch:    35, epoch:   39/100, mu: 0.9120297432, learning_rate: 0.0000287024, weight_decay: 0.0000682185

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971_2_r1948774422_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.514681
read mu: 0.915561
read mu_delta: 0.954708
read initial_learning_rate: 0.000500765
read learning_rate: 2.66734e-05
read learning_rate_delta: 0.929311
read initial_weight_decay: 0.000530569
read weight_decay: 6.47232e-05
read weight_decay_delta: 0.948763
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0973601
read input_dropout_probability: 0.00140173
read hidden_dropout_probability: 0.105021
read velocity_reset: 1882
read epoch: 40
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 35
read number_validation_images: 10000
read best_validation_predictions: 6125
read best_validation_error: 11351.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5971
read normal distribution: '1 0 0'
generator_str: '867143925'
read generator: 867143925
read generated_by_map:
1 enable_edge 1
reading 68 nodes.
reading 307 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971_2_r1948774422_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.514681
read mu: 0.915561
read mu_delta: 0.954708
read initial_learning_rate: 0.000500765
read learning_rate: 2.66734e-05
read learning_rate_delta: 0.929311
read initial_weight_decay: 0.000530569
read weight_decay: 6.47232e-05
read weight_decay_delta: 0.948763
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0973601
read input_dropout_probability: 0.00140173
read hidden_dropout_probability: 0.105021
read velocity_reset: 1882
read epoch: 40
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 35
read number_validation_images: 10000
read best_validation_predictions: 6125
read best_validation_error: 11351.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5971
read normal distribution: '1 0 0'
generator_str: '867143925'
read generator: 867143925
read generated_by_map:
1 enable_edge 1
reading 68 nodes.
reading 307 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 744.3486328125s, input_fired_time: 10.9266061782837, output_fired_time: 3.84619855880737, propagate_forward_time: 274.918518066406, propagate_backward_time: 446.11962890625, weight_update_time: 0.329128831624985, other_time: 8.53765869140625
epoch time: 58.5833587646484s, input_fired_time: 0.664302408695221, output_fired_time: 0, propagate_forward_time: 56.5532493591309, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.36580657958984
validation[          , genome  5971] predictions:    6133/  10000 (61.33%), best:    6125/10000 (61.25%), error:     11366.96680, best error:     11351.87012 on epoch:    35, epoch:   40/100, mu: 0.9155611992, learning_rate: 0.0000266734, weight_decay: 0.0000647232

epoch time: 777.6021728516s, input_fired_time: 11.4248085022, output_fired_time: 4.0288772583, propagate_forward_time: 287.4166870117, propagate_backward_time: 465.9834594727, weight_update_time: 0.3422820568, other_time: 8.7483520508
epoch time: 60.1318969727s, input_fired_time: 0.6878324151, output_fired_time: 0.0000000000, propagate_forward_time: 58.0674972534, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3765678406
validation[          , genome  5971] predictions:    6111/  10000 (61.11%), best:    6125/10000 (61.25%), error:     11457.46777, best error:     11351.87012 on epoch:    35, epoch:   41/100, mu: 0.9189327359, learning_rate: 0.0000247879, weight_decay: 0.0000614070

epoch time: 779.8288574219s, input_fired_time: 11.4424543381, output_fired_time: 4.0226674080, propagate_forward_time: 288.1494750977, propagate_backward_time: 467.4096069336, weight_update_time: 0.3432374299, other_time: 8.8046875000
epoch time: 59.8298454285s, input_fired_time: 0.6812161803, output_fired_time: 0.0000000000, propagate_forward_time: 57.7629547119, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3856735229
validation[          , genome  5971] predictions:    6130/  10000 (61.30%), best:    6130/10000 (61.30%), error:     11256.54785, best error:     11256.54785 on epoch:    42, epoch:   42/100, mu: 0.9221515656, learning_rate: 0.0000230357, weight_decay: 0.0000582607

epoch time: 780.3098144531s, input_fired_time: 11.4748964310, output_fired_time: 4.0542087555, propagate_forward_time: 288.1939697266, propagate_backward_time: 467.8442993164, weight_update_time: 0.3407046795, other_time: 8.7424316406
epoch time: 59.7016143799s, input_fired_time: 0.6831660867, output_fired_time: 0.0000000000, propagate_forward_time: 57.6291542053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3892936707
validation[          , genome  5971] predictions:    6088/  10000 (60.88%), best:    6130/10000 (61.30%), error:     11351.65332, best error:     11256.54785 on epoch:    42, epoch:   43/100, mu: 0.9252246022, learning_rate: 0.0000214073, weight_decay: 0.0000552757

epoch time: 782.5196533203s, input_fired_time: 11.5005607605, output_fired_time: 4.0529150963, propagate_forward_time: 289.1604003906, propagate_backward_time: 469.0179748535, weight_update_time: 0.3431328833, other_time: 8.7878112793
epoch time: 60.5835609436s, input_fired_time: 0.6908165216, output_fired_time: 0.0000000000, propagate_forward_time: 58.5056266785, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3871192932
validation[          , genome  5971] predictions:    6187/  10000 (61.87%), best:    6187/10000 (61.87%), error:     11168.57910, best error:     11168.57910 on epoch:    44, epoch:   44/100, mu: 0.9281584620, learning_rate: 0.0000198940, weight_decay: 0.0000524435

epoch time: 782.0657958984s, input_fired_time: 11.5402278900, output_fired_time: 4.0575113297, propagate_forward_time: 289.5472412109, propagate_backward_time: 468.1275939941, weight_update_time: 0.3440244794, other_time: 8.7932434082
epoch time: 60.1264457703s, input_fired_time: 0.6884651780, output_fired_time: 0.0000000000, propagate_forward_time: 58.0779991150, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3599815369
validation[          , genome  5971] predictions:    5989/  10000 (59.89%), best:    6187/10000 (61.87%), error:     11485.11719, best error:     11168.57910 on epoch:    44, epoch:   45/100, mu: 0.9309594035, learning_rate: 0.0000184878, weight_decay: 0.0000497565

epoch time: 787.0295410156s, input_fired_time: 11.5951585770, output_fired_time: 4.0741963387, propagate_forward_time: 290.9952697754, propagate_backward_time: 471.5186462402, weight_update_time: 0.3460301161, other_time: 8.8462524414
epoch time: 59.9954185486s, input_fired_time: 0.6884331703, output_fired_time: 0.0000000000, propagate_forward_time: 57.9263153076, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3806686401
validation[          , genome  5971] predictions:    6161/  10000 (61.61%), best:    6187/10000 (61.87%), error:     11191.09863, best error:     11168.57910 on epoch:    44, epoch:   46/100, mu: 0.9336335063, learning_rate: 0.0000171809, weight_decay: 0.0000472072

epoch time: 841.8524780273s, input_fired_time: 11.8029356003, output_fired_time: 4.1384105682, propagate_forward_time: 296.6913452148, propagate_backward_time: 520.1914062500, weight_update_time: 0.3494631052, other_time: 9.0283813477
epoch time: 59.8297424316s, input_fired_time: 0.6848116517, output_fired_time: 0.0000000000, propagate_forward_time: 57.7738990784, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3710327148
validation[          , genome  5971] predictions:    6064/  10000 (60.64%), best:    6187/10000 (61.87%), error:     11384.07715, best error:     11168.57910 on epoch:    44, epoch:   47/100, mu: 0.9361864924, learning_rate: 0.0000159664, weight_decay: 0.0000447884

epoch time: 776.6472167969s, input_fired_time: 11.4389677048, output_fired_time: 4.0059771538, propagate_forward_time: 287.1585083008, propagate_backward_time: 465.3495178223, weight_update_time: 0.3374884725, other_time: 8.6942443848
epoch time: 60.2082366943s, input_fired_time: 0.6913971305, output_fired_time: 0.0000000000, propagate_forward_time: 58.1293220520, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3875160217
validation[          , genome  5971] predictions:    6119/  10000 (61.19%), best:    6187/10000 (61.87%), error:     11180.28711, best error:     11168.57910 on epoch:    44, epoch:   48/100, mu: 0.9386238456, learning_rate: 0.0000148377, weight_decay: 0.0000424936

epoch time: 777.3409423828s, input_fired_time: 11.4297714233, output_fired_time: 4.0150270462, propagate_forward_time: 286.7684936523, propagate_backward_time: 466.3529357910, weight_update_time: 0.3410203457, other_time: 8.7747497559
epoch time: 59.8414878845s, input_fired_time: 0.6862782836, output_fired_time: 0.0000000000, propagate_forward_time: 57.8016624451, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3535461426
validation[          , genome  5971] predictions:    6040/  10000 (60.40%), best:    6187/10000 (61.87%), error:     11485.88184, best error:     11168.57910 on epoch:    44, epoch:   49/100, mu: 0.9409508109, learning_rate: 0.0000137889, weight_decay: 0.0000403164

epoch time: 778.4797973633s, input_fired_time: 11.4989194870, output_fired_time: 4.0118751526, propagate_forward_time: 287.9778747559, propagate_backward_time: 466.2606201172, weight_update_time: 0.3400985599, other_time: 8.7304992676
epoch time: 59.8552017212s, input_fired_time: 0.6869630814, output_fired_time: 0.0000000000, propagate_forward_time: 57.7910842896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3771553040
validation[          , genome  5971] predictions:    6202/  10000 (62.02%), best:    6202/10000 (62.02%), error:     11150.47266, best error:     11150.47266 on epoch:    50, epoch:   50/100, mu: 0.9431723952, learning_rate: 0.0000128142, weight_decay: 0.0000382507

epoch time: 785.0838012695s, input_fired_time: 11.5453405380, output_fired_time: 4.0776896477, propagate_forward_time: 289.6149291992, propagate_backward_time: 471.0836486816, weight_update_time: 0.3423288465, other_time: 8.7621765137
epoch time: 59.6908149719s, input_fired_time: 0.6861510277, output_fired_time: 0.0000000000, propagate_forward_time: 57.6151618958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3895034790
validation[          , genome  5971] predictions:    6219/  10000 (62.19%), best:    6219/10000 (62.19%), error:     11079.74805, best error:     11079.74805 on epoch:    51, epoch:   51/100, mu: 0.9452933073, learning_rate: 0.0000119083, weight_decay: 0.0000362909

epoch time: 776.9124145508s, input_fired_time: 11.4568290710, output_fired_time: 4.0189433098, propagate_forward_time: 287.3529357910, propagate_backward_time: 465.3902893066, weight_update_time: 0.3389421999, other_time: 8.6934204102
epoch time: 59.6037445068s, input_fired_time: 0.6816213727, output_fired_time: 0.0000000000, propagate_forward_time: 57.5588340759, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3632888794
validation[          , genome  5971] predictions:    6024/  10000 (60.24%), best:    6219/10000 (62.19%), error:     11455.99902, best error:     11079.74805 on epoch:    51, epoch:   52/100, mu: 0.9473181963, learning_rate: 0.0000110665, weight_decay: 0.0000344315

epoch time: 779.1827392578s, input_fired_time: 11.5006713867, output_fired_time: 4.0173192024, propagate_forward_time: 288.0339965820, propagate_backward_time: 466.8795166016, weight_update_time: 0.3415950537, other_time: 8.7512207031
epoch time: 59.7711982727s, input_fired_time: 0.6886262298, output_fired_time: 0.0000000000, propagate_forward_time: 57.7031555176, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3794174194
validation[          , genome  5971] predictions:    6099/  10000 (60.99%), best:    6219/10000 (62.19%), error:     11353.72070, best error:     11079.74805 on epoch:    51, epoch:   53/100, mu: 0.9492513537, learning_rate: 0.0000102843, weight_decay: 0.0000326673

epoch time: 780.3724365234s, input_fired_time: 11.5196084976, output_fired_time: 4.0527515411, propagate_forward_time: 288.4013061523, propagate_backward_time: 467.6717224121, weight_update_time: 0.3392012417, other_time: 8.7270812988
epoch time: 59.7823448181s, input_fired_time: 0.6905525923, output_fired_time: 0.0000000000, propagate_forward_time: 57.7136001587, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3781929016
validation[          , genome  5971] predictions:    6244/  10000 (62.44%), best:    6219/10000 (62.19%), error:     11110.73535, best error:     11079.74805 on epoch:    51, epoch:   54/100, mu: 0.9510969520, learning_rate: 0.0000095573, weight_decay: 0.0000309936

epoch time: 660.9088134766s, input_fired_time: 9.7152214050, output_fired_time: 3.4309887886, propagate_forward_time: 243.3162384033, propagate_backward_time: 396.6826477051, weight_update_time: 0.2914290428, other_time: 7.7637634277
epoch time: 47.6752471924s, input_fired_time: 0.5478069186, output_fired_time: 0.0000000000, propagate_forward_time: 45.9286956787, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1987457275
validation[          , genome  5971] predictions:    6120/  10000 (61.20%), best:    6219/10000 (62.19%), error:     11312.26758, best error:     11079.74805 on epoch:    51, epoch:   55/100, mu: 0.9528589845, learning_rate: 0.0000088817, weight_decay: 0.0000294056

epoch time: 623.2201538086s, input_fired_time: 9.1319456100, output_fired_time: 3.2058267593, propagate_forward_time: 228.8510284424, propagate_backward_time: 374.6047668457, weight_update_time: 0.2736013234, other_time: 7.4266052246
epoch time: 47.7755584717s, input_fired_time: 0.5488653183, output_fired_time: 0.0000000000, propagate_forward_time: 46.0409240723, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1857681274
validation[          , genome  5971] predictions:    6099/  10000 (60.99%), best:    6219/10000 (62.19%), error:     11268.97070, best error:     11079.74805 on epoch:    51, epoch:   56/100, mu: 0.9545412064, learning_rate: 0.0000082539, weight_decay: 0.0000278989

epoch time: 623.2665405273s, input_fired_time: 9.1761016846, output_fired_time: 3.2242946625, propagate_forward_time: 229.6948852539, propagate_backward_time: 373.7246093750, weight_update_time: 0.2752036452, other_time: 7.4466552734
epoch time: 48.0189514160s, input_fired_time: 0.5542496443, output_fired_time: 0.0000000000, propagate_forward_time: 46.2735443115, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1911582947
validation[          , genome  5971] predictions:    6264/  10000 (62.64%), best:    6264/10000 (62.64%), error:     11030.37305, best error:     11030.37305 on epoch:    57, epoch:   57/100, mu: 0.9561471939, learning_rate: 0.0000076704, weight_decay: 0.0000264695

epoch time: 626.0763549805s, input_fired_time: 9.2278594971, output_fired_time: 3.2524480820, propagate_forward_time: 230.8781738281, propagate_backward_time: 375.2389526367, weight_update_time: 0.2764645517, other_time: 7.4789428711
epoch time: 48.7324333191s, input_fired_time: 0.5614284277, output_fired_time: 0.0000000000, propagate_forward_time: 46.9610214233, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2099838257
validation[          , genome  5971] predictions:    6172/  10000 (61.72%), best:    6264/10000 (62.64%), error:     11111.66797, best error:     11030.37305 on epoch:    57, epoch:   58/100, mu: 0.9576804638, learning_rate: 0.0000071282, weight_decay: 0.0000251133

epoch time: 625.9464721680s, input_fired_time: 9.2177000046, output_fired_time: 3.2418518066, propagate_forward_time: 230.6384429932, propagate_backward_time: 375.4047851563, weight_update_time: 0.2744917572, other_time: 7.4437255859
epoch time: 48.0444259644s, input_fired_time: 0.5545884371, output_fired_time: 0.0000000000, propagate_forward_time: 46.2929840088, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1968536377
validation[          , genome  5971] predictions:    6254/  10000 (62.54%), best:    6254/10000 (62.54%), error:     11008.22559, best error:     11008.22559 on epoch:    59, epoch:   59/100, mu: 0.9591442943, learning_rate: 0.0000066243, weight_decay: 0.0000238266

epoch time: 624.4497680664s, input_fired_time: 9.2089223862, output_fired_time: 3.2292962074, propagate_forward_time: 230.4383544922, propagate_backward_time: 374.1662902832, weight_update_time: 0.2725354433, other_time: 7.4068908691
epoch time: 48.1302986145s, input_fired_time: 0.5536383986, output_fired_time: 0.0000000000, propagate_forward_time: 46.3912277222, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1854324341
validation[          , genome  5971] predictions:    6169/  10000 (61.69%), best:    6254/10000 (62.54%), error:     11147.54199, best error:     11008.22559 on epoch:    59, epoch:   60/100, mu: 0.9605418444, learning_rate: 0.0000061560, weight_decay: 0.0000226058

epoch time: 624.0718383789s, input_fired_time: 9.1960706711, output_fired_time: 3.2302205563, propagate_forward_time: 230.2150268555, propagate_backward_time: 373.9860839844, weight_update_time: 0.2746978402, other_time: 7.4444580078
epoch time: 47.8551368713s, input_fired_time: 0.5528605580, output_fired_time: 0.0000000000, propagate_forward_time: 46.1120491028, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1902275085
validation[          , genome  5971] predictions:    6205/  10000 (62.05%), best:    6254/10000 (62.54%), error:     11021.86230, best error:     11008.22559 on epoch:    59, epoch:   61/100, mu: 0.9618760943, learning_rate: 0.0000057209, weight_decay: 0.0000214475

epoch time: 627.3209228516s, input_fired_time: 9.2287292480, output_fired_time: 3.2304463387, propagate_forward_time: 230.9276885986, propagate_backward_time: 376.4819335938, weight_update_time: 0.2755343318, other_time: 7.4520874023
epoch time: 48.2369918823s, input_fired_time: 0.5555998087, output_fired_time: 0.0000000000, propagate_forward_time: 46.4831542969, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1982383728
validation[          , genome  5971] predictions:    6270/  10000 (62.70%), best:    6254/10000 (62.54%), error:     11046.91016, best error:     11008.22559 on epoch:    59, epoch:   62/100, mu: 0.9631499052, learning_rate: 0.0000053165, weight_decay: 0.0000203486

epoch time: 631.2739257813s, input_fired_time: 9.3180694580, output_fired_time: 3.2947950363, propagate_forward_time: 233.1151580811, propagate_backward_time: 378.0979003906, weight_update_time: 0.2746377289, other_time: 7.4479980469
epoch time: 48.4420928955s, input_fired_time: 0.5589718223, output_fired_time: 0.0000000000, propagate_forward_time: 46.6695785522, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2135429382
validation[          , genome  5971] predictions:    6239/  10000 (62.39%), best:    6254/10000 (62.54%), error:     11011.52539, best error:     11008.22559 on epoch:    59, epoch:   63/100, mu: 0.9643660188, learning_rate: 0.0000049407, weight_decay: 0.0000193060

epoch time: 629.6223144531s, input_fired_time: 9.2569637299, output_fired_time: 3.2826302052, propagate_forward_time: 231.1487426758, propagate_backward_time: 378.4666442871, weight_update_time: 0.2758473754, other_time: 7.4673156738
epoch time: 48.7105674744s, input_fired_time: 0.5653114319, output_fired_time: 0.0000000000, propagate_forward_time: 46.9429473877, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2023086548
validation[          , genome  5971] predictions:    6282/  10000 (62.82%), best:    6282/10000 (62.82%), error:     10982.77148, best error:     10982.77148 on epoch:    64, epoch:   64/100, mu: 0.9655270576, learning_rate: 0.0000045914, weight_decay: 0.0000183169

epoch time: 631.5936889648s, input_fired_time: 9.3206615448, output_fired_time: 3.2733149529, propagate_forward_time: 232.9909973145, propagate_backward_time: 378.5050048828, weight_update_time: 0.2781835198, other_time: 7.5036926270
epoch time: 48.6418151855s, input_fired_time: 0.5598506331, output_fired_time: 0.0000000000, propagate_forward_time: 46.8703231812, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2116432190
validation[          , genome  5971] predictions:    6154/  10000 (61.54%), best:    6282/10000 (62.82%), error:     11146.43848, best error:     10982.77148 on epoch:    64, epoch:   65/100, mu: 0.9666355252, learning_rate: 0.0000042668, weight_decay: 0.0000173784

epoch time: 629.3571777344s, input_fired_time: 9.2632522583, output_fired_time: 3.2839701176, propagate_forward_time: 231.5153198242, propagate_backward_time: 377.8276367188, weight_update_time: 0.2752662003, other_time: 7.4669799805
epoch time: 48.1425552368s, input_fired_time: 0.5560684204, output_fired_time: 0.0000000000, propagate_forward_time: 46.3881301880, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1983566284
validation[          , genome  5971] predictions:    6280/  10000 (62.80%), best:    6280/10000 (62.80%), error:     10955.78613, best error:     10955.78613 on epoch:    66, epoch:   66/100, mu: 0.9676937461, learning_rate: 0.0000039652, weight_decay: 0.0000164880

epoch time: 628.8652954102s, input_fired_time: 9.2846441269, output_fired_time: 3.2671923637, propagate_forward_time: 231.9935150146, propagate_backward_time: 376.8359985352, weight_update_time: 0.2766880691, other_time: 7.4838867188
epoch time: 48.3248138428s, input_fired_time: 0.5592346787, output_fired_time: 0.0000000000, propagate_forward_time: 46.5648498535, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2007293701
validation[          , genome  5971] predictions:    6303/  10000 (63.03%), best:    6280/10000 (62.80%), error:     11082.41797, best error:     10955.78613 on epoch:    66, epoch:   67/100, mu: 0.9687040448, learning_rate: 0.0000036849, weight_decay: 0.0000156432

epoch time: 628.7055053711s, input_fired_time: 9.2548236847, output_fired_time: 3.2471961975, propagate_forward_time: 231.1489257813, propagate_backward_time: 377.5726013184, weight_update_time: 0.2768273950, other_time: 7.4819641113
epoch time: 48.3659324646s, input_fired_time: 0.5607549548, output_fired_time: 0.0000000000, propagate_forward_time: 46.5989189148, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2062568665
validation[          , genome  5971] predictions:    6271/  10000 (62.71%), best:    6271/10000 (62.71%), error:     10943.07813, best error:     10943.07813 on epoch:    68, epoch:   68/100, mu: 0.9696685672, learning_rate: 0.0000034244, weight_decay: 0.0000148417

epoch time: 630.3483276367s, input_fired_time: 9.2997674942, output_fired_time: 3.2669858932, propagate_forward_time: 232.0679626465, propagate_backward_time: 378.2544860840, weight_update_time: 0.2753814161, other_time: 7.4591674805
epoch time: 48.4302291870s, input_fired_time: 0.5621702075, output_fired_time: 0.0000000000, propagate_forward_time: 46.6639022827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2041549683
validation[          , genome  5971] predictions:    6287/  10000 (62.87%), best:    6271/10000 (62.71%), error:     10956.27734, best error:     10943.07813 on epoch:    68, epoch:   69/100, mu: 0.9705893993, learning_rate: 0.0000031824, weight_decay: 0.0000140812

epoch time: 627.2908935547s, input_fired_time: 9.2768697739, output_fired_time: 3.2574312687, propagate_forward_time: 231.5104217529, propagate_backward_time: 375.8062133789, weight_update_time: 0.2741888165, other_time: 7.4399414063
epoch time: 48.3311309814s, input_fired_time: 0.5570658445, output_fired_time: 0.0000000000, propagate_forward_time: 46.5776100159, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1964569092
validation[          , genome  5971] predictions:    6300/  10000 (63.00%), best:    6271/10000 (62.71%), error:     10977.47363, best error:     10943.07813 on epoch:    68, epoch:   70/100, mu: 0.9714685678, learning_rate: 0.0000029574, weight_decay: 0.0000133598

epoch time: 625.4155273438s, input_fired_time: 9.2221422195, output_fired_time: 3.2454199791, propagate_forward_time: 230.1316223145, propagate_backward_time: 375.3611755371, weight_update_time: 0.2751532793, other_time: 7.4551391602
epoch time: 48.2467155457s, input_fired_time: 0.5583605170, output_fired_time: 0.0000000000, propagate_forward_time: 46.4975738525, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1907806396
validation[          , genome  5971] predictions:    6199/  10000 (61.99%), best:    6271/10000 (62.71%), error:     11064.91309, best error:     10943.07813 on epoch:    68, epoch:   71/100, mu: 0.9723079205, learning_rate: 0.0000027484, weight_decay: 0.0000126753

epoch time: 623.3435668945s, input_fired_time: 9.2137470245, output_fired_time: 3.2217774391, propagate_forward_time: 230.0947875977, propagate_backward_time: 373.3816833496, weight_update_time: 0.2739252746, other_time: 7.4315490723
epoch time: 48.1510276794s, input_fired_time: 0.5591971278, output_fired_time: 0.0000000000, propagate_forward_time: 46.3921661377, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1996650696
validation[          , genome  5971] predictions:    6166/  10000 (61.66%), best:    6271/10000 (62.71%), error:     11128.77051, best error:     10943.07813 on epoch:    68, epoch:   72/100, mu: 0.9731092453, learning_rate: 0.0000025541, weight_decay: 0.0000120258

epoch time: 624.2424316406s, input_fired_time: 9.2061529160, output_fired_time: 3.2324793339, propagate_forward_time: 229.9211273193, propagate_backward_time: 374.4352111816, weight_update_time: 0.2748935521, other_time: 7.4474182129
epoch time: 48.0378875732s, input_fired_time: 0.5584760904, output_fired_time: 0.0000000000, propagate_forward_time: 46.2802276611, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1991844177
validation[          , genome  5971] predictions:    6260/  10000 (62.60%), best:    6271/10000 (62.71%), error:     10960.23047, best error:     10943.07813 on epoch:    68, epoch:   73/100, mu: 0.9738742709, learning_rate: 0.0000023735, weight_decay: 0.0000114097

epoch time: 623.8532104492s, input_fired_time: 9.2252378464, output_fired_time: 3.2070102692, propagate_forward_time: 230.4886016846, propagate_backward_time: 373.5124511719, weight_update_time: 0.2735379636, other_time: 7.4199218750
epoch time: 47.8636322021s, input_fired_time: 0.5552870035, output_fired_time: 0.0000000000, propagate_forward_time: 46.1188735962, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1894721985
validation[          , genome  5971] predictions:    6203/  10000 (62.03%), best:    6271/10000 (62.71%), error:     11028.91895, best error:     10943.07813 on epoch:    68, epoch:   74/100, mu: 0.9746046662, learning_rate: 0.0000022058, weight_decay: 0.0000108251

epoch time: 625.2581787109s, input_fired_time: 9.2348194122, output_fired_time: 3.2440309525, propagate_forward_time: 230.3351745605, propagate_backward_time: 374.9658508301, weight_update_time: 0.2764708698, other_time: 7.4783325195
epoch time: 47.9790878296s, input_fired_time: 0.5571246147, output_fired_time: 0.0000000000, propagate_forward_time: 46.2274360657, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1945266724
validation[          , genome  5971] predictions:    6260/  10000 (62.60%), best:    6260/10000 (62.60%), error:     10940.72754, best error:     10940.72754 on epoch:    75, epoch:   75/100, mu: 0.9753019810, learning_rate: 0.0000020498, weight_decay: 0.0000102704

epoch time: 623.3300781250s, input_fired_time: 9.2066078186, output_fired_time: 3.2294762135, propagate_forward_time: 229.5343627930, propagate_backward_time: 373.9541015625, weight_update_time: 0.2723852396, other_time: 7.4055175781
epoch time: 48.4528694153s, input_fired_time: 0.5639916658, output_fired_time: 0.0000000000, propagate_forward_time: 46.6865501404, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2023277283
validation[          , genome  5971] predictions:    6275/  10000 (62.75%), best:    6260/10000 (62.60%), error:     11034.86133, best error:     10940.72754 on epoch:    75, epoch:   76/100, mu: 0.9759677052, learning_rate: 0.0000019049, weight_decay: 0.0000097442

epoch time: 623.0315551758s, input_fired_time: 9.1772937775, output_fired_time: 3.2160699368, propagate_forward_time: 228.9634552002, propagate_backward_time: 374.2247619629, weight_update_time: 0.2750330269, other_time: 7.4499816895
epoch time: 47.5269699097s, input_fired_time: 0.5521376133, output_fired_time: 0.0000000000, propagate_forward_time: 45.7842597961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1905708313
validation[          , genome  5971] predictions:    6333/  10000 (63.33%), best:    6333/10000 (63.33%), error:     10901.71777, best error:     10901.71777 on epoch:    77, epoch:   77/100, mu: 0.9766032696, learning_rate: 0.0000017703, weight_decay: 0.0000092450

epoch time: 619.5368041992s, input_fired_time: 9.1143131256, output_fired_time: 3.1890690327, propagate_forward_time: 227.3578338623, propagate_backward_time: 372.5006408691, weight_update_time: 0.2706053257, other_time: 7.3749084473
epoch time: 47.9428062439s, input_fired_time: 0.5580493808, output_fired_time: 0.0000000000, propagate_forward_time: 46.1883888245, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1963691711
validation[          , genome  5971] predictions:    6307/  10000 (63.07%), best:    6333/10000 (63.33%), error:     10912.89160, best error:     10901.71777 on epoch:    77, epoch:   78/100, mu: 0.9772100449, learning_rate: 0.0000016451, weight_decay: 0.0000087713

epoch time: 618.9813842773s, input_fired_time: 9.1301507950, output_fired_time: 3.1969990730, propagate_forward_time: 227.9336700439, propagate_backward_time: 371.3290710449, weight_update_time: 0.2716640234, other_time: 7.3915100098
epoch time: 47.7070465088s, input_fired_time: 0.5572257638, output_fired_time: 0.0000000000, propagate_forward_time: 45.9537658691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1960563660
validation[          , genome  5971] predictions:    6350/  10000 (63.50%), best:    6350/10000 (63.50%), error:     10864.42383, best error:     10864.42383 on epoch:    79, epoch:   79/100, mu: 0.9777893424, learning_rate: 0.0000015288, weight_decay: 0.0000083219

epoch time: 651.9114379883s, input_fired_time: 9.4875373840, output_fired_time: 3.3350071907, propagate_forward_time: 236.3086700439, propagate_backward_time: 395.1981506348, weight_update_time: 0.2799390852, other_time: 7.5820617676
epoch time: 239.2689208984s, input_fired_time: 41.1778182983, output_fired_time: 0.0000000000, propagate_forward_time: 196.7310180664, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3600769043
validation[          , genome  5971] predictions:    6206/  10000 (62.06%), best:    6350/10000 (63.50%), error:     11077.75391, best error:     10864.42383 on epoch:    79, epoch:   80/100, mu: 0.9783424139, learning_rate: 0.0000014208, weight_decay: 0.0000078955

epoch time: 650.3484497070s, input_fired_time: 9.5750818253, output_fired_time: 3.3481438160, propagate_forward_time: 238.8635101318, propagate_backward_time: 390.8744506836, weight_update_time: 0.2853294313, other_time: 7.6872558594
epoch time: 48.6174125671s, input_fired_time: 0.5661026239, output_fired_time: 0.0000000000, propagate_forward_time: 46.8527145386, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1985969543
validation[          , genome  5971] predictions:    6286/  10000 (62.86%), best:    6350/10000 (63.50%), error:     10919.77246, best error:     10864.42383 on epoch:    79, epoch:   81/100, mu: 0.9788703918, learning_rate: 0.0000013203, weight_decay: 0.0000074909

epoch time: 628.9146118164s, input_fired_time: 9.3087673187, output_fired_time: 3.2624728680, propagate_forward_time: 232.0598297119, propagate_backward_time: 376.8143005371, weight_update_time: 0.2757711112, other_time: 7.4692687988
epoch time: 48.4906921387s, input_fired_time: 0.5657932162, output_fired_time: 0.0000000000, propagate_forward_time: 46.7262687683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1986312866
validation[          , genome  5971] predictions:    6288/  10000 (62.88%), best:    6350/10000 (63.50%), error:     10926.97656, best error:     10864.42383 on epoch:    79, epoch:   82/100, mu: 0.9793744683, learning_rate: 0.0000012270, weight_decay: 0.0000071071

epoch time: 631.1964721680s, input_fired_time: 9.3095350266, output_fired_time: 3.2440168858, propagate_forward_time: 232.2981567383, propagate_backward_time: 378.9163208008, weight_update_time: 0.2733463645, other_time: 7.4284667969
epoch time: 48.9977607727s, input_fired_time: 0.5696091056, output_fired_time: 0.0000000000, propagate_forward_time: 47.2228202820, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2053298950
validation[          , genome  5971] predictions:    6280/  10000 (62.80%), best:    6350/10000 (63.50%), error:     10976.38379, best error:     10864.42383 on epoch:    79, epoch:   83/100, mu: 0.9798557162, learning_rate: 0.0000011403, weight_decay: 0.0000067430

epoch time: 630.3121948242s, input_fired_time: 9.3062715530, output_fired_time: 3.2452974319, propagate_forward_time: 231.9596405029, propagate_backward_time: 378.3047180176, weight_update_time: 0.2771826088, other_time: 7.4962463379
epoch time: 48.0622787476s, input_fired_time: 0.5600003600, output_fired_time: 0.0000000000, propagate_forward_time: 46.3082885742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1939888000
validation[          , genome  5971] predictions:    6312/  10000 (63.12%), best:    6350/10000 (63.50%), error:     10915.77832, best error:     10864.42383 on epoch:    79, epoch:   84/100, mu: 0.9803151488, learning_rate: 0.0000010597, weight_decay: 0.0000063975

epoch time: 624.0900268555s, input_fired_time: 9.2333612442, output_fired_time: 3.2233037949, propagate_forward_time: 230.0656585693, propagate_backward_time: 374.1466064453, weight_update_time: 0.2731625736, other_time: 7.4210815430
epoch time: 47.9836730957s, input_fired_time: 0.5594715476, output_fired_time: 0.0000000000, propagate_forward_time: 46.2374038696, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1867980957
validation[          , genome  5971] predictions:    6295/  10000 (62.95%), best:    6350/10000 (63.50%), error:     10981.69824, best error:     10864.42383 on epoch:    79, epoch:   85/100, mu: 0.9807537794, learning_rate: 0.0000009848, weight_decay: 0.0000060697

epoch time: 633.2482910156s, input_fired_time: 9.3408041000, output_fired_time: 3.2913787365, propagate_forward_time: 232.7754364014, propagate_backward_time: 380.3375244141, weight_update_time: 0.2774307728, other_time: 7.5031127930
epoch time: 48.6633377075s, input_fired_time: 0.5688594580, output_fired_time: 0.0000000000, propagate_forward_time: 46.8848724365, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2096061707
validation[          , genome  5971] predictions:    6349/  10000 (63.49%), best:    6349/10000 (63.49%), error:     10863.35352, best error:     10863.35352 on epoch:    86, epoch:   86/100, mu: 0.9811725616, learning_rate: 0.0000009151, weight_decay: 0.0000057587

epoch time: 627.7697143555s, input_fired_time: 9.2763490677, output_fired_time: 3.2516560555, propagate_forward_time: 230.9599304199, propagate_backward_time: 376.8784484863, weight_update_time: 0.2719527483, other_time: 7.4033203125
epoch time: 48.2508583069s, input_fired_time: 0.5625535250, output_fired_time: 0.0000000000, propagate_forward_time: 46.4865264893, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2017784119
validation[          , genome  5971] predictions:    6295/  10000 (62.95%), best:    6349/10000 (63.49%), error:     10920.99902, best error:     10863.35352 on epoch:    86, epoch:   87/100, mu: 0.9815723896, learning_rate: 0.0000008505, weight_decay: 0.0000054637

epoch time: 628.4333496094s, input_fired_time: 9.2980108261, output_fired_time: 3.2576730251, propagate_forward_time: 231.4452819824, propagate_backward_time: 376.9636535645, weight_update_time: 0.2760723829, other_time: 7.4686889648
epoch time: 48.3699493408s, input_fired_time: 0.5654140711, output_fired_time: 0.0000000000, propagate_forward_time: 46.6076622009, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1968727112
validation[          , genome  5971] predictions:    6375/  10000 (63.75%), best:    6375/10000 (63.75%), error:     10844.41699, best error:     10844.41699 on epoch:    88, epoch:   88/100, mu: 0.9819540977, learning_rate: 0.0000007903, weight_decay: 0.0000051837

epoch time: 629.2607421875s, input_fired_time: 9.3120851517, output_fired_time: 3.2734024525, propagate_forward_time: 231.8256530762, propagate_backward_time: 377.4068298340, weight_update_time: 0.2744205594, other_time: 7.4428100586
epoch time: 48.2950706482s, input_fired_time: 0.5632785559, output_fired_time: 0.0000000000, propagate_forward_time: 46.5407409668, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1910514832
validation[          , genome  5971] predictions:    6312/  10000 (63.12%), best:    6375/10000 (63.75%), error:     10871.03125, best error:     10844.41699 on epoch:    88, epoch:   89/100, mu: 0.9823185205, learning_rate: 0.0000007345, weight_decay: 0.0000049181

epoch time: 631.3391113281s, input_fired_time: 9.3460874557, output_fired_time: 3.2970209122, propagate_forward_time: 232.4365997314, propagate_backward_time: 378.7980041504, weight_update_time: 0.2753389180, other_time: 7.4614562988
epoch time: 48.8958892822s, input_fired_time: 0.5709215403, output_fired_time: 0.0000000000, propagate_forward_time: 47.1292266846, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1957397461
validation[          , genome  5971] predictions:    6363/  10000 (63.63%), best:    6375/10000 (63.75%), error:     10845.94336, best error:     10844.41699 on epoch:    88, epoch:   90/100, mu: 0.9826664329, learning_rate: 0.0000006826, weight_decay: 0.0000046661

epoch time: 632.0267333984s, input_fired_time: 9.3484315872, output_fired_time: 3.2897973061, propagate_forward_time: 232.8587188721, propagate_backward_time: 379.0483093262, weight_update_time: 0.2766271830, other_time: 7.4814758301
epoch time: 48.6834449768s, input_fired_time: 0.5681878328, output_fired_time: 0.0000000000, propagate_forward_time: 46.9134140015, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2018432617
validation[          , genome  5971] predictions:    6359/  10000 (63.59%), best:    6375/10000 (63.75%), error:     10872.75879, best error:     10844.41699 on epoch:    88, epoch:   91/100, mu: 0.9829986095, learning_rate: 0.0000006343, weight_decay: 0.0000044271

epoch time: 632.4838867188s, input_fired_time: 9.3754644394, output_fired_time: 3.2971029282, propagate_forward_time: 233.0871124268, propagate_backward_time: 379.2633056641, weight_update_time: 0.2754881084, other_time: 7.4608764648
epoch time: 48.9721145630s, input_fired_time: 0.5725491047, output_fired_time: 0.0000000000, propagate_forward_time: 47.1954421997, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2041244507
validation[          , genome  5971] predictions:    6361/  10000 (63.61%), best:    6375/10000 (63.75%), error:     10858.34277, best error:     10844.41699 on epoch:    88, epoch:   92/100, mu: 0.9833157063, learning_rate: 0.0000005895, weight_decay: 0.0000042002

epoch time: 633.0429077148s, input_fired_time: 9.3866319656, output_fired_time: 3.2727320194, propagate_forward_time: 233.8883819580, propagate_backward_time: 378.9818420410, weight_update_time: 0.2790777683, other_time: 7.5133361816
epoch time: 48.7979812622s, input_fired_time: 0.5692680478, output_fired_time: 0.0000000000, propagate_forward_time: 47.0261497498, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2025642395
validation[          , genome  5971] predictions:    6366/  10000 (63.66%), best:    6375/10000 (63.75%), error:     10868.24023, best error:     10844.41699 on epoch:    88, epoch:   93/100, mu: 0.9836184382, learning_rate: 0.0000005478, weight_decay: 0.0000039850

epoch time: 630.5026245117s, input_fired_time: 9.3125114441, output_fired_time: 3.2461206913, propagate_forward_time: 232.2064819336, propagate_backward_time: 378.2653503418, weight_update_time: 0.2765241861, other_time: 7.4721984863
epoch time: 48.4780769348s, input_fired_time: 0.5683675408, output_fired_time: 0.0000000000, propagate_forward_time: 46.7043151855, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2053947449
validation[          , genome  5971] predictions:    6363/  10000 (63.63%), best:    6363/10000 (63.63%), error:     10836.92285, best error:     10836.92285 on epoch:    94, epoch:   94/100, mu: 0.9839074612, learning_rate: 0.0000005091, weight_decay: 0.0000037809

epoch time: 631.6466064453s, input_fired_time: 9.3185977936, output_fired_time: 3.2788321972, propagate_forward_time: 232.2539520264, propagate_backward_time: 379.3424682617, weight_update_time: 0.2752227485, other_time: 7.4527587891
epoch time: 48.2650260925s, input_fired_time: 0.5655426979, output_fired_time: 0.0000000000, propagate_forward_time: 46.4946708679, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2048110962
validation[          , genome  5971] predictions:    6354/  10000 (63.54%), best:    6363/10000 (63.63%), error:     10844.24023, best error:     10836.92285 on epoch:    94, epoch:   95/100, mu: 0.9841834307, learning_rate: 0.0000004731, weight_decay: 0.0000035871

epoch time: 633.2651977539s, input_fired_time: 9.3479986191, output_fired_time: 3.2830853462, propagate_forward_time: 232.9209899902, propagate_backward_time: 380.2306823730, weight_update_time: 0.2769091725, other_time: 7.4824218750
epoch time: 48.5568199158s, input_fired_time: 0.5691300631, output_fired_time: 0.0000000000, propagate_forward_time: 46.7857437134, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2019462585
validation[          , genome  5971] predictions:    6361/  10000 (63.61%), best:    6363/10000 (63.63%), error:     10844.43652, best error:     10836.92285 on epoch:    94, epoch:   96/100, mu: 0.9844468832, learning_rate: 0.0000004396, weight_decay: 0.0000034033

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542277804_189_5971_2_r1948774422_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.514681
read mu: 0.984698
read mu_delta: 0.954708
read initial_learning_rate: 0.000500765
read learning_rate: 4.08566e-07
read learning_rate_delta: 0.929311
read initial_weight_decay: 0.000530569
read weight_decay: 3.22897e-06
read weight_decay_delta: 0.948763
read batch_size: 83
read epsilon: 1e-07
read alpha: 0.0973601
read input_dropout_probability: 0.00140173
read hidden_dropout_probability: 0.105021
read velocity_reset: 1882
read epoch: 97
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 94
read number_validation_images: 10000
read best_validation_predictions: 6363
read best_validation_error: 10836.9
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 5971
read normal distribution: '1 0 0'
generator_str: '601452203'
read generator: 601452203
read generated_by_map:
1 enable_edge 1
reading 68 nodes.
reading 307 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 607.930541992188s, input_fired_time: 9.00101470947266, output_fired_time: 3.1663601398468, propagate_forward_time: 224.000457763672, propagate_backward_time: 364.441192626953, weight_update_time: 0.264787405729294, other_time: 7.32147216796875
epoch time: 48.8954582214355s, input_fired_time: 0.574442088603973, output_fired_time: 0, propagate_forward_time: 47.1099510192871, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.21106338500977
validation[          , genome  5971] predictions:    6372/  10000 (63.72%), best:    6363/10000 (63.63%), error:     10862.70313, best error:     10836.92285 on epoch:    94, epoch:   97/100, mu: 0.9846984148, learning_rate: 0.0000004086, weight_decay: 0.0000032290

epoch time: 640.9627685547s, input_fired_time: 9.4934806824, output_fired_time: 3.3325612545, propagate_forward_time: 235.9862670898, propagate_backward_time: 384.5121459961, weight_update_time: 0.2842485905, other_time: 7.6383056641
epoch time: 48.7329978943s, input_fired_time: 0.5706892014, output_fired_time: 0.0000000000, propagate_forward_time: 46.9584121704, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2038955688
validation[          , genome  5971] predictions:    6339/  10000 (63.39%), best:    6363/10000 (63.63%), error:     10851.50195, best error:     10836.92285 on epoch:    94, epoch:   98/100, mu: 0.9849385619, learning_rate: 0.0000003797, weight_decay: 0.0000030635

epoch time: 632.4122924805s, input_fired_time: 9.3511400223, output_fired_time: 3.2821717262, propagate_forward_time: 233.0807800293, propagate_backward_time: 379.1777954102, weight_update_time: 0.2776882350, other_time: 7.5204162598
epoch time: 48.0815467834s, input_fired_time: 0.5627079606, output_fired_time: 0.0000000000, propagate_forward_time: 46.3235054016, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1953315735
validation[          , genome  5971] predictions:    6330/  10000 (63.30%), best:    6330/10000 (63.30%), error:     10830.73242, best error:     10830.73242 on epoch:    99, epoch:   99/100, mu: 0.9851678014, learning_rate: 0.0000003528, weight_decay: 0.0000029066

epoch time: 665.2352905273s, input_fired_time: 9.4134531021, output_fired_time: 3.2884707451, propagate_forward_time: 234.2475280762, propagate_backward_time: 410.7953796387, weight_update_time: 0.2751298845, other_time: 7.4904785156
epoch time: 48.5130500793s, input_fired_time: 0.5681319833, output_fired_time: 0.0000000000, propagate_forward_time: 46.7361793518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2087402344
validation[          , genome  5971] predictions:    6349/  10000 (63.49%), best:    6330/10000 (63.30%), error:     10865.52832, best error:     10830.73242 on epoch:    99, epoch:  100/100, mu: 0.9853866696, learning_rate: 0.0000003279, weight_decay: 0.0000027576

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 241.9783325195s, input_fired_time: 2.8352315426, output_fired_time: 0.0000000000, propagate_forward_time: 233.1190948486, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.0240020752
best training[          , genome  5971] predictions:   33470/  50000 (66.94%), best:    6330/10000 (63.30%), error:     50869.33594, best error:     10830.73242 on epoch:    99, epoch:  101/100, mu: 0.9855956435, learning_rate: 0.0000003047, weight_decay: 0.0000026164
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 48.5274353027s, input_fired_time: 0.5686749816, output_fired_time: 0.0000000000, propagate_forward_time: 46.7568244934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2019348145
   testing[          , genome  5971] predictions:    6330/  10000 (63.30%), best:    6330/10000 (63.30%), error:     10830.73242, best error:     10830.73242 on epoch:    99, epoch:  101/100, mu: 0.9855956435, learning_rate: 0.0000003047, weight_decay: 0.0000026164
backpropagation finished successfully!
13:06:30 (5720): called boinc_finish(0)

</stderr_txt>
]]>