Nameexact_genome_1542649503_172_6320_2
Workunit4098916
Created19 Nov 2018, 22:50:12 UTC
Sent19 Nov 2018, 22:50:44 UTC
Report deadline18 Dec 2018, 6:18:36 UTC
Received5 Dec 2018, 15:05:29 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID13956
Run time12 hours 31 min 46 sec
CPU time11 hours 57 min 15 sec
Validate stateInvalid
Credit0.00
Device peak FLOPS2.92 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size585.66 MB
Peak swap size605.53 MB
Peak disk usage13.13 MB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<stderr_txt>
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6320_2_r1808486732_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from input file: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6320.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.507075
read mu: 0.507075
read mu_delta: 0.940074
read initial_learning_rate: 0.00064945
read learning_rate: 0.00064945
read learning_rate_delta: 0.935531
read initial_weight_decay: 0.000215232
read weight_decay: 0.000215232
read weight_decay_delta: 0.943193
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0965148
read input_dropout_probability: 0.00124212
read hidden_dropout_probability: 0.0909201
read velocity_reset: 2095
read epoch: 0
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 0
read number_validation_images: 10000
read best_validation_predictions: 0
read best_validation_error: 1e+07
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6320
read normal distribution: '0 0.251959174871445 -0.795902907848358'
generator_str: '1715399466'
read generator: 1715399466
read generated_by_map:
1 split_edge 1
reading 114 nodes.
reading 452 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 0
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
generator min: 1, generator max: 2147483646
pre shuffle 1: 750864087
post shuffle 1: 944651036
epoch time: 1610.18676757813s, input_fired_time: 39.0368957519531, output_fired_time: 15.142894744873, propagate_forward_time: 476.414703369141, propagate_backward_time: 1044.51904296875, weight_update_time: 0.579149961471558, other_time: 35.0732421875
epoch time: 92.1583251953125s, input_fired_time: 2.16075754165649, output_fired_time: 0, propagate_forward_time: 85.191162109375, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.80640411376953
validation[          , genome  6320] predictions:    6142/  10000 (61.42%), best:    6142/10000 (61.42%), error:     10983.06445, best error:     10983.06445 on epoch:     0, epoch:    0/25, mu: 0.5070749521, learning_rate: 0.0006494503, weight_decay: 0.0002152319

epoch time: 1446.8769531250s, input_fired_time: 33.9233512878, output_fired_time: 11.5622911453, propagate_forward_time: 425.7867431641, propagate_backward_time: 947.7957153320, weight_update_time: 0.4762418866, other_time: 27.8088989258
epoch time: 92.9966354370s, input_fired_time: 2.2399461269, output_fired_time: 0.0000000000, propagate_forward_time: 85.8478088379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9088821411
validation[          , genome  6320] predictions:    6199/  10000 (61.99%), best:    6142/10000 (61.42%), error:     10995.76855, best error:     10983.06445 on epoch:     0, epoch:    1/25, mu: 0.5360145569, learning_rate: 0.0006075811, weight_decay: 0.0002030052

epoch time: 1500.1223144531s, input_fired_time: 35.5786857605, output_fired_time: 12.8627319336, propagate_forward_time: 440.2773437500, propagate_backward_time: 981.9324951172, weight_update_time: 0.5225950480, other_time: 29.4710693359
epoch time: 102.6269302368s, input_fired_time: 2.8652226925, output_fired_time: 0.0000000000, propagate_forward_time: 93.0987091064, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.6630020142
validation[          , genome  6320] predictions:    6243/  10000 (62.43%), best:    6243/10000 (62.43%), error:     10757.72461, best error:     10757.72461 on epoch:     2, epoch:    2/25, mu: 0.5632199645, learning_rate: 0.0005684111, weight_decay: 0.0001914732

epoch time: 1449.4353027344s, input_fired_time: 33.8517951965, output_fired_time: 11.4150505066, propagate_forward_time: 426.3348693848, propagate_backward_time: 951.4433593750, weight_update_time: 0.4752243161, other_time: 26.3902587891
epoch time: 91.1301651001s, input_fired_time: 2.1325230598, output_fired_time: 0.0000000000, propagate_forward_time: 84.4916152954, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5060272217
validation[          , genome  6320] predictions:    6083/  10000 (60.83%), best:    6243/10000 (62.43%), error:     11186.20703, best error:     10757.72461 on epoch:     2, epoch:    3/25, mu: 0.5887950659, learning_rate: 0.0005317664, weight_decay: 0.0001805962

epoch time: 1472.9908447266s, input_fired_time: 34.0474891663, output_fired_time: 11.8149700165, propagate_forward_time: 434.1068725586, propagate_backward_time: 966.3470458984, weight_update_time: 0.4813572168, other_time: 26.6744995117
epoch time: 93.9387359619s, input_fired_time: 2.2468245029, output_fired_time: 0.0000000000, propagate_forward_time: 86.7854843140, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9064254761
validation[          , genome  6320] predictions:    6172/  10000 (61.72%), best:    6243/10000 (62.43%), error:     11077.59961, best error:     10757.72461 on epoch:     2, epoch:    4/25, mu: 0.6128375530, learning_rate: 0.0004974841, weight_decay: 0.0001703371

epoch time: 1441.6968994141s, input_fired_time: 33.7012977600, output_fired_time: 11.2730216980, propagate_forward_time: 422.8677978516, propagate_backward_time: 947.5345458984, weight_update_time: 0.4688570499, other_time: 26.3201904297
epoch time: 7882.0087890625s, input_fired_time: 3.9325745106, output_fired_time: 0.0000000000, propagate_forward_time: 7871.9316406250, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1445312500
validation[          , genome  6320] predictions:    6146/  10000 (61.46%), best:    6243/10000 (62.43%), error:     11394.24316, best error:     10757.72461 on epoch:     2, epoch:    5/25, mu: 0.6354392767, learning_rate: 0.0004654119, weight_decay: 0.0001606608

epoch time: 1478.3000488281s, input_fired_time: 35.4590377808, output_fired_time: 12.4175157547, propagate_forward_time: 434.0751647949, propagate_backward_time: 967.0103759766, weight_update_time: 0.5045888424, other_time: 29.3380126953
epoch time: 91.4337692261s, input_fired_time: 2.1975355148, output_fired_time: 0.0000000000, propagate_forward_time: 84.2624511719, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9737854004
validation[          , genome  6320] predictions:    5900/  10000 (59.00%), best:    6243/10000 (62.43%), error:     11549.83594, best error:     10757.72461 on epoch:     2, epoch:    6/25, mu: 0.6566865444, learning_rate: 0.0004354074, weight_decay: 0.0001515342

epoch time: 40397.2187500000s, input_fired_time: 37.2564468384, output_fired_time: 13.8068914413, propagate_forward_time: 452.8576660156, propagate_backward_time: 39861.1914062500, weight_update_time: 0.5434147120, other_time: 32.1015625000
epoch time: 88.5073013306s, input_fired_time: 2.1035470963, output_fired_time: 0.0000000000, propagate_forward_time: 81.8844757080, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.5192794800
validation[          , genome  6320] predictions:    6274/  10000 (62.74%), best:    6274/10000 (62.74%), error:     10696.78418, best error:     10696.78418 on epoch:     7, epoch:    7/25, mu: 0.6766605973, learning_rate: 0.0004073372, weight_decay: 0.0001429260

epoch time: 1586.4160156250s, input_fired_time: 38.4461402893, output_fired_time: 14.2168064117, propagate_forward_time: 469.9793090820, propagate_backward_time: 1030.6970214844, weight_update_time: 0.5839112401, other_time: 33.0766601563
epoch time: 101.4630432129s, input_fired_time: 2.7656567097, output_fired_time: 0.0000000000, propagate_forward_time: 91.9728012085, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7245864868
validation[          , genome  6320] predictions:    6129/  10000 (61.29%), best:    6274/10000 (62.74%), error:     11013.49316, best error:     10696.78418 on epoch:     7, epoch:    8/25, mu: 0.6954376698, learning_rate: 0.0003810767, weight_decay: 0.0001348068

epoch time: 1568.7082519531s, input_fired_time: 38.9744338989, output_fired_time: 16.2233619690, propagate_forward_time: 469.4490661621, propagate_backward_time: 1006.4434204102, weight_update_time: 0.7062166929, other_time: 37.6178588867
epoch time: 5654.1181640625s, input_fired_time: 6.0544118881, output_fired_time: 0.0000000000, propagate_forward_time: 5640.7802734375, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.2836914063
validation[          , genome  6320] predictions:    6263/  10000 (62.63%), best:    6274/10000 (62.74%), error:     10759.84473, best error:     10696.78418 on epoch:     7, epoch:    9/25, mu: 0.7130895257, learning_rate: 0.0003565091, weight_decay: 0.0001271489

epoch time: 1555.0084228516s, input_fired_time: 37.9137687683, output_fired_time: 14.9382123947, propagate_forward_time: 457.1587219238, propagate_backward_time: 1011.4088745117, weight_update_time: 0.5719687939, other_time: 33.5888061523
epoch time: 95.5743560791s, input_fired_time: 2.3890333176, output_fired_time: 0.0000000000, propagate_forward_time: 87.7184600830, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4668655396
validation[          , genome  6320] predictions:    6102/  10000 (61.02%), best:    6274/10000 (62.74%), error:     11123.49512, best error:     10696.78418 on epoch:     7, epoch:   10/25, mu: 0.7296835780, learning_rate: 0.0003335254, weight_decay: 0.0001199260

epoch time: 1537.1156005859s, input_fired_time: 36.6619682312, output_fired_time: 13.7042036057, propagate_forward_time: 453.3910827637, propagate_backward_time: 1001.8161010742, weight_update_time: 0.5465130806, other_time: 31.5421752930
epoch time: 113.4906539917s, input_fired_time: 3.4731225967, output_fired_time: 0.0000000000, propagate_forward_time: 102.6791763306, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 7.3383560181
validation[          , genome  6320] predictions:    6350/  10000 (63.50%), best:    6350/10000 (63.50%), error:     10655.53125, best error:     10655.53125 on epoch:    11, epoch:   11/25, mu: 0.7452831864, learning_rate: 0.0003120235, weight_decay: 0.0001131134

epoch time: 1527.9161376953s, input_fired_time: 35.2114028931, output_fired_time: 12.8100442886, propagate_forward_time: 460.3029174805, propagate_backward_time: 990.8119506836, weight_update_time: 0.5168494582, other_time: 28.7797851563
epoch time: 95.6250152588s, input_fired_time: 2.2209706306, output_fired_time: 0.0000000000, propagate_forward_time: 88.5013122559, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.9027328491
validation[          , genome  6320] predictions:    6242/  10000 (62.42%), best:    6350/10000 (63.50%), error:     10736.68066, best error:     10655.53125 on epoch:    11, epoch:   12/25, mu: 0.7599480152, learning_rate: 0.0002919077, weight_decay: 0.0001066877

epoch time: 1510.6409912109s, input_fired_time: 35.8319015503, output_fired_time: 12.9960603714, propagate_forward_time: 444.7297058105, propagate_backward_time: 987.5614013672, weight_update_time: 0.5108590722, other_time: 29.5218505859
epoch time: 98.6494445801s, input_fired_time: 2.3849601746, output_fired_time: 0.0000000000, propagate_forward_time: 90.8428497314, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.4216308594
validation[          , genome  6320] predictions:    6253/  10000 (62.53%), best:    6253/10000 (62.53%), error:     10614.97070, best error:     10614.97070 on epoch:    13, epoch:   13/25, mu: 0.7737340331, learning_rate: 0.0002730888, weight_decay: 0.0001006272

epoch time: 1548.3944091797s, input_fired_time: 37.5574836731, output_fired_time: 14.2711238861, propagate_forward_time: 457.1232910156, propagate_backward_time: 1007.1444702148, weight_update_time: 0.5664661527, other_time: 32.2980346680
epoch time: 99.3529586792s, input_fired_time: 2.6821410656, output_fired_time: 0.0000000000, propagate_forward_time: 90.7011947632, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9696197510
validation[          , genome  6320] predictions:    6300/  10000 (63.00%), best:    6300/10000 (63.00%), error:     10571.89648, best error:     10571.89648 on epoch:    14, epoch:   14/25, mu: 0.7866939306, learning_rate: 0.0002554831, weight_decay: 0.0000949109

epoch time: 1595.0041503906s, input_fired_time: 38.7978477478, output_fired_time: 14.7710037231, propagate_forward_time: 470.7193603516, propagate_backward_time: 1036.5546875000, weight_update_time: 0.5685188174, other_time: 34.1612548828
epoch time: 94.0726776123s, input_fired_time: 2.1784422398, output_fired_time: 0.0000000000, propagate_forward_time: 87.0850372314, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 4.8091964722
validation[          , genome  6320] predictions:    6083/  10000 (60.83%), best:    6300/10000 (63.00%), error:     11121.41309, best error:     10571.89648 on epoch:    14, epoch:   15/25, mu: 0.7988771796, learning_rate: 0.0002390124, weight_decay: 0.0000895193

epoch time: 1517.7546386719s, input_fired_time: 35.9507942200, output_fired_time: 12.5973520279, propagate_forward_time: 447.7029113770, propagate_backward_time: 992.2798461914, weight_update_time: 0.5052780509, other_time: 29.2236938477
epoch time: 103.4291610718s, input_fired_time: 2.9442749023, output_fired_time: 0.0000000000, propagate_forward_time: 94.3691329956, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.1157531738
validation[          , genome  6320] predictions:    6350/  10000 (63.50%), best:    6350/10000 (63.50%), error:     10429.10449, best error:     10429.10449 on epoch:    16, epoch:   16/25, mu: 0.8103303313, learning_rate: 0.0002236036, weight_decay: 0.0000844340

epoch time: 1591.5699462891s, input_fired_time: 38.0719909668, output_fired_time: 15.2126836777, propagate_forward_time: 469.0812377930, propagate_backward_time: 1036.6929931641, weight_update_time: 0.5791604519, other_time: 32.5111083984
epoch time: 132.3923187256s, input_fired_time: 4.7172460556, output_fired_time: 0.0000000000, propagate_forward_time: 114.9836349487, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 12.6914367676
validation[          , genome  6320] predictions:    5926/  10000 (59.26%), best:    6350/10000 (63.50%), error:     11564.73633, best error:     10429.10449 on epoch:    16, epoch:   17/25, mu: 0.8210971355, learning_rate: 0.0002091882, weight_decay: 0.0000796375

epoch time: 1713.1058349609s, input_fired_time: 42.2610855103, output_fired_time: 17.3860874176, propagate_forward_time: 491.0520324707, propagate_backward_time: 1122.2922363281, weight_update_time: 0.6736487150, other_time: 40.1143798828
epoch time: 95.6609573364s, input_fired_time: 2.3346412182, output_fired_time: 0.0000000000, propagate_forward_time: 88.0361099243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2902069092
validation[          , genome  6320] predictions:    6314/  10000 (63.14%), best:    6350/10000 (63.50%), error:     10548.88281, best error:     10429.10449 on epoch:    16, epoch:   18/25, mu: 0.8312187791, learning_rate: 0.0001957021, weight_decay: 0.0000751136

epoch time: 1567.6176757813s, input_fired_time: 38.0113372803, output_fired_time: 13.7014541626, propagate_forward_time: 461.3601074219, propagate_backward_time: 1023.2126464844, weight_update_time: 0.5432351828, other_time: 31.3321533203
epoch time: 97.1726074219s, input_fired_time: 2.5342564583, output_fired_time: 0.0000000000, propagate_forward_time: 89.2986297607, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3397216797
validation[          , genome  6320] predictions:    6346/  10000 (63.46%), best:    6350/10000 (63.50%), error:     10492.82910, best error:     10429.10449 on epoch:    16, epoch:   19/25, mu: 0.8407338262, learning_rate: 0.0001830854, weight_decay: 0.0000708466

epoch time: 1649.5860595703s, input_fired_time: 39.6062049866, output_fired_time: 15.5545511246, propagate_forward_time: 482.0405883789, propagate_backward_time: 1076.5808105469, weight_update_time: 0.6058493853, other_time: 35.8039550781
epoch time: 99.5907897949s, input_fired_time: 2.5374851227, output_fired_time: 0.0000000000, propagate_forward_time: 91.6773529053, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.3759536743
validation[          , genome  6320] predictions:    6243/  10000 (62.43%), best:    6350/10000 (63.50%), error:     10792.90918, best error:     10429.10449 on epoch:    16, epoch:   20/25, mu: 0.8496786952, learning_rate: 0.0001712821, weight_decay: 0.0000668221

epoch time: 1595.2385253906s, input_fired_time: 37.8772468567, output_fired_time: 14.4747200012, propagate_forward_time: 472.5716857910, propagate_backward_time: 1037.6435546875, weight_update_time: 0.5858936310, other_time: 32.6713867188
epoch time: 113.1743621826s, input_fired_time: 3.0375142097, output_fired_time: 0.0000000000, propagate_forward_time: 103.3755645752, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.7612838745
validation[          , genome  6320] predictions:    6340/  10000 (63.40%), best:    6350/10000 (63.50%), error:     10459.82227, best error:     10429.10449 on epoch:    16, epoch:   21/25, mu: 0.8580875397, learning_rate: 0.0001602398, weight_decay: 0.0000630261

epoch time: 1660.0197753906s, input_fired_time: 38.9252700806, output_fired_time: 15.1922502518, propagate_forward_time: 493.6330566406, propagate_backward_time: 1077.4985351563, weight_update_time: 0.5926404595, other_time: 34.7706298828
epoch time: 111.9505844116s, input_fired_time: 2.9895629883, output_fired_time: 0.0000000000, propagate_forward_time: 102.7171554565, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 6.2438659668
validation[          , genome  6320] predictions:    6320/  10000 (63.20%), best:    6350/10000 (63.50%), error:     10545.78516, best error:     10429.10449 on epoch:    16, epoch:   22/25, mu: 0.8659924865, learning_rate: 0.0001499093, weight_decay: 0.0000594458

epoch time: 1590.8382568359s, input_fired_time: 37.4598426819, output_fired_time: 14.0616760254, propagate_forward_time: 479.9300842285, propagate_backward_time: 1028.0902099609, weight_update_time: 0.5780560374, other_time: 31.2965087891
epoch time: 100.2585525513s, input_fired_time: 2.4011845589, output_fired_time: 0.0000000000, propagate_forward_time: 92.6127319336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.2446365356
validation[          , genome  6320] predictions:    6348/  10000 (63.48%), best:    6348/10000 (63.48%), error:     10401.07617, best error:     10401.07617 on epoch:    23, epoch:   23/25, mu: 0.8734236956, learning_rate: 0.0001402448, weight_decay: 0.0000560689

epoch time: 1506.4569091797s, input_fired_time: 35.2391815186, output_fired_time: 12.7368097305, propagate_forward_time: 455.0286560059, propagate_backward_time: 974.4089355469, weight_update_time: 0.5322435498, other_time: 29.0433349609
epoch time: 90.7960662842s, input_fired_time: 2.2520954609, output_fired_time: 0.0000000000, propagate_forward_time: 83.5184478760, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.0255203247
validation[          , genome  6320] predictions:    6434/  10000 (64.34%), best:    6434/10000 (64.34%), error:     10146.35059, best error:     10146.35059 on epoch:    24, epoch:   24/25, mu: 0.8804095984, learning_rate: 0.0001312034, weight_decay: 0.0000528838

epoch time: 11875.1005859375s, input_fired_time: 33.5375328064, output_fired_time: 11.5879344940, propagate_forward_time: 414.4555969238, propagate_backward_time: 11387.9960937500, weight_update_time: 0.4789553583, other_time: 27.5234375000
epoch time: 104.4591217041s, input_fired_time: 2.7080149651, output_fired_time: 0.0000000000, propagate_forward_time: 96.1757812500, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.5753250122
validation[          , genome  6320] predictions:    6445/  10000 (64.45%), best:    6434/10000 (64.34%), error:     10155.46094, best error:     10146.35059 on epoch:    24, epoch:   25/25, mu: 0.8869768977, learning_rate: 0.0001227449, weight_decay: 0.0000498796

evaluating best weights on full training data.
evaluting training set with running mean/variance:
arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/cifar_10_training.bin'
boincified validation filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified testing filename: '../../projects/csgrid.org_csg/cifar_10_testing.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6320.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1542649503_172_6320_2_r1808486732_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.507075
read mu: 0.893151
read mu_delta: 0.940074
read initial_learning_rate: 0.00064945
read learning_rate: 0.000114832
read learning_rate_delta: 0.935531
read initial_weight_decay: 0.000215232
read weight_decay: 4.70461e-05
read weight_decay_delta: 0.943193
read batch_size: 103
read epsilon: 1e-07
read alpha: 0.0965148
read input_dropout_probability: 0.00124212
read hidden_dropout_probability: 0.0909201
read velocity_reset: 2095
read epoch: 26
read max_epochs: 25
read reset_weights: 0
read padding: 2
read best_epoch: 24
read number_validation_images: 10000
read best_validation_predictions: 6434
read best_validation_error: 10146.4
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 6320
read normal distribution: '0 0.251959174871445 -0.795902907848358'
generator_str: '781944118'
read generator: 781944118
read generated_by_map:
1 split_edge 1
reading 114 nodes.
reading 452 edges.
number input nodes: 3
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
reading image set with 5000 images.
image_size: 3x32x32 = 3072
read 50000 images.
    class    0: 5000
    class    1: 5000
    class    2: 5000
    class    3: 5000
    class    4: 5000
    class    5: 5000
    class    6: 5000
    class    7: 5000
    class    8: 5000
    class    9: 5000
calculating averages and standard deviations for images
average pixel value for channel 0: 0.491399973630905
average pixel value for channel 1: 0.482159107923508
average pixel value for channel 2: 0.446530401706696
pixel variance for channel 0: 0.0610253289341927
pixel standard deviation for channel 0: 0.247033059597015
pixel variance for channel 1: 0.0592849254608154
pixel standard deviation for channel 1: 0.24348495900631
pixel variance for channel 2: 0.0684280395507813
pixel standard deviation for channel 2: 0.261587530374527
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 3
width: 32
height: 32
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 3x32x32 = 3072
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
loaded images
starting backpropagation!
epoch time: 1562.32885742188s, input_fired_time: 36.3347625732422, output_fired_time: 12.3626699447632, propagate_forward_time: 454.802398681641, propagate_backward_time: 1030.39660644531, weight_update_time: 0.48690465092659, other_time: 28.4324951171875
epoch time: 89.9152603149414s, input_fired_time: 2.11932563781738, output_fired_time: 0, propagate_forward_time: 83.3161010742188, propagate_backward_time: 0, weight_update_time: 0, other_time: 4.47983551025391
validation[          , genome  6320] predictions:    6484/  10000 (64.84%), best:    6484/10000 (64.84%), error:     10041.70020, best error:     10041.70020 on epoch:    26, epoch:   26/25, mu: 0.8931506276, learning_rate: 0.0001148317, weight_decay: 0.0000470461

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 940.3787231445s, input_fired_time: 488.5889282227, output_fired_time: 0.0000000000, propagate_forward_time: 426.4137268066, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 25.3760681152
best training[          , genome  6320] predictions:   35771/  50000 (71.54%), best:    6484/10000 (64.84%), error:     42463.36328, best error:     10041.70020 on epoch:    26, epoch:   27/25, mu: 0.8989543915, learning_rate: 0.0001074286, weight_decay: 0.0000443736
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 110.5905914307s, input_fired_time: 2.8509628773, output_fired_time: 0.0000000000, propagate_forward_time: 101.7867202759, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 5.9529113770
   testing[          , genome  6320] predictions:    6484/  10000 (64.84%), best:    6484/10000 (64.84%), error:     10041.70020, best error:     10041.70020 on epoch:    26, epoch:   27/25, mu: 0.8989543915, learning_rate: 0.0001074286, weight_decay: 0.0000443736
backpropagation finished successfully!
09:04:08 (5244): called boinc_finish(0)

</stderr_txt>
]]>