1、CNN的早期歷史卷積神經(jīng)網(wǎng)絡(luò)CNNK. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “

2、Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541551, 1989Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 22782324, 19981DL時代的CNN擴展A Krizhe

3、vsky, I Sutskever, GE Hinton. ImageNet classification with deep convolutional neural networks. NIPS2012Y. Jia et al. Caffe: Convolutional Architecture for Fast Feature Embedding. ACM MM2014K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint a

4、rXiv:1409.1556, 2014C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)2卷積示例3卷積形式化4卷積why?1. sparse interactions有限連接，Kernel比輸入小連接數(shù)少很多，學習難度小，計算復雜度低m個節(jié)點與n個節(jié)點相連O(mn)限定k(m)個節(jié)點與n個節(jié)點相連，則為O(kn)

5、5卷積why?1. sparse interactions有限連接，Kernel比輸入小連接數(shù)少很多，學習難度小，計算復雜度低m個節(jié)點與n個節(jié)點相連O(mn)限定k(m)個節(jié)點與n個節(jié)點相連，則為O(kn)6卷積why?1. sparse interactions有限(稀疏)連接Kernel比輸入小局部連接連接數(shù)少很多學習難度小計算復雜度低層級感受野（生物啟發(fā)）越高層的神經(jīng)元，感受野越大7卷積why?2. Parameter Sharing（參數(shù)共享）Tied weights進一步極大的縮減參數(shù)數(shù)量3. Equivariant representations等變性配合Pooling可以獲得平移

6、不變性對scale和rotation不具有此屬性8CNN的基本結(jié)構(gòu)三個步驟卷積突觸前激活，net非線性激活DetectorPoolingLayer的兩種定義復雜定義簡單定義有些層沒有參數(shù)9Pooling10定義（沒有需要學習的參數(shù)）replaces the output of the net at a certain location with a summary statistic of the nearby outputs種類max pooling (weighted) average poolingWhy Pooling?11獲取不變性小的平移不變性：有即可，不管在哪里很強的先驗假設(shè)Th

7、e function the layer learns must be invariant to small translationsWhy Pooling?12獲取不變性小的平移不變性：有即可，不管在哪里旋轉(zhuǎn)不變性？9個不同朝向的kernels（模板）0.20.610.10.50.30.020.050.1Why Pooling?13獲取不變性小的平移不變性：有即可，不管在哪里旋轉(zhuǎn)不變性？9個不同朝向的kernels（模板）0.50.30.0210.40.30.60.30.1Pooling與下采樣結(jié)合更好的獲取平移不變性更高的計算效率（減少了神經(jīng)元數(shù)）14從全連接到有限連接部分鏈接權(quán)重被強制設(shè)

8、置為0通常：非鄰接神經(jīng)元，僅保留相鄰的神經(jīng)元全連接網(wǎng)絡(luò)的特例，大量連接權(quán)重為015Why Convolution & Pooling？a prior probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.模型參數(shù)的先驗概率分布(No free lunch)在見到任何數(shù)據(jù)之前，我們的信念（經(jīng)驗）告訴我們，什么樣的模型參數(shù)是合理的Local connections；對

9、平移的不變性；tied weigts來自生物神經(jīng)系統(tǒng)的啟發(fā)16源起：Neocognitron (1980)SimplecomplexLower orderhigh order17K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980Local Connection源起：Ne

10、ocognitron (1980)18源起：Neocognitron (1980)訓練方法分層自組織competitive learning 無監(jiān)督輸出層獨立訓練有監(jiān)督19LeCun-CNN1989用于字符識別簡化了Neocognitron的結(jié)構(gòu)訓練方法監(jiān)督訓練BP算法正切函數(shù)收斂更快，Sigmoid Loss，SGD用于郵編識別大量應(yīng)用20LeCun-CNN1989用于字符識別輸入16x16圖像L1H112個5x5 kernel8x8個神經(jīng)元L2-H212個5x5x8 kernel4x4個神經(jīng)元L3H330個神經(jīng)元L4輸出層10個神經(jīng)元總連接數(shù)5*5*12*64+5*5*8*12*16+19

11、2*30，約66,000個21LeCun-CNN1989用于字符識別Tied weights對同一個feature map，kernel對不同位置是相同的！22LeCun-CNN1989用于字符識別231998年LeNet數(shù)字/字符識別LeNet-5Feature mapa set of units whose weighs are constrained to be identical. 241998年LeNet數(shù)字/字符識別 例如：C3層參數(shù)個數(shù)(3*6+4*9+6*1)*25 + 16 = 1516 25后續(xù)：CNN用于目標檢測與識別26AlexNet for ImageNet (201

12、2)大規(guī)模CNN網(wǎng)絡(luò)650K神經(jīng)元60M參數(shù)使用了各種技巧DropoutData augmentReLULocal Response NormalizationContrast normalization.27Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems. 2012.AlexNet for Image

13、Net (2012)ReLU激活函數(shù)28AlexNet for ImageNet (2012)實現(xiàn)2塊GPU卡輸入層150,528其它層253,440186,624 64,896 64,896 43,264 4096 4096 1000.29Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems. 2012.A

14、lexNet for ImageNet (2012)ImageNet物體分類任務(wù)上1000類，1,431,167幅圖像30RankNameError rates(TOP5)Description1U. Toronto0.153Deep learning2U. Tokyo0.261Hand-crafted features and learning models.Bottleneck.3U. Oxford0.2704Xerox/INRIA0.271Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classifi

15、cation with deep convolutional neural networks.Advances in neural information processing systems. 2012.AlexNet for ImageNet深度的重要性31網(wǎng)絡(luò)深度87664參數(shù)數(shù)量60M44M10M59M10M性能損失0%1.1%5.7%3.0%33.5%Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural netwo

16、rks.Advances in neural information processing systems. 2012.VGG Net (2014)多個stage每個stage多個卷積層卷積采樣間隔1x1卷積核大小3x31個Pooling層(2x2)16-19層多尺度融合K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014VGG Net (2014)幾種配置Cov3-64:3x3感受野6

17、4個channel33K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014VGG Net (2014)34K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014GoogLeNet (

18、2014)超大規(guī)模22個卷積層的網(wǎng)絡(luò)計算復雜度是AlexNet的4倍左右C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)GoogLeNet (2014)超大規(guī)模24層網(wǎng)絡(luò)Inception結(jié)構(gòu)提取不同scale的特征然后串接起來1x1 convolutions3x3 convolutions5x5 convolut

19、ionsFilter concatenationPrevious layerC. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)GoogLeNet (2014)超大規(guī)模24層網(wǎng)絡(luò)Inception結(jié)構(gòu)提取不同scale的特征，然后串接起來增加1x1的卷積：把響應(yīng)圖的數(shù)量縮小了1x1 convolutions3x3 convolutions5x5 convolutionsFilter concatenationPre