234

DiscreteValueContinuousValue圖像表示：Gabor圖像表示：Gabor圖像集表示：Manifold,GMM,LMNN,NCA…..詞典學習&CreditedtoProf.XiaogangWangandProf.Shiguang CreditedtoProf.Songchun DiscreteValueContinuousValue端到端學習(EndtoCreditedtoProf.Shiguang CreditedtoDr.Naiyan CreditedtoDr.Naiyan 多多PascalVOC目標檢測CreditedtoProf.ShiguangShanwith

Fastdescriptorcoding(LLCJinjunWang’sCVPR10&Super+SVMLin,Y’sCompressedFisherVectorsPerronnin,F’sECCV10,Sanchez,J’sDCNNAlexNet8層網絡KrizhevskyA’sDCNN:基于網絡可視化技術Zeiler,M.D’sECCV14在AlexNetDCNN:NetChristianSzegedy’sCVPR15，22層+Inception結DCNN:VGGNetKarenSimonyan’sarXiv14，19DCNN:ResidualNetKaimingHe’sarXiv15,152SelectiveSearchJ.R.R.Uijlings’sIJCV13+efficientencodingdenselysampledcolordescriptorsvandeSande’sTPAMI10,CVPR14RCNNRossGirshick’sCVPR14+NetworkInNetworkMinLin’sFasterRCNNShaoqingRen’sNIPS15DeepResidualNetworkKaimingHe’sMeanMean

CombinedMultishot

VectorVector

gh-dimintgh-dim

TLTL

Unrestricted Outside Unrestricted+LabeledOutside 網絡變大變深（VGGFace16FaceNet22層數據量不斷增大（DeepFace400萬，FaceNet2億3DeepID[Sun’sDeepID2[Sun’sDeepID2+[Sun’s-1FaceNet[Schroff’s1ModifiedfromProf.ModifiedfromProf.LeCunandProf. 來自：BoleiZhouetalObjectDetectorsEmergeinDeepSceneCNNsICLRConv，Conv，Pooling,ReLU,connected,BN，NIN， N,SiameseNet,機制LossFunc,e.g.SoftmaxLoss,SigmoidCross-entropy,…BP,SGD,AdaDelta,Path-Dropout,Fine-tune,CreditedtoProf.MeinaKan,PHDStudentXinLiuandShuzhe McCulloch,Warren;WalterPitts(1943)."ALogicalCalculusofIdeasImmanentinNervousActivity".BulletinofMathematicalBiophysics5(4):115–133F.Rosenblatt.Theperceptron:Aprobabilisticmodelforinformationstorageandorganizationinthebrain. PsychologicalReview,65:386-408,Minsky&Papert的專著Perceptron(1969) Novikoff,A.B.J.(1962).Onconvergenceproofsonperceptrons.ProceedingsoftheSymposiumontheMathematicalTheoryofAutomata(pp.615–622)IEEEFrankRosenblatt2014-GeoffreyE.Hinton2012-VladimirN.Vapnik2009-JohnJ.Hopfield多層感知機卷積網絡

DecisionSparseGraphRumelhart,DavidE.;Hinton,GeoffreyE.;Williams,RonaldJ.(8October1986)."Learningrepresentationsbyback-propagatingerrors".Nature323(6088):533–536.CreditedtoProf.ShiguangHinton,G.E.,Osindero,S.andTeh,Y.,Afastlearningalgorithmfordeepbeliefnets.NeuralComputation18:1527-1554,2006Hinton,G.E.andSalakhutdinov,R.R.Reducingthedimensionalityofdatawithneuralnetworks.Science,Vol.313.no.5786,pp.504-507,28July2006YoshuaBengio,PascalLamblin,DanPopoviciandHugoLaroce,GreedyLayer-WiseTrainingofDeepNetworks,NIPS2006CreditedtoProf.Eric deeplearningDCNNAlexKrizhevsky’s止過擬合,LocalResponseNormalization增強泛化能力評價評價 評價RNNLanguage評價CreditedtoProf.MeinaKan,PHDStudentXinLiuandShuzhe ReLU/

MoonMoonStart...：... ，其中??∈?????1,??∈?????1,??∈??：...Backward運算：????=???????，????=?? 連續卷積???,

=?????(??,??)

????,?????????,?????離散卷積????,Caffe實現：????

=?????(??,??) ??????,?????????,???=?????(??,??) ??????+??,??+??????,多個FeatureMap: 接

CreditedtoDr.ShaoxinCreditedtoDr.ShaoxinPleasealsoreferto“NotesonConvolutionalNeural …Image:??×??×…

FeatureMatrixX:(????????×????????)×(C×K×…Kernel:????????×??×??×…bias:????????×1×1×

??????+??:????????×(??×??× …………??:????????×(????????×……OutputMatrix:????????×(????????×

Kernelsize:2x2Kernelsize:2x24346378445555686823

S(x)=1/(1+e-xy=xifx>

a(e-1)ifx￡max(w1x1+b1,w2x2+b2 LeCun組2010年的文章Whatisthebestmulti-stagearchitectureforobjectrecognition?嘗試了各種非線性激活函Hinton組的“RectifiedLinearUnitsImproveRestrictedBoltzmannMachines”將NReLU用于RBM訓練 48引入Bernoulli隨機數????代表dropout測試階段：DoG.E.Hinton,N.Srivastava,A.Krizhevsky,I.Sutskever,andR.R.Salakhutdinov.Improvingneuralnetworksbypreventingco-adaptationoffeaturedetectors.arXivpreprintarXiv:1207.0580,2012. Batch UntersuchungenzudynamischenneuronalenNetzen(1991LearningLong-TermDependencieswithGradientDescentisDifficult(1994引入三個Gate結構：ForgetGate,InputGate和Output 狀態與#1.狀態“遺忘”控制門forget狀態與#1.狀態“遺忘”控制門forget#2.輸入控制門input狀態與#1.狀態“遺忘”控制門forget#2.輸入控制門input狀態與#1.狀態“遺忘”控制門forget#2.輸入控制門input#3:#4.輸出控制門outputPeepholeGRU（inputgate和forgetgate合并為updateSoftmax+CrossEntropyE=-1N

log( ),l?[0,1,...,K-

n??nkn

CrossEntropyLossH(p,q)=-p(x)logxlog(x)'=x

當xSigmoidCrossEntropy

?npnlo1?Tips1:目標輸出需要歸一化到[0,1]XinLiu*,ShaoxinLi*etal.AgeNet:DeeplyLearnedRegressorandClassifierforRobustApparentAgeEstimation.InternationalConferenceonComputerVisionChaLearnLaPWorkshop(ICCVW),2015.SoftmaxSigmoidCrossEntropy

E=- E=-

log(?nl),ln?[0,1,...,K-n?n?nSigmoidCrossEntropyLossGroud_truth：Pn=(pn,1-pnSigmoid??,1?

?n)

log?

plo1?

L[ log o1? SoftmaxLoss

Euclidean損失函數：E=

?n Tips1:默認的，Data層和ImageData Tips2:歐式損失前可以增加Sigmoid操 ContrastiveLoss（Siamese

1

[(y)d2+(1-y)max(margin-d,0)2,d

a-

nTips2:Caffe中的例子：TripletBasic適用場景：Learningtorank,人臉識別F.Schroff,D.Kalenichenko,andJ.Philbin,FaceNet:AUnifiedEmbeddingforFaceRecognition MoonEthanM.Ruddetal,MOON:AMixedObjectiveOptimizationNetworkfortheRecognitionofFacialAttributesCreditedtoProf.Back Rumelhart,DavidE.;Hinton,GeoffreyE.;Williams,RonaldJ.(8October1986)."Learningrepresentationsbyback-propagatingerrors".Nature323(6088):533–536.GradientDescentanditswt+1=wt w

StochasticGradientDescendwt+1=wt wL(w,xi,yi

: =w-1h

L(w,x,y

Mini-batchMini-batchfori=1: WeightL(w)=w)+L(w)= L(wtwt+1=wt+Without WithSGDvs.給定等高線下不同優化算法的比 鞍點條件下不同優化算法的比Inmathematics,asaddlepointisapointinthe ofafunctionwheretheslopes(derivatives)oforthogonalfunctioncomponentsdefiningthesurfaceezero(astationarypoint)butarenotaloc9以為例，點（0,0）是鞍點，CreditedtoPHDStudentXinLiuandXiaoyi LearningCreditedtoProf.LearningRate在每隔stepsize輪迭代后減少base_lr: :base_lr: :“poly”power:0.5

base_lr:base_lr: :base_lr: :“step” :0.1stepsize:

LR(t)=base_lr·

t)Tbase_lr:0.01 :“inv”base_lr:0.01 :“inv” :power:LR(t)=base_lr·(1+g*iter)-LearningtolearnwithrecurrentneuralMa