人工神經網絡,中國科學院自動化研究所 吳高巍 2016-11-29,聯結主義學派,又稱仿生學派或生理學派 認為人的思維基元是神經元，而不是符號處理過程 認為人腦不同于電腦 核心：智能的本質是聯接機制。 原理：神經網絡及神經網絡間的連接機制和學習算法,麥卡洛可（McCulloch）,皮茨（Pitts),什么是神經網絡,所謂的人工神經網絡就是基于模仿生物大腦的結構和功能而構成的一種信息處理系統（計算機）。 個體單元相互連接形成多種類型結構的圖 循環、非循環 有向、無向 自底向上（Bottom-Up）AI 起源于生物神經系統 從結構模擬到功能模擬,仿生,人工神經網絡,內容,生物學啟示 多層神經網絡 Hopfield網絡 自組織網絡,生物學啟示,神經元組成：細胞體，軸突，樹突，突觸 神經元之間通過突觸兩兩相連。信息的傳遞發生在突觸。 突觸記錄了神經元間聯系的強弱。 只有達到一定的興奮程度，神經元才向外界傳輸信息。,生物神經元,神經元,神經元特性 信息以預知的確定方向傳遞 一個神經元的樹突細胞體軸突突觸另一個神經元樹突 時空整合性 對不同時間通過同一突觸傳入的信息具有時間整合功能 對同一時間通過不同突觸傳入的信息具有空間整合功能,神經元,工作狀態 興奮狀態，對輸入信息整合后使細胞膜電位升高，當高于動作電位的閾值時，產生神經沖動，并由軸突輸出。 抑制狀態，對輸入信息整合后使細胞膜電位降低，當低于動作電位的閾值時，無神經沖動產生。 結構的可塑性 神經元之間的柔性連接：突觸的信息傳遞特性是可變的 學習記憶的基礎,神經元模型,從生物學結構到數學模型,人工神經元,M-P模型,McClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943,f: 激活函數(Activation Function) g: 組合函數(Combination Function),Weighted Sum Radial Distance,組合函數,(e) (f),Threshold,Linear,Saturating Linear,Logistic Sigmoid,Hyperbolic tangent Sigmoid,Gaussian,激活函數,人工神經網絡,多個人工神經元按照特定的網絡結構聯接在一起，就構成了一個人工神經網絡。 神經網絡的目標就是將輸入轉換成有意義的輸出。,生物系統中的學習,自適應學習 適應的目標是基于對環境信息的響應獲得更好的狀態 在神經層面上，通過突觸強度的改變實現學習 消除某些突觸，建立一些新的突觸,生物系統中的學習,Hebb學習律 神經元同時激活，突觸強度增加 異步激活，突觸強度減弱 學習律符合能量最小原則 保持突觸強度需要能量，所以在需要的地方保持，在不需要的地方不保持。,ANN的學習規則,能量最小 ENERGY MINIMIZATION 對人工神經網絡，需要確定合適的能量定義；可以使用數學上的優化技術來發現如何改變神經元間的聯接權重。,ENERGY = measure of task performance error,兩個主要問題 結構 How to interconnect individual units? 學習方法 How to automatically determine the connection weights or even structure of ANN?,Solutions to these two problems leads to a concrete ANN!,人工神經網絡,前饋結構(Feedforward Architecture) - without loops - static 反饋/循環結構(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems),ANN結構,General structures of feedforward networks,General structures of feedback networks,通過神經網絡所在環境的模擬過程，調整網絡中的自由參數 Learning by data 學習模型 Incremental vs. Batch 兩種類型 Supervised vs. Unsupervised,ANN的學習方法,若兩端的神經元同時激活，增強聯接權重 Unsupervised Learning,學習策略: Hebbrian Learning,最小化實際輸出與期望輸出之間的誤差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P Learning,Objective:,Solution:,學習策略: Error Correction,采用隨機模式，跳出局部極小 - 如果網絡性能提高，新參數被接受. - 否則，新參數依概率接受,學習策略: Stochastic Learning,“勝者為王”(Winner-take-all ) Unsupervised How to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated.,學習策略: Competitive Learning,重要的人工神經網絡模型,多層神經網絡 徑向基網絡 Hopfield網絡 Boltzmann機 自組織網絡 ,多層感知機(MLP),感知機實質上是一種神經元模型 閾值激活函數,Rosenblatt, 1957,感知機,判別規則,輸入空間中 樣本是空間中的一個點 權向量是一個超平面 超平面一邊對應 Y=1 另一邊對應 Y=-1,單層感知機學習,調整權值，減少訓練集上的誤差 簡單的權值更新規則: 初始化 對每一個訓練樣本: Classify with current weights If correct, no change! If wrong: adjust the weight vector,30,學習: Binary Perceptron,初始化 對每一個訓練樣本: Classify with current weights If correct (i.e., y=y*), no change! If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1.,多類判別情況,If we have multiple classes: A weight vector for each class: Score (activation) of a class y: Prediction highest score wins,學習: Multiclass Perceptron,初始化 依次處理每個樣本 Predict with current weights If correct, no change! If wrong: lower score of wrong answer, raise score of right answer,感知機特性,可分性: true if some parameters get the training set perfectly correct Can represent AND, OR, NOT, etc., but not XOR 收斂性: if the training is separable, perceptron will eventually converge (binary case),Separable,Non-Separable,感知機存在的問題,噪聲（不可分情況）: if the data isnt separable, weights might thrash 泛化性: finds a “barely” separating solution,改進感知機,線性可分情況,Which of these linear separators is optimal?,Support Vector Machines,Maximizing the margin: good according to intuition, theory, practice Only support vectors matter; other training examples are ignorable Support vector machines (SVMs) find the separator with max margin,SVM,優化學習,問題描述 訓練數據 目標：發現最好的權值，使得對每一個樣本x的輸出都符合類別標簽 樣本xi的標簽可等價于標簽向量 采用不同的激活函數 平方損失：,單層感知機,單層感知機,單層感知機,單層感知機,采用線性激活函數，權值向量具有解析解 批處理模式 一次性更新權重 缺點：收斂慢 增量模式 逐樣本更新權值 隨機近似，但速度快并能保證收斂,多層感知機 (MLP),層間神經元全連接,MLPs表達能力,3 layers: All continuous functions 4 layers: all functions,How to learn the weights?,waiting B-P algorithm until 1986,B-P Network,結構 A kind of multi-layer perceptron, in which the Sigmoid activation function is used.,B-P 算法,學習方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out layer to hidder layer, then to input layer,Rumelhart & Meclelland, Nature,1986,B-P 算法,Global Error Measure,desired output,generated output,squared error,The objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE),B-P 算法,Step1. Select a pattern from the training set and present it to the network. Step2. Compute activation of input, hidden and output neurons in that sequence. Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs. Step4. Use the calculated error to update all weights in the network, such that a global error measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.,梯度下降方法,Optimization method for finding out the weight vector leading to the MSE,learning rate,gradient,vector form:,element form:,權值更新規則,For output layer:,權值更新規則,For output layer:,權值更新規則,For hidden layer,權值更新規則,For hidden layer,應用: Handwritten digit recognition,3-nearest-neighbor = 2.4% error 40030010 unit MLP = 1.6% error LeNet: 768 192 30 10 unit MLP = 0.9% error Current best (SVMs) 0.4% error,MLPs：討論,實際應用中 Preprocessing is important Normalize each dimension of data to -1, 1 Adapting the learning rate t = 1/t,MLPs：討論,優點： 很強的表達能力 容易執行 缺點： 收斂速度慢 過擬合（Over-fitting） 局部極小,采用Newton法,加正則化項，約束權值的平滑性 采用更少（但足夠數量）的隱層神經元,嘗試不同的初始化 增加擾動,Hopfield 網絡,反饋 結構 可用加權無向圖表示 Dynamic System 兩種類型 Discrete (1982) and Continuous (science, 1984), by Hopfield,Hopfield網絡,Combination function：Weighted Sum Activation function：Threshold,吸引子與穩定性,How do we “program” the solutions of the problem into stable states (attractors) of the network? How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector, called a Lyapunov (Energy) Function.,Hopfield網絡的能量函數,With input Without input,Hopfield 模型,Hopfield證明了異步Hopfield網絡是穩定的，其中權值定義為 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.,Hopfield 網絡: 聯想記憶,Hopfield網絡的一個主要應用 基于與數據部分相似的輸入，可以回想起數據本身（attractor state） 也稱作內容尋址記憶(content-addressable memory).,Stored Pattern,Memory Association,虞臺文, Feedback Networks and Associative Memories,Hopfield 網絡: Associative Memories,Stored Pattern,Memory Association,虞臺文, Feedback Networks and Associative Memories,Hopfield網絡的一個主要應用 基于與數據部分相似的輸入，可以回想起數據本身（attractor state） 也稱作內容尋址記憶(content-addressable memory).,How to store patterns?,=?,How to store patterns?,=?,: Dimension of the stored pattern,權值確定: 外積(Outer Product),Vector form: Element form: Why? Satisfy the Hopfield model,An example of Hopfield memory,虞臺文, Feedback Networks and Associative Memories,Stable,E=4,E=0,E=4,Recall the first pattern (x1),Stable,E=4,E=0,E=4,Recall the second pattern (x2),Hopfield 網絡: 組合優化(Combinatorial Optimization),Hopfield網絡的另一個主要應用 將優化目標函數轉換成能量函數(energy function) 網絡的穩定狀態是優化問題的解,例: Solve Traveling Salesman Problem (TSP),Given n cities with distances dij, what is the shortest tour?,Illustration of TSP Graph,1,2,3,4,5,6,7,8,9,10,11,Hopfield Network for TSP,=?,Hopfield Network for TSP,=,City matrix,Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will be on.,Hopfield Network for TSP,1,2,4,3,5,Time,City,The salesman reaches city 5 at time 3.,Weight determination for TSP: Design Energy Function,Constraint-1,Constraint-2,Constraint-3,能量函數轉換為2DHopfield網絡形式,Network is built!,Hopfield網絡迭代（TSP ）,The initial state generated randomly goes to the stable state (solution) with minimum energy,A 4-city example 阮曉剛， 神經計算科學，2006,自組織特征映射 (SOFM),What is SOFM?,Neural Network with Unsupervised Learning Dimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - Cooperation,Structure of SOFM,競爭(Competition),Finding the best matching weight vector for the present input. Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance,合作(Co