Artificial Neural Network 简明教程

Learning and Adaptation

如前所述，人工神经网络完全受到生物神经系统（即人脑工作方式）的启发。人脑最令人印象深刻的特点是学习，因此人工神经网络也获取了相同的特征。

As stated earlier, ANN is completely inspired by the way biological nervous system, i.e. the human brain works. The most impressive characteristic of the human brain is to learn, hence the same feature is acquired by ANN.

What Is Learning in ANN?

从本质上说，学习意味着根据环境的变化进行调整和适应。人工神经网络是一个复杂的系统，更准确地说，我们可以说它是一个复杂的适应系统，它可以根据通过它的信息来改变其内部结构。

Basically, learning means to do and adapt the change in itself as and when there is a change in environment. ANN is a complex system or more precisely we can say that it is a complex adaptive system, which can change its internal structure based on the information passing through it.

Why Is It important?

作为一种复杂的适应系统，人工神经网络中的学习意味着处理单元能够由于环境的变化而改变其输入/输出行为。当构建特定网络时，人工神经网络中学习的重要性会增加，因为激活函数是固定的，并且输入/输出向量也是固定的。现在要改变输入/输出行为，我们需要调整权重。

Being a complex adaptive system, learning in ANN implies that a processing unit is capable of changing its input/output behavior due to the change in environment. The importance of learning in ANN increases because of the fixed activation function as well as the input/output vector, when a particular network is constructed. Now to change the input/output behavior, we need to adjust the weights.

Classification

可以将其定义为通过发现同一类别的样本之间的共同特征，对样本数据进行区分的学习过程。例如，要执行人工神经网络的训练，我们有一些具有唯一特征的训练样本，并且要执行其测试，我们有一些具有其他唯一特征的测试样本。分类是监督学习的一个示例。

It may be defined as the process of learning to distinguish the data of samples into different classes by finding common features between the samples of the same classes. For example, to perform training of ANN, we have some training samples with unique features, and to perform its testing we have some testing samples with other unique features. Classification is an example of supervised learning.

Neural Network Learning Rules

我们知道，在人工神经网络学习期间，要改变输入/输出行为，我们需要调整权重。因此，需要使用一种方法，借助这种方法可以改变权重。这些方法称为学习规则，它们只是算法或方程。以下是一些神经网络的学习规则 -

We know that, during ANN learning, to change the input/output behavior, we need to adjust the weights. Hence, a method is required with the help of which the weights can be modified. These methods are called Learning rules, which are simply algorithms or equations. Following are some learning rules for the neural network −

Hebbian Learning Rule

这条规则是最古老、最简单的规则之一，是由唐纳德·赫布在 1949 年出版的《行为的组织》一书中提出的。它是一种前馈、无监督学习。

This rule, one of the oldest and simplest, was introduced by Donald Hebb in his book The Organization of Behavior in 1949. It is a kind of feed-forward, unsupervised learning.

Basic Concept - 此规则基于赫布提出的建议，他写道-

Basic Concept − This rule is based on a proposal given by Hebb, who wrote −

"如果细胞 A 的轴突足够接近于激发细胞 B，并且反复或持续地参与激发细胞 B，那么在其中一个或两个细胞中就会发生某种生长过程或代谢变化，使得 A 作为激发 B 的细胞之一的效率会提高。"

“When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”

从以上假设可以得出结论，如果神经元同时激发，那么两个神经元之间的连接可能会加强，如果它们在不同时间激发，那么可能会减弱。

From the above postulate, we can conclude that the connections between two neurons might be strengthened if the neurons fire at the same time and might weaken if they fire at different times.

Mathematical Formulation - 根据赫布学习规则，以下是用于在每个时间步长时增加连接权重的公式。

Mathematical Formulation − According to Hebbian learning rule, following is the formula to increase the weight of connection at every time step.

\Delta w_{ji}(t)\:=\:\alpha x_{i}(t).y_{j}(t)

这里，$\Delta w_{ji}(t)$ ⁡= 时间步长 t 时连接权重增加的增量

Here, $\Delta w_{ji}(t)$ ⁡= increment by which the weight of connection increases at time step t

$\alpha$ = 正的常数学习率

$\alpha$ = the positive and constant learning rate

$x_{i}(t)$ = 时间步长 t 时来自突触前神经元的输入值

$x_{i}(t)$ = the input value from pre-synaptic neuron at time step t

$y_i(t)$= 同一时间步长的突触前神经元输出 t

$y_{i}(t)$ = the output of pre-synaptic neuron at same time step t

Perceptron Learning Rule

此规则是 Rosenblatt 引入的具有线性激活函数的单层前馈网络监督学习算法的纠错。

This rule is an error correcting the supervised learning algorithm of single layer feedforward networks with linear activation function, introduced by Rosenblatt.

Basic Concept − 由于其本质上是监督性的，因此，要计算误差，必须比较期望/目标输出与实际输出。如果发现有任何差异，则必须更改连接的权重。

Basic Concept − As being supervised in nature, to calculate the error, there would be a comparison between the desired/target output and the actual output. If there is any difference found, then a change must be made to the weights of connection.

Mathematical Formulation − 为了解释其数学公式，假设我们有 “n” 个有限输入向量 x(n) 及其期望/目标输出向量 t(n)，其中 n = 1 至 N。

Mathematical Formulation − To explain its mathematical formulation, suppose we have ‘n’ number of finite input vectors, x(n), along with its desired/target output vector t(n), where n = 1 to N.

现在，可以基于净输入计算输出 “y”，如前文所述，表示为净输入上应用的激活函数，如下所示 −

Now the output ‘y’ can be calculated, as explained earlier on the basis of the net input, and activation function being applied over that net input can be expressed as follows −

y\:= f(y_{in})\:= \begin{cases}1, &y_{in}> \theta \\\0, &y_{in} \leqslant \theta\end{cases}

y\:=\:f(y_{in})\:=\:\begin{cases}1, & y_{in}\:>\:\theta \\0, & y_{in}\:\leqslant\:\theta\end{cases}

其中 θ 是阈值。

Where θ is threshold.

可以在以下两种情况下更新权重 −

The updating of weight can be done in the following two cases −

Case I − 当 t ≠ y 时，

Case I − when t ≠ y, then

w(new)\:= w(old)+tx

w(new)\:=\:w(old)\:+\;tx

Case II − 当 t = y 时，

Case II − when t = y, then

不更改权重

No change in weight

Delta Learning Rule (Widrow-Hoff Rule)

由 Bernard Widrow 和 Marcian Hoff 引入，也称为最小均方 (LMS) 方法，以最大限度地减少所有训练模式上的误差。这是一种带有连续激活函数的监督学习算法。

It is introduced by Bernard Widrow and Marcian Hoff, also called Least Mean Square (LMS) method, to minimize the error over all training patterns. It is kind of supervised learning algorithm with having continuous activation function.

Basic Concept − 此规则的基础是梯度下降法，该方法会一直持续下去。Delta 规则更新突触权重，以最大程度地减少输出单元的净输入和目标值。

Basic Concept − The base of this rule is gradient-descent approach, which continues forever. Delta rule updates the synaptic weights so as to minimize the net input to the output unit and the target value.

Mathematical Formulation − 要更新突触权重，Delta 规则由以下内容给出

Mathematical Formulation − To update the synaptic weights, delta rule is given by

\Delta w_{i}\:= \alpha.x_{i}.e_{j}

\Delta w_{i}\:=\:\alpha\:.x_{i}.e_{j}

其中 $\Delta w_{i}$ = 第 i ⁡ 个模式的权重变化；

Here $\Delta w_{i}$ = weight change for ith ⁡pattern;

$\alpha$ = 正的常量学习率；

$\alpha$ = the positive and constant learning rate;

$x_{i}$ = 突触前神经元的输入值；

$x_{i}$ = the input value from pre-synaptic neuron;

$e_{j}$ = $(t\:-\:y_{in})$，理想/目标输出与实际输出 ⁡$y_{in}$ 之间的差值

$e_{j}$ = $(t\:-\:y_{in})$, the difference between the desired/target output and the actual output ⁡$y_{in}$

上述 delta 规则仅适用于单输出单元。

The above delta rule is for a single output unit only.

可以在以下两种情况下更新权重 −

The updating of weight can be done in the following two cases −

Case-I − 当 t ≠ y 时，则

Case-I − when t ≠ y, then

w(new)\:=\:w(old)\:+\:\Delta w

Case-II − 当 t = y 时，则

Case-II − when t = y, then

不更改权重

No change in weight

Competitive Learning Rule (Winner-takes-all)

它涉及无监督训练，其中输出节点试图彼此竞争以表示输入模式。要理解此学习规则，我们必须理解竞争网络，如下所示 −

It is concerned with unsupervised training in which the output nodes try to compete with each other to represent the input pattern. To understand this learning rule, we must understand the competitive network which is given as follows −

Basic Concept of Competitive Network − 此网络就像一个带有输出之间反馈连接的单层前馈网络。输出之间的连接是抑制类型，用虚线表示，这意味着竞争者从不自我支持。

Basic Concept of Competitive Network − This network is just like a single layer feedforward network with feedback connection between outputs. The connections between outputs are inhibitory type, shown by dotted lines, which means the competitors never support themselves.

competitive network

Basic Concept of Competitive Learning Rule − 如前所述，输出节点之间会存在竞争。因此，训练中的主要概念是，对于给定的输入模式，激活程度最高的输出单元将被宣布为获胜者。此规则也称为优胜者全得，因为只有获胜的神经元会更新，其余神经元保持不变。

Basic Concept of Competitive Learning Rule − As said earlier, there will be a competition among the output nodes. Hence, the main concept is that during training, the output unit with the highest activation to a given input pattern, will be declared the winner. This rule is also called Winner-takes-all because only the winning neuron is updated and the rest of the neurons are left unchanged.

Mathematical formulation − 以下是在这个学习规则中重要的三个数学表述因素 −

Mathematical formulation − Following are the three important factors for mathematical formulation of this learning rule −

  1. Condition to be a winner − Suppose if a neuron $y_{k}$⁡ ⁡wants to be the winner then there would be the following condition − y_{k}\:=\:\begin{cases}1 & if\:v_{k}\:>\:v_{j}\:for\:all\:j,\:j\:\neq\:k\\0 & otherwise\end{cases}

这意味着如果某个神经元，比如说 $y_{k}$⁡ ，想获胜，那么它的局部感应场（求和单元的输出），比如说 $v_{k}$，必须在网络中所有其他神经元中最大。

It means that if any neuron, say $y_{k}$⁡ , wants to win, then its induced local field (the output of summation unit), say $v_{k}$, must be the largest among all the other neurons in the network.

  1. Condition of sum total of weight − Another constraint over the competitive learning rule is, the sum total of weights to a particular output neuron is going to be 1. For example, if we consider neuron k then − \displaystyle\sum\limits_{j}w_{kj}\:=\:1\:\:\:\:\:\:\:\:\:for\:all\:k

  2. Change of weight for winner − If a neuron does not respond to the input pattern, then no learning takes place in that neuron. However, if a particular neuron wins, then the corresponding weights are adjusted as follows \Delta w_{kj}\:=\:\begin{cases}-\alpha(x_{j}\:-\:w_{kj}), & if\:neuron\:k\:wins\\0, & if\:neuron\:k\:losses\end{cases}

这里 $\alpha$ 是学习率。

Here $\alpha$ is the learning rate.

这清楚地表明，我们通过调整权重来支持获胜的神经元，如果神经元失败，我们就不必费事重新调整其权重了。

This clearly shows that we are favoring the winning neuron by adjusting its weight and if there is a neuron loss, then we need not bother to re-adjust its weight.

Outstar Learning Rule

格罗斯伯格引入的这一规则与监督式学习有关，因为已知期望输出。它也被称为格罗斯伯格学习。

This rule, introduced by Grossberg, is concerned with supervised learning because the desired outputs are known. It is also called Grossberg learning.

Basic Concept − 该规则应用于按层排列的神经元。它经过特别设计，可产生 p 神经元的 d 层的期望输出。

Basic Concept − This rule is applied over the neurons arranged in a layer. It is specially designed to produce a desired output d of the layer of p neurons.

Mathematical Formulation − 可以在这一规则中计算权重调整，如下所示

Mathematical Formulation − The weight adjustments in this rule are computed as follows

\Delta w_{j}\:=\:\alpha\:(d\:-\:w_{j})

此处 d 是期望的神经元输出，$\alpha$ 是学习率。

Here d is the desired neuron output and $\alpha$ is the learning rate.