Artificial Neural Network 简明教程

Artificial Neural Network - Hopfield Networks

霍普菲尔德神经网络由约翰·J·霍普菲尔德博士于 1982 年发明。它包括一个层，其包含一个或多个完全连接的递归神经元。霍普菲尔德网络通常用于自动关联和优化任务。

Hopfield neural network was invented by Dr. John J. Hopfield in 1982. It consists of a single layer which contains one or more fully connected recurrent neurons. The Hopfield network is commonly used for auto-association and optimization tasks.

Discrete Hopfield Network

以离散线方式运行的霍普菲尔德网络，或者换句话说，可以认为输入和输出模式是离散矢量，它们在性质上可以是二进制（0,1）或双极（+1，-1）。该网络具有对称的权重，没有自连接，即 wij = wji 和 wii = 0 。

A Hopfield network which operates in a discrete line fashion or in other words, it can be said the input and output patterns are discrete vector, which can be either binary (0,1) or bipolar (+1, -1) in nature. The network has symmetrical weights with no self-connections i.e., wij = wji and wii = 0.

Architecture

以下是一些需要牢记关于离散霍普菲尔德网络的重要要点 −

Following are some important points to keep in mind about discrete Hopfield network −

  1. This model consists of neurons with one inverting and one non-inverting output.

  2. The output of each neuron should be the input of other neurons but not the input of self.

  3. Weight/connection strength is represented by wij.

  4. Connections can be excitatory as well as inhibitory. It would be excitatory, if the output of the neuron is same as the input, otherwise inhibitory.

  5. Weights should be symmetrical, i.e. wij = wji

hopfield

从 Y1 传输到 Y2 、 Yi 和 Yn 的输出分别具有权重 w12 、 w1i 和 w1n 。同样，其他弧线上也有权重。

The output from Y1 going to Y2, Yi and Yn have the weights w12, w1i and w1n respectively. Similarly, other arcs have the weights on them.

Training Algorithm

在离散霍普菲尔德网络训练期间，将更新权重。众所周知，我们可以有二进制输入矢量以及双极输入矢量。因此，在这两种情况下，都可以利用以下关系更新权重

During training of discrete Hopfield network, weights will be updated. As we know that we can have the binary input vectors as well as bipolar input vectors. Hence, in both the cases, weight updates can be done with the following relation

Case 1 − 二进制输入模式

Case 1 − Binary input patterns

对于一组二进制模式 s(p), p = 1 to P

For a set of binary patterns s(p), p = 1 to P

在此， s(p) = s1(p), s2(p),…​, si(p),…​, sn(p)

Here, s(p) = s1(p), s2(p),…​, si(p),…​, sn(p)

权重矩阵由以下公式给出：

Weight Matrix is given by

w_{ij}\:=\:\sum_{p=1}^P[2s_{i}(p)-\:1][2s_{j}(p)-\:1]\:\:\:\:\:对于 i \:\neq\: j

w_{ij}\:=\:\sum_{p=1}^P[2s_{i}(p)-\:1][2s_{j}(p)-\:1]\:\:\:\:\:for\:i\:\neq\:j

Case 2 − 双极性输入模式

Case 2 − Bipolar input patterns

对于一组二进制模式 s(p), p = 1 to P

For a set of binary patterns s(p), p = 1 to P

在此， s(p) = s1(p), s2(p),…​, si(p),…​, sn(p)

Here, s(p) = s1(p), s2(p),…​, si(p),…​, sn(p)

权重矩阵由以下公式给出：

Weight Matrix is given by

w_{ij}\:=\:\sum_{p=1}^P[s_{i}(p)][s_{j}(p)]\:\:\:\:\:对于 i \:\neq\: j

w_{ij}\:=\:\sum_{p=1}^P[s_{i}(p)][s_{j}(p)]\:\:\:\:\:for\:i\:\neq\:j

Testing Algorithm

Step 1 − 使用赫布原理从训练算法获得的权重进行初始化。

Step 1 − Initialize the weights, which are obtained from training algorithm by using Hebbian principle.

Step 2 − 如果网络的激活未合并，则执行步骤 3-9。

Step 2 − Perform steps 3-9, if the activations of the network is not consolidated.

Step 3 − 对于每个输入向量 X ，执行步骤 4-8。

Step 3 − For each input vector X, perform steps 4-8.

Step 4 − 按如下方式使网络的初始激活等于外部输入向量 X −

Step 4 − Make initial activation of the network equal to the external input vector X as follows −

y_{i}\:=\:x_{i}\:\:\对于 i\:=\:1\:至\:n

y_{i}\:=\:x_{i}\:\:\:for\:i\:=\:1\:to\:n

Step 5 − 对于每个单位 Yi ，执行步骤 6-9。

Step 5 − For each unit Yi, perform steps 6-9.

Step 6 − 按如下方式计算网络的净输入 −

Step 6 − Calculate the net input of the network as follows −

y_{ini}\:=\:x_{i}\:+\:\displaystyle\sum\limits_{j}y_{j}w_{ji}

Step 7 − 在净输入上应用激活计算输出 −

Step 7 − Apply the activation as follows over the net input to calculate the output −

y_{i}\:=\begin{cases}1 & 如果\:y_{ini}\:>\:\theta_{i}\\y_{i} & 如果\:y_{ini}\:=\:\theta_{i}\\0 & 如果\:y_{ini}\:<\:\theta_{i}\end{cases}

y_{i}\:=\begin{cases}1 & if\:y_{ini}\:>\:\theta_{i}\\y_{i} & if\:y_{ini}\:=\:\theta_{i}\\0 & if\:y_{ini}\:<\:\theta_{i}\end{cases}

此处，$\theta_{i}$ 为阈值。

Here $\theta_{i}$ is the threshold.

Step 8 − 将此输出 yi 广播至所有其他单位。

Step 8 − Broadcast this output yi to all other units.

Step 9 − 测试连接网络。

Step 9 − Test the network for conjunction.

Energy Function Evaluation

能量函数被定义为系统状态的绑定且非递增函数。

An energy function is defined as a function that is bonded and non-increasing function of the state of the system.

能量函数 Ef ⁡，也称 Lyapunov function 决定离散霍普菲尔德网络的稳定性，其特征如下 −

Energy function Ef⁡, ⁡also called Lyapunov function determines the stability of discrete Hopfield network, and is characterized as follows −

E_{f}\ := \:-\frac{1}{2}\displaystyle\sum\limits_{i=1} n\displaystyle\sum\limits_{j=1} n y_{i}y_{j}w_{ij}\:-\:\displaystyle\sum\limits_{i=1}^n x_{i}y_{i}\:+\:\displaystyle\sum\limits_{i=1}^n \theta_{i}y_{i}

E_{f}\:=\:-\frac{1}{2}\displaystyle\sum\limits_{i=1}n\displaystyle\sum\limits_{j=1}n y_{i}y_{j}w_{ij}\:-\:\displaystyle\sum\limits_{i=1}^n x_{i}y_{i}\:+\:\displaystyle\sum\limits_{i=1}^n \theta_{i}y_{i}

Condition − 在稳定网络中，每当节点状态发生变化，上述能量函数会减少。

Condition − In a stable network, whenever the state of node changes, the above energy function will decrease.

假设 i 节点状态从 $y_i^{(k)}$ 更改为 $y_i^{(k\:+\:1)}$⁡，那么通过以下关系给出了能量变化 $\Delta E_{f}$

Suppose when node i has changed state from $y_i^{(k)}$ to $y_i^{(k\:+\:1)}$ ⁡then the Energy change $\Delta E_{f}$ is given by the following relation

\Delta E_{f}\ := \:E_{f}(y_i {(k+1)})\:-\:E_{f}(y_i {(k)})

\Delta E_{f}\:=\:E_{f}(y_i{(k+1)})\:-\:E_{f}(y_i{(k)})

=\:-\left(\begin{array}{c}\displaystyle\sum\limits_{j=1}^n w_{ij}y_i {(k)}\:+\:x_{i}\:-\:\theta_{i}\end{array}\right)(y_i {(k+1)}\:-\:y_i^{(k)})

=\:-\left(\begin{array}{c}\displaystyle\sum\limits_{j=1}^n w_{ij}y_i{(k)}\:+\:x_{i}\:-\:\theta_{i}\end{array}\right)(y_i{(k+1)}\:-\:y_i^{(k)})

=\:-\:(net_{i})\Delta y_{i}

此处 $\Delta y_{i}\ := \:y_i {(k\:+\:1)}\:-\:y_i {(k)}$

Here $\Delta y_{i}\:=\:y_i{(k\:+\:1)}\:-\:y_i{(k)}$

能量变化取决于这样一个事实，即每次只有一个单元能更新其激活。

The change in energy depends on the fact that only one unit can update its activation at a time.

Continuous Hopfield Network

与离散霍普菲尔德网络相比，连续网络的时间是一个连续变量。它还用于自关联和优化问题，如旅行商问题。

In comparison with Discrete Hopfield network, continuous network has time as a continuous variable. It is also used in auto association and optimization problems such as travelling salesman problem.

Model − 该模型或架构可以通过添加电气组件（如放大器）构建，放大器可将输入电压映射到输出电压的 sigmoid 激活函数上。

Model − The model or architecture can be build up by adding electrical components such as amplifiers which can map the input voltage to the output voltage over a sigmoid activation function.

Energy Function Evaluation

E_f = \frac{1}{2}\displaystyle\sum\limits_{i=1}^n\sum_{\substack{j = 1\\ j \ne i}}^n y_i y_j w_{ij} - \displaystyle\sum\limits_{i=1}^n x_i y_i + \frac{1}{\lambda} \displaystyle\sum\limits_{i=1}^n \sum_{\substack{j = 1\\ j \ne i}}^n w_{ij} g_{ri} \int_{0}^{y_i} a^{-1}(y) dy

此处 λ 是增益参数， gri 输入电导。

Here λ is gain parameter and gri input conductance.