Hopfield Network

Definition

Hopfield network is an associative memory model, which consists of a single layer of fully connected units with symmetric weights and no self-connections. The model recovers complete patterns from partial or noisy inputs.

$E=-\sum _{i<j}{w}_{ij}{x}_i{x}_j$ The network’s state is characterized by its energy value, which the system updates to minimize.

Hebbian Learning Rule

The objective of training is storing specific patterns as local minima of the energy landscape. This is achieved using the Hebbian rule, which strengthens connections between units that are active together.

For a set of patterns $\{{\xi }^{(m)}{\}}_{m=1}^M$, the weights of the network are defined as $w_{ij}=\sum _{m=1}^M{\xi }_i^{(m)}{\xi }_j^{(m)}$

Update Rule

$x_i=\text{sgn}\left({\sum }_{i\ne j}{w}_{ij}{x}_j\right)$ To retrieve a complete pattern from an input, the model does not perform an explicit search. Instead, it descends along the energy landscape’s surface through a process of energy minimization. Starting from a partial or noisy input, the system iteratively updates each unit toward a stable state, where each update monotonically decreases the energy.

Limitation

The Hopfield network has a limited storage capacity. If too many patterns are stored, they begin to interfere with each other, creating unintended local minima called spurious states. The maximum capacity of the network is approximately $0.14N$, where $N$ is the number of units.