Hopfield networks
(3.4 hours to learn)
Summary
Hopfield networks are a kind of recurrent neural network which implements an associative memory. The behavior of the network can be modeled in terms of minimizing an energy function.
Context
-this concept has no prerequisites-
Goals
- Know the rules for
- learning the weights of a Hopfield net
- finding a low energy configuration
- Understand why the dynamics of the network can be described in terms of an energy function.
- What happens if you try to store too many memories in a Hopfield net?
- Know how Hopfield nets can be applied to optimization problems (e.g. image interpretation, Traveling Salesman)
Core resources (read/watch one of the following)
-Free-
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
Other notes:
- Skim the introductory networks for a general motivation of neural nets.
→ Information Theory, Inference, and Learning Algorithms
A graudate-level textbook on machine learning and information theory.
Other notes:
- The argument about variational energy and the mathematical analysis of Section 42.7 are optional.
See also
- Boltzmann machines (and in particular, [restricted Boltzmann machines (RBMs)](restricted_boltzmann_machines) ), are a modern probabilistic analogue of Hopfield nets.
- The mean field approximation updates in an Ising model have a similar form to Hopfield nets.