recurrent neural networks

(3.2 hours to learn)

Summary

Recurrent neural networks (RNNs) are a kind of neural net often used to model sequence data. They maintain a hidden state which can "remember" certain aspects of the sequence it has seen. RNNs can be trained using backpropagation through time, although efficient training remains an open problem.

Context

This concept has the prerequisites:

Goals

  • Understand what a recurrent neural net is and how it can be used to model sequence data.
  • Know how to train an RNN using backpropagation through time.
  • Be aware of why training RNNs with backpropagation is unstable.

Core resources (read/watch one of the following)

-Free-

→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
[external website]
Author: Geoffrey E. Hinton
Other notes:
  • It may be helpful to first skim Lecture4, which covers sequence modeling.

See also