backpropagation

(2.2 hours to learn)

Summary

Backpropagation is the standard algorithm for training supervised feed-forward neural nets. More precisely, it isn't actually a learning algorithm, but a way of computing the gradient of the loss function with respect to the network parameters. Mathematically, it's just an instance of the chain rule for derivatives, but it has an intuitive interpretation in terms of passing messages between the units.

Context

This concept has the prerequisites:

Core resources (read/watch one of the following)

-Free-

Coursera: Machine Learning
An online machine learning course aimed at advanced undergraduates.
Author: Pedro Domingos
Other notes:
  • Click on "Preview" to see the videos.
Coursera: Machine Learning (2013)
An online machine learning course aimed at a broad audience.
Location: Lecture series "Neural networks: learning"
Author: Andrew Y. Ng
Other notes:
  • Click on "Preview" to see the videos.
Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
  • Lecture "Learning the weights of a logistic output neuron"
  • Lecture "The backpropagation algorithm"
  • Lecture "Using the derivatives computed by backpropagation"
Author: Geoffrey E. Hinton
The Elements of Statistical Learning
A graudate-level statistical learning textbook with a focus on frequentist methods.
Authors: Trevor Hastie,Robert Tibshirani,Jerome Friedman
Other notes:
  • Sections 11.5-11.7 discuss practical issues and examples in training neural nets.

-Paid-

Supplemental resources (the following are optional, but you may find them useful)

-Paid-

See also