learning invariances in neural nets
(40 minutes to learn)
Summary
The human visual system is capable of recognizing objects despite changes in factors such as location, orientation, and lighting. We'd like the representations learned by neural networks to be invariant to at least some of these things as well. There are several different strategies for achieving this, including enforcing invariance in the network architecture, using an appropriate regularization term, or generating randomly perturbed training data.
Context
This concept has the prerequisites:
- backpropagation (We try to modify the backpropagation procedure to encourage invariances.)
- generalization (The point of learning invariances is to improve generalization performance.)
Core resources (read/watch one of the following)
-Free-
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
- Lecture "Why object recognition is difficult"
- Lecture "Achieving viewpoint invariance"
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 5.5.3, pages 261-263
See also
- Some particular strategies for learning invariances:
- building it explicitly into the architecture, as in convolutional nets
- augmenting the training set with warped examples
- tangent propagation , which penalizes instability with respect to transformations
- Tikhonov regularization , which penalizes instability with respect to noise
- The representations in the human visual system are also invariant to various transformations.