learning invariances in neural nets
(40 minutes to learn)
The human visual system is capable of recognizing objects despite changes in factors such as location, orientation, and lighting. We'd like the representations learned by neural networks to be invariant to at least some of these things as well. There are several different strategies for achieving this, including enforcing invariance in the network architecture, using an appropriate regularization term, or generating randomly perturbed training data.
This concept has the prerequisites:
Core resources (read/watch one of the following)
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
- Lecture "Why object recognition is difficult"
- Lecture "Achieving viewpoint invariance"
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Section 5.5.3, pages 261-263
- Some particular strategies for learning invariances:
- building it explicitly into the architecture, as in convolutional nets
- augmenting the training set with warped examples
- tangent propagation , which penalizes instability with respect to transformations
- Tikhonov regularization , which penalizes instability with respect to noise
- The representations in the human visual system are also invariant to various transformations.