bias-variance decomposition
(55 minutes to learn)
Summary
The bias-variance decomposition (often referred to as the bias-variance tradeoff) is a frequentist analysis of the generalization capability of an estimator, i.e. a learning algorithm.
Context
This concept has the prerequisites:
- linear regression (The bias-variance decomposition holds exactly in the case of linear regression.)
- linear regression: closed-form solution (The bias-variance decomposition can be derived from the closed-form solution to linear regression.)
- generalization (The bias-variance decomposition is a way of analyzing generalization error.)
Core resources (read/watch one of the following)
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 3.2
Supplemental resources (the following are optional, but you may find them useful)
-Free-
→ Mathematical Monk: Machine Learning (2011)
Online videos on machine learning.
→ The Elements of Statistical Learning
A graudate-level statistical learning textbook with a focus on frequentist methods.
-Paid-
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location:
Section 6.4.4
Other notes:
- not self-contained but provides some nice examples
See also
- The variance term depends on the effective number of parameters of a model.
- Statistical learning theory gives a way of analyzing bias and variance of estimators more generally.
- Cross-validation is a simple and general technique for managing the bias-variance tradeoff. [ (go to concept)](cross_validation)