the Laplace approximation
(40 minutes to learn)
The Laplace approximation is a way of approximating Bayesian parameter estimation and Bayesian model comparison. It is based on a second-order Taylor approximation of the log posterior around the MAP estimate, which results in a Gaussian approximation to the posterior.
This concept has the prerequisites:
- Bayesian parameter estimation (The Laplace approximation is an approximation to Bayesian parameter estimation.)
- Bayesian model comparison (The Laplace approximation is a method for Bayesian model comparison.)
- MAP parameter estimation (The Laplace approximation is a second-order Taylor expansion around the MAP estimate.)
- Taylor approximations (The Laplace approximation is a second-order Taylor expansion around the MAP estimate.)
- multivariate Gaussian distribution (The posterior distribution is approximated with a multivariate Gaussian.)
Core resources (read/watch one of the following)
→ Information Theory, Inference, and Learning Algorithms
A graudate-level textbook on machine learning and information theory.
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Section 4.4, pages 213-217
Supplemental resources (the following are optional, but you may find them useful)
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location: Section 8.4.1, page 255
- Some other approximations to Bayesian parameter estimation include:Bayesian information criterion (BIC) can be [justified](justifying_aic_and_bic) in terms of the Laplace approximation.
- Some other ways to approximate the model evidence include:
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation