exponential families
(45 minutes to learn)
Summary
Exponential families are a broad class of probability distributions which includes many basic distributions such as Bernoullis and Gaussians, as well as Markov random fields. What they have in common is that the distributions can be represented in terms of log-linear functions of sufficient statistics.
Context
This concept has the prerequisites:
- binomial distribution (The binomial distribution is an important example of an exponential family distribution.)
- Gaussian distribution (The Gaussian distribution is an important example of an exponential family distribution.)
- expectation and variance (Expected value is needed to motivate the notion of sufficient statistics.)
Goals
- Know the basic definitions: exponential family, natural parameter, sufficient statistic
- Derive the exponential family representations of some simple distributions, e.g.
- Gaussian
- Bernoulli
- Give an example of a family of distributions which is not an exponential family.
Core resources (read/watch one of the following)
-Free-
→ Mathematical Monk: Machine Learning (2011)
Online videos on machine learning.
→ Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 2.4, "The exponential family," not counting subsections, pages 113-116
→ Probabilistic Graphical Models: Principles and Techniques
A very comprehensive textbook for a graduate-level course on probabilistic AI.
Location:
Section 8.2, "Exponential families," pages 261-266
Supplemental resources (the following are optional, but you may find them useful)
-Paid-
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location:
Section 9.2, "The exponential family," up to 9.2.4, "MLE for the exponential family," pages 281-283
See also
- Exponential families can be parameterized in terms of sufficient statistics .
- Generalized linear models are a generalization of linear regression where the observation model is an exponential family distribution.
- A lot of aspects of statistical modeling wind up having convenient forms for exponential families:
- conjugate priors
- maximum likelihood estimation
- variational inference , a kind of approximate probabilistic inference algorithm