exponential families

(45 minutes to learn)


Exponential families are a broad class of probability distributions which includes many basic distributions such as Bernoullis and Gaussians, as well as Markov random fields. What they have in common is that the distributions can be represented in terms of log-linear functions of sufficient statistics.


This concept has the prerequisites:


  • Know the basic definitions: exponential family, natural parameter, sufficient statistic
  • Derive the exponential family representations of some simple distributions, e.g.
    • Gaussian
    • Bernoulli
  • Give an example of a family of distributions which is not an exponential family.

Core resources (read/watch one of the following)


Mathematical Monk: Machine Learning (2011)
Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
Author: Andrew Y. Ng


Supplemental resources (the following are optional, but you may find them useful)


See also