maximum likelihood in exponential families
(30 minutes to learn)
For any exponential family model, the maximum likelihood parameters are such that the model moments match the data moments.
This concept has the prerequisites:
- maximum likelihood
- exponential families
- optimization problems (We characterize the maximum likelihood solutions in terms of solutions to an optimization problem.)
- Derive the general formula for the maximum likelihood parameter estimate for exponential families: namely, that the model moments match the empirical moments.
Core resources (read/watch one of the following)
→ Mathematical Monk: Machine Learning (2011)
Online videos on machine learning.
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location: Section 9.2.4, "MLE for the exponential family," pages 286-287
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Section 2.4.1, "Maximum likelihood and sufficient statistics," pages 116-117
- Estimating the parameters of a Markov random field (MRF) is a surprising example of maximum likelihood in exponential families.
- Bayesian parameter estimation often has a [convenient form](bayes_param_exp_fam) in exponential families.
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation