# maximum likelihood in exponential families

(30 minutes to learn)

## Summary

For any exponential family model, the maximum likelihood parameters are such that the model moments match the data moments.

## Context

This concept has the prerequisites:

- maximum likelihood
- exponential families
- optimization problems (We characterize the maximum likelihood solutions in terms of solutions to an optimization problem.)

## Goals

- Derive the general formula for the maximum likelihood parameter estimate for exponential families: namely, that the model moments match the empirical moments.

## Core resources (read/watch one of the following)

## -Free-

→ Mathematical Monk: Machine Learning (2011)

Online videos on machine learning.

## -Paid-

→ Machine Learning: a Probabilistic Perspective

A very comprehensive graudate-level machine learning textbook.

Location:
Section 9.2.4, "MLE for the exponential family," pages 286-287

→ Pattern Recognition and Machine Learning

A textbook for a graduate machine learning course, with a focus on Bayesian methods.

Location:
Section 2.4.1, "Maximum likelihood and sufficient statistics," pages 116-117

## See also

- Estimating the parameters of a Markov random field (MRF) is a surprising example of maximum likelihood in exponential families.
- Bayesian parameter estimation often has a [convenient form](bayes_param_exp_fam) in exponential families.