generalized linear models
(2.1 hours to learn)
Summary
Generalized linear models are a class of linear models which unify several widely used models, including linear regression and logistic regression. The distribution over each output is assumed to be an exponential family distribution whose natural parameters are a linear function of the inputs.
Context
This concept has the prerequisites:
- exponential families (Generalized linear models are defined in terms of exponential family distributions.)
- maximum likelihood (Generalized linear models can be fit using maximum likelihood.)
- optimization problems (Maximum likelihood is an optimization problem.)
- linear regression as maximum likelihood (Linear regression is an instructive example of a generalized linear model.)
- gradient descent (The gradient ascent update rule for generalized linear models has a convenient form.)
Goals
- Understand the basic assumptions behind a generalized linear model.
- In particular, the natural parameters are assumed to be a linear function of the inputs. What role does this play in the maximum likelihood update rule?
- Show that some other model (e.g. linear regression or logistic regression) is a special case of generalized linear models.
- Derive the gradient ascent update rule for GLMs.
Core resources (read/watch one of the following)
-Free-
→ Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
Other notes:
- Section 5 explains logistic regression.
Supplemental resources (the following are optional, but you may find them useful)
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 4.3.6, "Canonical link functions," pages 212-213
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location:
Section 9.3, "Generalized linear models (GLMs)," pages 290-293
See also
- Important example of generalized linear models include: Exponential family PCA is a class of matrix factorization models based on the same ideas as generalized linear models.