fitting logistic regression with iterative reweighted least squares
(30 minutes to learn)
One way of fitting logistic regression is using Newton's method. This winds up having an intuitive form, where each update takes the form of a linear regression problem and the data points are all assigned weights depending how far they are from the decision boundary.
This concept has the prerequisites:
- logistic regression
- linear regression: closed-form solution (The IRLS updates have the same form as the linear regression solution, but with reweighted data points.)
Core resources (read/watch one of the following)
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Section 4.3.3, pages 207-208
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location: Sections 8.3.3-8.3.4, pages 249-251
Supplemental resources (the following are optional, but you may find them useful)
→ Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
Location: Chapter 1, section 7, pages 20-21
- While Newton's method is very fast for small scale problems, it doesn't scale very well.
- For large numbers of features, consider quasi-Newton methods .
- For large numbers of data points, consider stochastic gradient descent .
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation