linear regression
(2.3 hours to learn)
Summary
Linear regression is an algorithm for learning to predict a real-valued ``target'' variable as a linear function of one or more real-valued ``input'' variables. It is one of the most widely used statistical learning algorithms, and with care it can be made to work very well in practice. Because it has a closed-form solution, we can exactly analyze many properties of linear regression which have no exact form for other models. This makes it a useful starting point for understanding many other statistical learning algorithms.
Context
This concept has the prerequisites:
- matrix multiplication (Linear regression is conveniently represented in terms of matrix-vector multiplication.)
Core resources (read/watch one of the following)
-Free-
→ Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
Location:
Chapter 1, section 1, pages 1-7
→ Coursera: Machine Learning (2013)
An online machine learning course aimed at a broad audience.
- Lecture sequence "Linear regression with one variable"
- Lecture sequence "Linear regression with multiple variables": "Multiple features" up through "Features and polynomial regression"
Other notes:
- Click on "Preview" to see the videos.
Supplemental resources (the following are optional, but you may find them useful)
-Free-
→ The Elements of Statistical Learning
A graudate-level statistical learning textbook with a focus on frequentist methods.
→ Bayesian Reasoning and Machine Learning
A textbook for a graudate machine learning course.
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
- Lecture "Learning the weight of a linear neuron"
- Lecture "The error surface of a linear neuron"
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Sections 3.1-3.1.1, pgs. 137-142
See also
- Linear regression is a model for predicting real-valued targets. Other kinds of targets include:
- binary
- categorical
- ordinal (i.e. only the ordering of the values is significant)
- Some extensions which deal with overfitting include:
- ridge regression
- L1-regularized linear regression (Lasso)
- PCA preprocessing
- Feature selection
- Model selection
- Not all variables of interest can be modeled as linear functions of the input variables. To model nonlinear dependencies, check out: Linear regression can be interpreted as [maximum likelihood estimation](maximum_likelihood) under a Gaussian noise model.