(2.3 hours to learn)
Linear regression is an algorithm for learning to predict a real-valued ``target'' variable as a linear function of one or more real-valued ``input'' variables. It is one of the most widely used statistical learning algorithms, and with care it can be made to work very well in practice. Because it has a closed-form solution, we can exactly analyze many properties of linear regression which have no exact form for other models. This makes it a useful starting point for understanding many other statistical learning algorithms.
This concept has the prerequisites:
Core resources (read/watch one of the following)
→ Stanford's Machine Learning lecture notes
Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.
Location: Chapter 1, section 1, pages 1-7
→ Coursera: Machine Learning (2013)
An online machine learning course aimed at a broad audience.
- Lecture sequence "Linear regression with one variable"
- Lecture sequence "Linear regression with multiple variables": "Multiple features" up through "Features and polynomial regression"
- Click on "Preview" to see the videos.
Supplemental resources (the following are optional, but you may find them useful)
→ The Elements of Statistical Learning
A graudate-level statistical learning textbook with a focus on frequentist methods.
→ Bayesian Reasoning and Machine Learning
A textbook for a graudate machine learning course.
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
- Lecture "Learning the weight of a linear neuron"
- Lecture "The error surface of a linear neuron"
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Sections 3.1-3.1.1, pgs. 137-142
- Linear regression is a model for predicting real-valued targets. Other kinds of targets include:overfitting .
- Some extensions which deal with overfitting include:
- ridge regression
- L1-regularized linear regression (Lasso)
- PCA preprocessing
- Feature selection
- Model selection
- Not all variables of interest can be modeled as linear functions of the input variables. To model nonlinear dependencies, check out:can be interpreted as [maximum likelihood estimation](maximum_likelihood) under a Gaussian noise model.
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation