# linear regression

(2.3 hours to learn)

## Summary

Linear regression is an algorithm for learning to predict a real-valued ``target'' variable as a linear function of one or more real-valued ``input'' variables. It is one of the most widely used statistical learning algorithms, and with care it can be made to work very well in practice. Because it has a closed-form solution, we can exactly analyze many properties of linear regression which have no exact form for other models. This makes it a useful starting point for understanding many other statistical learning algorithms.

## Context

This concept has the prerequisites:

- matrix multiplication (Linear regression is conveniently represented in terms of matrix-vector multiplication.)

## Core resources (read/watch one of the following)

## -Free-

→ Stanford's Machine Learning lecture notes

Lecture notes for Stanford's machine learning course, aimed at graduate and advanced undergraduate students.

Location:
Chapter 1, section 1, pages 1-7

→ Coursera: Machine Learning (2013)

An online machine learning course aimed at a broad audience.

- Lecture sequence "Linear regression with one variable"
- Lecture sequence "Linear regression with multiple variables": "Multiple features" up through "Features and polynomial regression"

Other notes:

- Click on "Preview" to see the videos.

## Supplemental resources (the following are optional, but you may find them useful)

## -Free-

→ The Elements of Statistical Learning

A graudate-level statistical learning textbook with a focus on frequentist methods.

→ Bayesian Reasoning and Machine Learning

A textbook for a graudate machine learning course.

→ Coursera: Neural Networks for Machine Learning (2012)

An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.

- Lecture "Learning the weight of a linear neuron"
- Lecture "The error surface of a linear neuron"

## -Paid-

→ Pattern Recognition and Machine Learning

A textbook for a graduate machine learning course, with a focus on Bayesian methods.

Location:
Sections 3.1-3.1.1, pgs. 137-142

## See also

- Linear regression is a model for predicting real-valued targets. Other kinds of targets include:
- binary
- categorical
- ordinal (i.e. only the ordering of the values is significant)

- Some extensions which deal with overfitting include:
- ridge regression
- L1-regularized linear regression (Lasso)
- PCA preprocessing
- Feature selection
- Model selection

- Not all variables of interest can be modeled as linear functions of the input variables. To model nonlinear dependencies, check out: Linear regression can be interpreted as [maximum likelihood estimation](maximum_likelihood) under a Gaussian noise model.