covariance
(1.4 hours to learn)
Summary
The covariance of two random variables is a measure of their relatedness. It is closely related to the correlation coefficient, but is more commonly used in probability theory because it has nice mathematical properties.
Context
This concept has the prerequisites:
- expectation and variance (Covariance is a generalization of variance.)
- multivariate distributions (Covariance is defined in terms of a multivariate PMF or PDF.)
- independent random variables (Independent random variables have zero covariance.)
Goals
- Know the definitions of covariance and correlation
- Write the covariance in terms of the moments of the distribution
- Know the Cauchy-Schwartz inequality for covariance (which bounds the covariance in terms of the individual variances)
- Be able to compute the variance of a linear combination of random variables in terms of their variances and covariances
- Know that independent random variables have zero covariance
- Show that variance of a sum of independent random variables is a sum of the variances
Core resources (read/watch one of the following)
-Free-
→ Mathematical Monk: Probability Primer (2011)
Online videos on probability theory.
Other notes:
- This uses the measure theoretic notion of probability, but should still be accessible without that background. Refer to Lecture 1.S for unfamiliar terms.
-Paid-
→ Probability and Statistics
An introductory textbook on probability theory and statistics.
Location:
Section 4.6, "Covariance and correlation," pages 214-221
→ Mathematical Statistics and Data Analysis
An undergraduate statistics textbook.
Location:
Section 4.3, "Covariance and correlation," pages 138-146
→ A First Course in Probability
An introductory probability textbook.
Location:
Section 7.4, "Covariance, variance of sums, and correlations," pages 355-364
→ An Introduction to Probability Theory and its Applications
A classic introductory probability textbook.
- Section 9.5, "Covariance
- variance of a sum," pages 215-219
See also
- Covariance matrices are a way of representing all of the variances and covariances between a set of random variables.
- Multivariate Gaussians are a widely used "bell-shaped" distribution, parameterized in terms of expectation and covariance.