statistical hypothesis testing
(2.4 hours to learn)
Summary
Statistical hypothesis testing is a method for deciding what conclusions can be drawn from data. A central question is determining whether an outcome is statistically significant, or unlikely to have arisen by chance.
Context
This concept has the prerequisites:
- probability
- random variables (Many hypothesis tests concern parameters of distributions.)
- Gaussian distribution (Some canonical examples of statistical tests involve the Gaussian distribution.)
- cumulative distribution function (P-values are determined using the CDF of the test statistic.)
Goals
- Understand what is required of a hypothesis test in the Neyman-Pearson paradigm
- Know basic terminology, including:
- null hypothesis and alternative hypothesis
- test statistic
- type 1 and type 2 errros
- the power function of a test
- the level of a test
- simple and composite hypotheses
- Know what is meant by a p-value and how to compute it from the test statistic and its distribution
- In particular, understand why it doesn't give the probability that a hypothesis is true
- Understand why statistical significance doesn't imply that a difference is large in magnitude
Core resources (read/watch one of the following)
-Paid-
→ Probability and Statistics
An introductory textbook on probability theory and statistics.
Location:
Section 9.1, "Problems of testing hypotheses," pages 530-547
→ All of Statistics
A very concise introductory statistics textbook.
- Chapter 10 introduction, pages 149-152
- Section 10.2, "p-values," pages 156-159
Other notes:
- Skim Section 10.1 to learn about the Wald test.
→ Mathematical Statistics and Data Analysis
An undergraduate statistics textbook.
Location:
Section 9.2, "The Neyman-Pearson paradigm," pages 331-336
See also
- Some examples of statistical hypothesis testing: Often, instead of hypothesis testing, it is preferable to compute a confidence interval .