AdaBoost
(1.1 hours to learn)
Summary
AdaBoost is an example of a boosting algorithm, where the goal is to take a "weak classifier" (one which performs slightly above chance) and make it into a "strong classifier" (one which performs well on the training set). It is widely used in data mining, especially in conjunction with decision trees, because of its simplicity and effectiveness.
Context
-this concept has no prerequisites-
Goals
- Understand the steps in the AdaBoost algorithm
- Be aware of the underlying motivation, namely taking a "weak classifier" which performs slightly above chance, and producing a "strong classifier," which classifies the whole training set correctly.
Core resources (read/watch one of the following)
-Free-
→ The Elements of Statistical Learning
A graudate-level statistical learning textbook with a focus on frequentist methods.
Other notes:
- Read the introductory chapters if you're not familiar with the basic machine learning setup.
→ Coursera: Machine Learning
An online machine learning course aimed at advanced undergraduates.
Other notes:
- Watch the Week One lectures if you're not familiar with the basic machine learning setup.
- Click on "Preview" to see the videos.
-Paid-
→ Artificial Intelligence: a Modern Approach
A textbook giving a broad overview of all of AI.
Location:
Section 18.4, "Ensemble learning," pages 664-668
Other notes:
- Read 18.1-18.2 if you're not familiar with the basic machine learning setup, and skim 18.3 to learn about decision trees.
Supplemental resources (the following are optional, but you may find them useful)
-Paid-
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 14.3, "Boosting," not including 14.3.1, pages 657-659
See also
- AdaBoost can be interpreted as a greedy algorithm to minimize exponential loss .
- Decision trees are often used as the base classifier.