learning Bayes net parameters with missing data
(2.4 hours to learn)
There is no closed-form solution for the maximum likelihood parameters of a Bayes net when some of the variables are unobserved. However, it is possible to apply the EM algorithm, where the E step involves computing marginals and the M step involves computing the maximum likelihood parameters with fully observed data.
This concept has the prerequisites:
- Be able to use the EM algorithm to learn Bayes net parameters when some of the variables are unobserved.
- Know how to derive the update rules.
- Know how you would implement it if you're given toolboxes for inference and for parameter learning with fully observed data. What outputs are needed from the inference algorithm?
- What is the missing at random assumption, and why is it needed to apply EM?
- In the fully observed case, maximum likelihood decomposed into separate estimation problems for each clique. Why doesn't that happen when there is missing data?
- And why does the decomposition hold in the M step?
- Give an example where the likelihood function is multimodal (and therefore you shouldn't always expect to the global optimum).
- Give an example where the model is unidentifiable, i.e. multiple parameter settings are equally good.
Core resources (read/watch one of the following)
→ Coursera: Probabilistic Graphical Models (2013)
An online course on probabilistic graphical models.
- The lecture "EM in practice" has good practical advice about using EM, and "Latent variables" talks about some cool applications.
- Click on "Preview" to see the videos.
→ Probabilistic Graphical Models: Principles and Techniques
A very comprehensive textbook for a graduate-level course on probabilistic AI.
- Section 19.1, "Foundations," pages 849-862
- Section 18.104.22.168, "The EM algorithm for Bayesian networks," pages 872-875
- The part of 22.214.171.124 about exponential families is optional.
Supplemental resources (the following are optional, but you may find them useful)
-No Additional Notes-
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation