# Gibbs sampling

(45 minutes to learn)

## Summary

Gibbs sampling is a Markov Chain Monte Carlo (MCMC) algorithm where each random variable is iteratively resampled from its conditional distribution given the remaining variables. It's a simple and often highly effective approach for performing posterior inference in probabilistic models.

## Context

This concept has the prerequisites:

- Markov chain Monte Carlo (Gibbs sampling is an MCMC algorithm.)
- conditional distributions (Gibbs sampling is defined in terms of conditional distributions.)
- Markov random fields (The structure of a graphical model shows why the Gibbs updates can be computed efficiently.)

## Core resources (read/watch one of the following)

## -Free-

→ Coursera: Probabilistic Graphical Models (2013)

An online course on probabilistic graphical models.

Location:
Lecture "Gibbs sampling"

Additional dependencies:

- Bayesian networks

Other notes:

- Click on "Preview" to see the videos.

→ Bayesian Reasoning and Machine Learning

A textbook for a graudate machine learning course.

Additional dependencies:

- Bayesian networks
- multivariate Gaussian distribution

## -Paid-

→ Pattern Recognition and Machine Learning

A textbook for a graduate machine learning course, with a focus on Bayesian methods.

Location:
Section 11.3, pages 542-546

Additional dependencies:

- multivariate Gaussian distribution

→ Probabilistic Graphical Models: Principles and Techniques

A very comprehensive textbook for a graduate-level course on probabilistic AI.

Location:
Sections 12.3.1 (pages 505-507) and 12.3.3 (pages 512-515)

Additional dependencies:

- Bayesian networks

## Supplemental resources (the following are optional, but you may find them useful)

## -Free-

→ Information Theory, Inference, and Learning Algorithms

A graudate-level textbook on machine learning and information theory.

Additional dependencies:

- Metropolis-Hastings algorithm

→ Machine learning summer school: Markov chain Monte Carlo (2009)

## -Paid-

→ Machine Learning: a Probabilistic Perspective

A very comprehensive graudate-level machine learning textbook.

Location:
Section 24.2-24.2.2, pages 838-839

## See also

- Gibbs sampling can be viewed as a special case of Metropolis-Hastings .
- Naive Gibbs sampling is often very slow to mix. Some improved versions include:
- block Gibbs sampling , where we sample multiple variables at a time
- collapsed Gibbs sampling , where some of the variables are integrated out in closed form

- We can analyze the mixing rate using spectral graph theory.