EECS E6892 Topics in Information Processing
Bayesian Models for Machine Learning

Columbia University, Spring 2014

Instructor: John Paisley
Location: 403 International Affairs Building
Time: Thursday 4:10-6:40

Office hours: Monday 11-12, Shapiro CEPSR 712
TA: Dawen Liang,

Synopsis: This course provides an introduction to Bayesian approaches to machine learning. Topics will include mixed-membership models, latent factor models and Bayesian nonparametric methods. We will also focus on mean-field variational Bayesian inference, an optimization-based approach to approximate posterior learning. Applications of these methods include image processing, topic modeling, collaborative filtering and recommendation systems. We will discuss a selection of these in class.

Text: There is no required text. There will be suggested readings from the literature, and much of the material will come from the following two textbooks:

Christopher Bishop, Pattern Recognition and Machine Learning
David MacKay, Information Theory, Inference, and Learning Algorithms (can be found online for free)

Grading: 4 homeworks (15% each), final project (40%, write-up and short presentation)

Topics covered
Suggested readings
Additional information
Probability review, Bayes rule, conjugate priors, exponential family
Bishop Ch. 1.2, 2.1-2.4
MacKay Ch. 2, 3, 23

Bayesian approaches to regression and classification
Bishop Ch. 1.5, 3.1-3.4, 4
MacKay Ch. 27
Homework 1 (Due Feb. 13)

Hierarchical models, matrix factorization, sparse regression models, sampling
Bishop Ch. 3.5, 7.2, 11.2-11.3
paper 1 (for more details),  paper 2

EM algorithm, mixture models, factor models
Bishop Ch. 9, 12.1-12.2
MacKay Ch. 22
Homework 2 (Due Feb. 27)

Variational inference I, mixture of Gaussians
Bishop Ch. 10.1-10.3
MacKay Ch. 33

Variational inference II, conjugate exponential models, latent Dirichlet allocation Bishop Ch. 10.4
paper 1, chapter (D. Blei)
Homework 3 (Due March 13)

Variational inference III, approximations for non-conjugate models Bishop 10.5-10.6
paper 1 (for more details), notes

Variational inference IV, inference for big data sets paper 1 (through Sec. 3.2)
chapter (example application)

No class (spring break)

Bayesian nonparameterics I, Dirichlet process and Chinese restaurant process
slides, notes
paper1, paper2
Homework 4 (Due April 17)

Bayesian nonparameterics II, Dirichlet process and stick-breaking
slides, paper, notes

Bayesian nonparameterics III, Gaussian process
book Chapters 1, 2, 4

Bayesian nonparameterics IV, beta process and sparse latent factor models
slides, paper1, paper2

No class

Project presentations I & II (Shapiro CEPSR 414)

Project write-up due May 5