EECS E6892 Topics in Information Processing
Bayesian Models for Machine Learning


Columbia University, Spring 2014


Instructor: John Paisley
Location: 403 International Affairs Building
Time: Thursday 4:10-6:40

Office hours: Monday 11-12, Shapiro CEPSR 712
TA: Dawen Liang, dl2771@columbia.edu

Synopsis: This course provides an introduction to Bayesian approaches to machine learning. Topics will include mixed-membership models, latent factor models and Bayesian nonparametric methods. We will also focus on mean-field variational Bayesian inference, an optimization-based approach to approximate posterior learning. Applications of these methods include image processing, topic modeling, collaborative filtering and recommendation systems. We will discuss a selection of these in class.

Text: There is no required text. There will be suggested readings from the literature, and much of the material will come from the following two textbooks:

Christopher Bishop, Pattern Recognition and Machine Learning
David MacKay, Information Theory, Inference, and Learning Algorithms (can be found online for free)

Grading: 4 homeworks (15% each), final project (40%, write-up and short presentation)

Date
Topics covered
Suggested readings
Additional information
1/23/2014
Probability review, Bayes rule, conjugate priors, exponential family
Bishop Ch. 1.2, 2.1-2.4
MacKay Ch. 2, 3, 23

1/30/2014
Bayesian approaches to regression and classification
Bishop Ch. 1.5, 3.1-3.4, 4
MacKay Ch. 27
Homework 1 (Due Feb. 13)

2/6/2014
Hierarchical models, matrix factorization, sparse regression models, sampling
Bishop Ch. 3.5, 7.2, 11.2-11.3
paper 1 (for more details),  paper 2

2/13/2014
EM algorithm, mixture models, factor models
Bishop Ch. 9, 12.1-12.2
MacKay Ch. 22
Homework 2 (Due Feb. 27)

2/20/2014
Variational inference I, mixture of Gaussians
Bishop Ch. 10.1-10.3
MacKay Ch. 33

2/27/2014
Variational inference II, conjugate exponential models, latent Dirichlet allocation Bishop Ch. 10.4
paper 1, chapter (D. Blei)
Homework 3 (Due March 13)

3/6/2014
Variational inference III, approximations for non-conjugate models Bishop 10.5-10.6
paper 1 (for more details), notes

3/13/2014
Variational inference IV, inference for big data sets paper 1 (through Sec. 3.2)
chapter (example application)

3/20/2014
No class (spring break)



3/27/2014
Bayesian nonparameterics I, Dirichlet process and Chinese restaurant process
slides, notes
paper1, paper2
Homework 4 (Due April 17)

4/3/2014
Bayesian nonparameterics II, Dirichlet process and stick-breaking
slides, paper, notes


4/10/2014
Bayesian nonparameterics III, Gaussian process
book Chapters 1, 2, 4
slides

4/17/2014
Bayesian nonparameterics IV, beta process and sparse latent factor models
slides, paper1, paper2


4/24/2014
No class



5/1/2014
Project presentations I & II (Shapiro CEPSR 414)

Project write-up due May 5