# Lecture Notes on Stochastic Modeling I

Professor Karl Sigman

Department of Industrial Engineering and Operations Research

karl.sigman@columbia.edu

http://www.ieor.columbia.edu/~sigman

Stochastic Modeling I Lecture Notes WEB SITE:
http://www.ieor.columbia.edu/~sigman/stochastic-I.html
### Lecture Notes

- Introduction to discrete-time Markov chains I
- Gambler's ruin problem
- Stopping times
- Markov chains II:
recurrence and limiting (stationary) distributions
- Time-reversible Markov chains
- Uniform integrability
- Poisson processes, elementary renewal theorem with proof
- Borel-Cantelli Lemmas
- Continuous-time Markov chains
- Little's law
- Renewal Theory I; inspection paradox, renewal reward theorem
- Renewal theory II; central limit theorem for
counting processes, stationary renewal processes, key renewal theorem, weak convergence
- Regenerative processes
- Discrete-time renewal processes
- Introduction to Discrete-time martingales; the optional stopping theorem with applications
- Using conditional expectations