Tutorial on Markov Chains
Sean Meyn, University of Illinois and the Coordinated Science Laboratory
Lectures presented at UCSB the first week of 2010:
| Monday, 01/04/10 2:00 - 3:30PM | An Introduction to Markov Models | |
| Tuesday, 01/05/10 9:30 - 11:30AM | Stochastic Stability and Dynamic Programming | |
| Thursday, 01/07/10 9:30 - 11:00AM | Approximate Dynamic Programming | |
Friday, 01/08/10 CCDC Winter 2010 Seminars 3:00 - 4:00PM |
CCDC Winter 2010 Seminar: Spectral Theory and Model Reduction for Markov Models |
A Markov model is a nonlinear state space model subject to stochastic disturbances. It would appear that the introduction of noise would make these models much more complex than their deterministic analogs. In fact, the opposite is frequently true: By considering the evolution of the underlying distributions, rather than the state itself, a linear system is obtained. This is the basis of a development of Markov models that parallels the theory of linear state space models, with or without control. In particular, spectral theory and Lyapunov theory play a fundamental role in analysis and design.
These lectures are designed for an audience with an understanding of stochastic processes (no need for measure theory), and exposure to deterministic state space control at the first-year graduate level. Examples will be used throughout to illustrate the theory.
More information:
- Overview
- Abstracts and Outline
- References
- CTCN Appendix on Markov Chains
- CTCN Section on TD Learning
- Handouts from ECE 555 Stochastic Control - Stochastic Approximation and Learning
See also the 2009 CDC tutorial, emphasizing continuous time models.
Control Techniques for Complex Networks |
Markov Chains & Stochastic Stability |
|
