ECE 555 Control of Stochastic Systems Spring 2011
Tuesday & Thursday 241 EL, 3:30­4:50 pm

Prof. Sean Meyn

This course introduces both the analytical and computational aspects of stochas-
tic control and performance evaluation, with applications to engineering, computer
science, economics, and management science. Topics include Markov models and
stochastic stability, development of control laws by dynamic programming, com-
plete and partial information, Kalman filtering, and an introduction to machine
learning. It is intended for graduate students who have some background in control and
stochastic processes. Experience with Matlab is also desirable.

References: The following textbooks are on reserve in the Engineering Library.

• P. R Kumar, Stochastic systems: Estimation, identification, and adaptive control.
• Torsten Soderstrom, Discrete-time stochastic systems: estimation and control.
  

The following are available free on-line (send your thanks to Cambridge University Press):

• S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability.
  https://netfiles.uiuc.edu/meyn/www/spm files/book.html
• S. P. Meyn, Control Techniques for Complex networks.
  https://netfiles.uiuc.edu/meyn/www/spm files/CTCN/CTCN.html
  

Office hours Wednesdays from 4:00-5:00p.m. (sometimes extended to 5:30) -- CSL 154.
I can be reached for questions by electronic mail at meyn@illinois.edu.

Exams, homework, and grading
Homework problems will be assigned on a bi-weekly basis, to be handed in at the beginning of class on the date due. They will be graded and returned the following week. Late homework cannot be accepted.

There will be two evening midterm exams, March 16 and April 20, from 7:00 - 8:30 p.m. in
our classroom, 241 EL. You will be allowed one sheet of notes (8 12 × 11; both sides) in the
first exam, and two in the second. Otherwise, the exams are closed-book and closed-notes.

Tentative grading scheme: Homework problems will count 20%, the midterm exam 25%, and
the final will count 30% towards the final grade in the course.

Course Outline

I. Markov Models

1) Overview and examples.
2) Linear and non-linear models.
3) Representations of and value functions.
4) Lyapunov theory.

II. Optimal Control

1) Controlled Markov chain models.
2) Markov and stationary policies.
3) Numerical techniques: Policy and value iteration, LP methods.
4) Partial information

First Midterm Exam, (7:00-8:30 p.m., Wednesday, March 16)

Spring Break: March 21­25.

III. Linear Theory

1) Linear Gaussian systems.
2) Optimal linear-quadradic and minimum-variance control.
3) Partial information and the Kalman filter.

Second Midterm Exam, (7:00-8:30 p.m., Wednesday, April 20)

IV. Adaptation and Learning

1) Stochastic approximation.
2) Approximate dynamic programming.
3) Introduction to adaptation and machine learning.

Final Exam (1:30-4:30 p.m., Friday, May 13)