Control of Stochastic Systems

 

ECE 555 Spring 2011 - Tuesday & Thursday 3:30 - 4:50 241 EL

This course introduces both the analytical and computational aspects of stochastic control and performance evaluation, with applications to engineering, computer science, economics, and management science. Topics include Markov models and stochastic stability, development of control laws by dynamic programming, complete and partial information, Kalman filtering, and an introduction to machine learning. It is intended for graduate students who have some background in control and stochastic processes. Experience with Matlab is also desirable.

Course Outline: html pdf

Prerequisites

Some exposure to control and optimization at the level of ECE 415, and exposure to stochastic processes at the level of ECE 434 or MATH 366 (or consent of instructor.)

Grading

Resources

The following texts are on reserve:

  • P. R Kumar, Stochastic systems: Estimation, identification, and adaptive control
  • Torsten Soderstrom, Discrete-time stochastic systems: estimation and control
   

Topics

1. Markov Models

  • Overview and examples.
  • Linear and non-linear models.
  • Representations of pi and value functions.
  • Lyapunov Theory.

2. Optimal Control

  • Controlled Markov chain models.
  • Markov and stationary policies.
  • Numerical techniques: Policy and value iteration, LP approaches.
  • Partial information

3. Linear Theory

  • Linear Gaussian systems.
  • Optimal linear-quadradic and minimum-variance control.
  • Partial information and the Kalman filter.

4. Adaptation and Learning.

  • Stochastic approximation.
  • Approximate dynamic programming.
  • Introduction to adaptation and machine learning.

spm