|
Control Techniques for Complex Networks
- Chapter 1 : Introduction
- Chapter 2 : Examples
- Chapter 3 : The Single Server Queue
- Chapter 8 : Foster-Lyapunov Techniques
- Chapter 9 : Optimization
- Chapter 10 : ODE method
- Chapter 11 : Simulation & Learning
- Appendices : Markov Models - Based on Markov Chains and Stochastic Stability
Handouts on Markov models
- Week 1 - Introductions
- See also Course Notes on Optimal Control for a brief introduction to the HJB equations and Minimum Principle for deterministic systems. The Hamiltonian described in this chapter is precisely the Q-function in Q-learning for stochastic systems that we will study later.
- Week 2 - Lyapunov functions & performance bounds
- Week 3 - Representation theory & Markov Chains and Stochastic Stability
- Markov Decision Theory: PIA and VIA
- See also, Tutorial on Markov Chains, UCSB Jan 4-8, 2010
Handouts on adpaptive control (course notes from ECE 517)
Handouts on stochastic approximation
- Stochastic approximation and the ODE method
- A few words on variance reduction
- Stochastic approximation and machine learning
- More on machine learning: Value function approximation
- See also the paper, The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning, Chapter 11 of Control Techniques for Complex Networks, and Vivek Borkar's book Stochastic Approximation: A Dynamical Systems Viewpoint
- Barton & Sutton’s book, Reinforcement Learning: An Introduction, contains introductory material on Q-learning.
