Tutorial on Markov Chains

---

References

[1] S. Balaji and S. P. Meyn. Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stoch. Proc. Applns., 90(1):123-144, 2000.

[2] V. S. Borkar and S. P. Meyn. The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim., 38(2):447-469, 2000. (also presented at the IEEE CDC, December, 1998).

[3] V. S. Borkar and S. P. Meyn. Risk-sensitive optimal control for Markov decision processes with monotone cost. Math. Oper. Res., 27(1):192-209, 2002.

[4] W. Chen, D. Huang, A. Kulkarni, J. Unnikrishnan, Q. Zhu, P. Mehta, S. Meyn, and A. Wierman. Approximate dynamic programming using fluid and diffusion approximations with applications to power management. Accepted for inclusion in the 48th IEEE Conference on Decision and Control, December 16-18 2009.

[5] I.-K. Cho and S. P. Meyn. Efficiency and marginal cost pricing in dynamic competitive markets. To appear in J. Theo. Economics, 2006.

[6] In-Koo Cho and S. Meyn. Efficiency and marginal cost pricing in dynamic competitive markets with friction. Decision and Control, 2007 46th IEEE Conference on, pages 771-778, Dec. 2007.

[7] W. Doeblin. Eléments d'une théorie générale des chaînes simples constantes de Markov. Annales Scientifiques de l'Ecole Normale Supérieure, 57(III):61-111, 1940.

[8] S. G. Henderson, S. P. Meyn, and V. B. Tadi\'c. Performance evaluation and policy selection in multiclass networks. Discrete Event Dynamic Systems: Theory and Applications, 13(1-2):149-189, 2003. Special issue on learning, optimization and decision making (invited).

[9] W. Huisinga, S. Meyn, and C. Schütte. Phase transitions and metastability in Markovian and molecular systems. Ann. Appl. Probab., 14(1):419-458, 2004. Presented at the 11TH INFORMS Applied Probability Society Conference, NYC, 2001.

[10] N. Jain and B. Jamison. Contributions to Doeblin's theory of Markov processes. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 8:19-40, 1967.

[11] I. Kontoyiannis and S. P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Ann. Appl. Probab., 13:304-362, 2003. Presented at the INFORMS Applied Probability Conference, NYC, July, 2001.

[12] I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics and the spectral theory of multiplicatively regular Markov processes. Electron. J. Probab., 10(3):61-123 (electronic), 2005.

[13] I. Kontoyiannis and S. P. Meyn. Computable exponential bounds for screened estimation and simulation. Ann. Appl. Probab., 18(4):1491-1518, 2008.

[14] A. N. Langville and C. D. Meyer. A survey of eigenvector methods for Web information retrieval. SIAM Rev., 47(1):135-161 (electronic), 2005.

[15] Amy N. Langville and Carl D. Meyer. Google's PageRank and Beyond: The Science of Search Engine Rankings. Princeton University Press, Princeton, NJ, USA, 2006.

[16] P. Mehta and S. Meyn. Q-learning and Pontryagin's Minimum Principle. To appear in Proceedings of the 48th IEEE Conference on Decision and Control, December 16-18 2009.

[17] S. P. Meyn. Control Techniques for Complex Networks. Cambridge University Press, Cambridge, 2007.

[18] S. P. Meyn and R. L. Tweedie. Markov chains and stochastic stability. Cambridge University Press, Cambridge, second edition, 2009. Published in the Cambridge Mathematical Library. 1993 edition online: http://black.csl.uiuc.edu/~ meyn/pages/book.html.

[19] C.C. Moallemi, S. Kumar, and B. Van Roy. Approximate and data-driven dynamic programming for queueing networks, 2008.

[20] E. Nummelin. General Irreducible Markov Chains and Nonnegative Operators. Cambridge University Press, Cambridge, 1984.

[21] S. Orey. Limit Theorems for Markov Chain Transition Probabilities. Van Nostrand Reinhold, London, 1971.

[22] J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Trans. Automat. Control, 42(5):674-690, 1997.

 

Control Techniques for Complex Networks

Markov Chains & Stochastic Stability