On the Connection between the Hamilton-Jacobi-Bellman and the Fokker-Planck Control Frameworks

Mario Annunziato, Alfio Borzì, Fabio Nobile, Raul Tempone

Dipartimento di Matematica, Università degli Studi di Salerno, Fisciano, Italy.
école Polytechnique Fédérale de Lausanne, Lausanne, Switzerland.
Institut für Mathematik, Universit?t Würzburg, Würzburg, Germany.
King Abdullah University of Science and Technology, Thuwal, Kingdom of Saudi Arabia.

DOI: 10.4236/am.2014.516239                

Abstract

In the framework of stochastic processes, the connection between the dynamic programming scheme given by the Hamilton-Jacobi-Bellman equation and a recently proposed control approach based on the Fokker-Planck equation is discussed. Under appropriate assumptions it is shown that the two strategies are equivalent in the case of expected cost functionals, while the Fokker-Planck formalism allows considering a larger classof objectives. To illustratethe connection between the two control strategies, the cases of an Itō stochastic process and of a piecewise-deterministic process are considered.

Keywords

Hamilton-Jacobi-Bellman Equation, Fokker-Planck Equation, Optimal Control Theory, Stochastic Differential Equations, Hybrid Systems

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	Ghman, I.I. and Skorohod, A.V. (1972) Stochastic Differential Equations. Springer-Verlag, New York. http://dx.doi.org/10.1007/978-3-642-88264-7
[2]	Bertsekas, D. (2007) Dynamic Programming and Optimal Control, Vols. I and II. Athena Scientific.
[3]	Fleming, W.H. and Soner, H.M. (2006) Controlled Markov Processes and Viscosity Solutions. Springer-Verlag, Berlin.
[4]	Annunziato, M. and Borzì, A. (2010) Optimal Control of Probability Density Func-tions of Stochastic Processes. Mathematical Modelling and Analysis, 15, 393-407. http://dx.doi.org/10.3846/1392-6292.2010.15.393-407
[5]	Annunziato, M. and Borzì, A. (2013) A Fokker-Planck Control Framework for Multidimensional Stochastic Processes. Journal of Computational and Applied Mathematics, 237, 487-507. http://dx.doi.org/10.1016/j.cam.2012.06.019
[6]	Cox, D.R. and Miller, H.D. (2001) The Theory of Stochastic Processes. Chapman & Hall/CRC, London.
[7]	Fleming, W.H. and Rishel, R.W. (1975) Deterministic and Stochastic Optimal Control. Springer-Verlag, Berlin. http://dx.doi.org/10.1007/978-1-4612-6380-7
[8]	Borkar, V.S. (2005) Controlled Diffusion Processes. Probability Surveys, 2, 213-244. http://dx.doi.org/10.1214/154957805100000131
[9]	Falcone, M. (2008) Numerical Solution of Dynamic Programming Equations. In: Bardi, M. and Dolcetta, I.C., Eds., Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations, Birkhauser.
[10]	Kushner, H.J. (1990) Numerical Methods for Stochastic Control Problems in Continuous Time. SIAM Journal on Control and Optimization, 28, 999-1048. http://dx.doi.org/10.1137/0328056
[11]	Forbes, M.G., Guay, M. and Forbes, J.F. (2004) Control Design for First-Order Processes: Shaping the Probability Density of the Process State. Journal of Process Control, 14, 399-410. http://dx.doi.org/10.1016/j.jprocont.2003.07.002
[12]	Jumarie, G. (1992) Tracking Control of Nonlinear Stochastic Systems by Using Path Cross-Entropy and Fokker-Planck Equation. International Journal of Systems Science, 23, 1101-1114. http://dx.doi.org/10.1080/00207729208949368
[13]	Kárny, M. (1996) Towards Fully Probabilistic Control Design. Automatica, 32, 1719-1722. http://dx.doi.org/10.1016/S0005-1098(96)80009-4
[14]	Wang, H. (1999) Robust Control of the Output Probability Density Functions for Multivariable Stochastic Systems with Guaranteed Stability. IEEE Transactions on Automatic Control, 44, 2103-2107. http://dx.doi.org/10.1109/9.802925
[15]	Primak, S., Kontorovich, V. and Lyandres, V. (2004) Stochastic Methods and Their Applications to Communications. John Wiley & Sons, Chichester.
[16]	Risken, R. (1996) The Fokker-Planck Equation: Methods of Solution and Applications. Springer, Berlin.
[17]	Lions, J.L. (1983) On the Hamilton-Jacobi-Bellman Equations. Acta Applicandae Mathematica, 1, 17-41. http://dx.doi.org/10.1007/BF02433840
[18]	Crandall, M., Ishii, H. and Lions, P.L. (1992) User’s Guide to Viscosity Solutions of Second Order Partial Differential Equations. Bulletin of the American Mathematical Society, 27, 1-67. http://dx.doi.org/10.1090/S0273-0979-1992-00266-5
[19]	Le Bris, C. and Lions, P.L. (2008) Existence and Uniqueness of Solutions to Fokker-Planck Type Equations with Irregular Coefficients. Communications in Partial Differential Equations, 33, 1272-1317.
[20]	Aronson, D.G. (1968) Non-Negative Solutions of Linear Parabolic Equations. Annali della Scuola Normale Superiore di Pisa. Classe di Scienze, 22, 607-694.
[21]	Bogachev, V., Da Prato, G. and Röckner, M. (2010) Existence and Uniqueness of Solutions for Fokker-Planck Equations on Hilbert Spaces. Journal of Evolution Equations, 10, 487-509. http://dx.doi.org/10.1007/s00028-010-0058-y
[22]	Lions, J.L. (1971) Optimal Control of Systems Governed by Partial Differential Equations. Springer, Berlin.
[23]	Lachapelle, A., Salomon, J. and Turinici, G. (2010) Computation of Mean Field Equilibria in Economics. Mathematical Models and Methods in Applied Sciences, 20, 567-588. http://dx.doi.org/10.1142/S0218202510004349
[24]	Carlier, G. and Salomon, J. (2008) A Monotonic Algorithm for the Optimal Control of the Fokker-Planck Equation. IEEE Conference on Decision and Control, CDC 2008, Cancun, 9-11 December 2008, 269-273.
[25]	Davis, M.H.A. (1984) Piecewise-Deterministic Markov Processes: A General Class of Non-Diffusion Stochastic Models. Journal of the Royal Statistical Society. Series B (Methodological), 46, 353-388.
[26]	Annunziato, M. (2008) Analysis of Upwind Method for Piecewise Deterministic Markov Processes. Compu-tational Methods in Applied Mathematics, 8, 3-20.
[27]	Capuzzo Dolcetta, I. and Evans, L.C. (1984) Optimal Switching for Ordinary Differential Equations. SIAM Journal on Control and Optimization, 22, 143-161. http://dx.doi.org/10.1137/0322011
[28]	Annunziato, M. and Borzì, A. (2014) Optimal Control of a Class of Piecewise Deterministic Processes. European Journal of Applied Mathematics, 25, 1-25. http://dx.doi.org/10.1017/S0956792513000259
[29]	Moresino, F., Pourtallier, O. and Tidball, M. (1988) Using Viscosity Solution for Approximations in Piecewise Deterministic Control Systems. Report RR-3687-HAL-INRIA, Sophia-Antipolis.