# American Institute of Mathematical Sciences

• Previous Article
A sparse Markov chain approximation of LQ-type stochastic control problems
• MCRF Home
• This Issue
• Next Article
Determining the waveguide conductivity in a hyperbolic equation from a single measurement on the lateral boundary
2016, 6(3): 391-406. doi: 10.3934/mcrf.2016008

## On the convergence of the Sakawa-Shindo algorithm in stochastic control

 1 INRIA-Saclay and Centre de Mathématiques Appliquées, Ecole Polytechnique and Laboratoire de Finance des Marchés d'Énergie, 91128 Palaiseau, France 2 CIFASIS - Centro Internacional Franco Argentino, de Ciencias de la Información y de Sistemas, CONICET - UNR - AMU, S2000EZP Rosario, Argentina 3 Institut de recherche XLIM-DMI, UMR-CNRS 7252, Faculté des sciences et techniques, Université de Limoges, 87060 Limoges, France

Received  May 2015 Revised  August 2015 Published  August 2016

We analyze an algorithm for solving stochastic control problems, based on Pontryagin's maximum principle, due to Sakawa and Shindo in the deterministic case and extended to the stochastic setting by Mazliak. We assume that either the volatility is an affine function of the state, or the dynamics are linear. We obtain a monotone decrease of the cost functions as well as, in the convex case, the fact that the sequence of controls is minimizing, and converges to an optimal solution if it is bounded. In a specific case we interpret the algorithm as the gradient plus projection method and obtain a linear convergence rate to the solution.
Citation: J. Frédéric Bonnans, Justina Gianatti, Francisco J. Silva. On the convergence of the Sakawa-Shindo algorithm in stochastic control. Mathematical Control & Related Fields, 2016, 6 (3) : 391-406. doi: 10.3934/mcrf.2016008
##### References:
 [1] J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective,, ESAIM: COCV, (). doi: 10.1051/cocv/2015039. [2] A. Bensoussan, Lectures on Stochastic Control,, Lectures notes in Maths. Vol. 972, (1982). [3] A. Bensoussan, Stochastic maximum principle for distributed parameter systems,, J. Franklin Inst., 315 (1983), 387. doi: 10.1016/0016-0032(83)90059-5. [4] J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients,, SIAM J. Control Optimization, 14 (1976), 419. doi: 10.1137/0314028. [5] J.-M. Bismut, Conjugate convex functions in optimal stochastic control,, J. Math. Anal. Appl., 44 (1973), 384. doi: 10.1016/0022-247X(73)90066-8. [6] J.-M. Bismut, An introductory approach to duality in optimal stochastic control,, SIAM Rev., 20 (1978), 62. doi: 10.1137/1020004. [7] J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle,, SIAM J. Control Optim., 24 (1986), 579. doi: 10.1137/0324034. [8] J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems,, Appl. Math. Optim., 65 (2012), 403. doi: 10.1007/s00245-012-9162-4. [9] A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients,, SIAM J. Control Optim., 33 (1995), 590. doi: 10.1137/S0363012992240722. [10] A. Goldstein, Convex programming in Hilbert space,, Bull. Amer. Math. Soc., 70 (1964), 709. doi: 10.1090/S0002-9904-1964-11178-2. [11] U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work,, SIAM Rev., 23 (1981), 292. doi: 10.1137/1023062. [12] H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems,, SIAM J. Control, 10 (1972), 550. doi: 10.1137/0310041. [13] H. J. Kushner, On the stochastic maximum principle: Fixed time of control,, J. Math. Anal. Appl., 11 (1965), 78. doi: 10.1016/0022-247X(65)90070-3. [14] H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems,, J. Math. Anal. Appl., 8 (1964), 287. doi: 10.1016/0022-247X(64)90070-8. [15] L. Mazliak, An algorithm for solving a stochastic control problem,, Stochastic analysis and applications, 14 (1996), 513. doi: 10.1080/07362999608809455. [16] L. Mou and J. Yong, A variational formula for stochastic controls and some applications,, Pure Appl. Math. Q., 3 (2007), 539. doi: 10.4310/PAMQ.2007.v3.n2.a7. [17] S. G. Peng, A general stochastic maximum principle for optimal control problems,, SIAM J. Control Optim., 28 (1990), 966. doi: 10.1137/0328054. [18] L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes,, Gordon & Breach Science Publishers, (1986). [19] Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control,, IEEE Trans. Automat. Control, 25 (1980), 1149. doi: 10.1109/TAC.1980.1102517. [20] J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations,, Springer-Verlag, (1999). doi: 10.1007/978-1-4612-1466-3.

show all references

##### References:
 [1] J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective,, ESAIM: COCV, (). doi: 10.1051/cocv/2015039. [2] A. Bensoussan, Lectures on Stochastic Control,, Lectures notes in Maths. Vol. 972, (1982). [3] A. Bensoussan, Stochastic maximum principle for distributed parameter systems,, J. Franklin Inst., 315 (1983), 387. doi: 10.1016/0016-0032(83)90059-5. [4] J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients,, SIAM J. Control Optimization, 14 (1976), 419. doi: 10.1137/0314028. [5] J.-M. Bismut, Conjugate convex functions in optimal stochastic control,, J. Math. Anal. Appl., 44 (1973), 384. doi: 10.1016/0022-247X(73)90066-8. [6] J.-M. Bismut, An introductory approach to duality in optimal stochastic control,, SIAM Rev., 20 (1978), 62. doi: 10.1137/1020004. [7] J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle,, SIAM J. Control Optim., 24 (1986), 579. doi: 10.1137/0324034. [8] J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems,, Appl. Math. Optim., 65 (2012), 403. doi: 10.1007/s00245-012-9162-4. [9] A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients,, SIAM J. Control Optim., 33 (1995), 590. doi: 10.1137/S0363012992240722. [10] A. Goldstein, Convex programming in Hilbert space,, Bull. Amer. Math. Soc., 70 (1964), 709. doi: 10.1090/S0002-9904-1964-11178-2. [11] U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work,, SIAM Rev., 23 (1981), 292. doi: 10.1137/1023062. [12] H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems,, SIAM J. Control, 10 (1972), 550. doi: 10.1137/0310041. [13] H. J. Kushner, On the stochastic maximum principle: Fixed time of control,, J. Math. Anal. Appl., 11 (1965), 78. doi: 10.1016/0022-247X(65)90070-3. [14] H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems,, J. Math. Anal. Appl., 8 (1964), 287. doi: 10.1016/0022-247X(64)90070-8. [15] L. Mazliak, An algorithm for solving a stochastic control problem,, Stochastic analysis and applications, 14 (1996), 513. doi: 10.1080/07362999608809455. [16] L. Mou and J. Yong, A variational formula for stochastic controls and some applications,, Pure Appl. Math. Q., 3 (2007), 539. doi: 10.4310/PAMQ.2007.v3.n2.a7. [17] S. G. Peng, A general stochastic maximum principle for optimal control problems,, SIAM J. Control Optim., 28 (1990), 966. doi: 10.1137/0328054. [18] L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes,, Gordon & Breach Science Publishers, (1986). [19] Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control,, IEEE Trans. Automat. Control, 25 (1980), 1149. doi: 10.1109/TAC.1980.1102517. [20] J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations,, Springer-Verlag, (1999). doi: 10.1007/978-1-4612-1466-3.
 [1] Huaiqiang Yu, Bin Liu. Pontryagin's principle for local solutions of optimal control governed by the 2D Navier-Stokes equations with mixed control-state constraints. Mathematical Control & Related Fields, 2012, 2 (1) : 61-80. doi: 10.3934/mcrf.2012.2.61 [2] Feng Bao, Yanzhao Cao, Weidong Zhao. A first order semi-discrete algorithm for backward doubly stochastic differential equations. Discrete & Continuous Dynamical Systems - B, 2015, 20 (5) : 1297-1313. doi: 10.3934/dcdsb.2015.20.1297 [3] Hancheng Guo, Jie Xiong. A second-order stochastic maximum principle for generalized mean-field singular control problem. Mathematical Control & Related Fields, 2018, 8 (2) : 451-473. doi: 10.3934/mcrf.2018018 [4] Shanjian Tang. A second-order maximum principle for singular optimal stochastic controls. Discrete & Continuous Dynamical Systems - B, 2010, 14 (4) : 1581-1599. doi: 10.3934/dcdsb.2010.14.1581 [5] Christine Burggraf, Wilfried Grecksch, Thomas Glauben. Stochastic control of individual's health investments. Conference Publications, 2015, 2015 (special) : 159-168. doi: 10.3934/proc.2015.0159 [6] Omid S. Fard, Javad Soolaki, Delfim F. M. Torres. A necessary condition of Pontryagin type for fuzzy fractional optimal control problems. Discrete & Continuous Dynamical Systems - S, 2018, 11 (1) : 59-76. doi: 10.3934/dcdss.2018004 [7] Hans Josef Pesch. Carathéodory's royal road of the calculus of variations: Missed exits to the maximum principle of optimal control theory. Numerical Algebra, Control & Optimization, 2013, 3 (1) : 161-173. doi: 10.3934/naco.2013.3.161 [8] Vladimir Kazakov. Sampling - reconstruction procedure with jitter of markov continuous processes formed by stochastic differential equations of the first order. Conference Publications, 2009, 2009 (Special) : 433-441. doi: 10.3934/proc.2009.2009.433 [9] Y. Gong, X. Xiang. A class of optimal control problems of systems governed by the first order linear dynamic equations on time scales. Journal of Industrial & Management Optimization, 2009, 5 (1) : 1-10. doi: 10.3934/jimo.2009.5.1 [10] Yanning Li, Edward Canepa, Christian Claudel. Efficient robust control of first order scalar conservation laws using semi-analytical solutions. Discrete & Continuous Dynamical Systems - S, 2014, 7 (3) : 525-542. doi: 10.3934/dcdss.2014.7.525 [11] Carlo Orrieri. A stochastic maximum principle with dissipativity conditions. Discrete & Continuous Dynamical Systems - A, 2015, 35 (11) : 5499-5519. doi: 10.3934/dcds.2015.35.5499 [12] J. R. Ward. Periodic solutions of first order systems. Discrete & Continuous Dynamical Systems - A, 2013, 33 (1) : 381-389. doi: 10.3934/dcds.2013.33.381 [13] Sie Long Kek, Mohd Ismail Abd Aziz, Kok Lay Teo, Rohanin Ahmad. An iterative algorithm based on model-reality differences for discrete-time nonlinear stochastic optimal control problems. Numerical Algebra, Control & Optimization, 2013, 3 (1) : 109-125. doi: 10.3934/naco.2013.3.109 [14] Torsten Lindström. Discrete models and Fisher's maximum principle in ecology. Conference Publications, 2003, 2003 (Special) : 571-579. doi: 10.3934/proc.2003.2003.571 [15] Yueling Li, Yingchao Xie, Xicheng Zhang. Large deviation principle for stochastic heat equation with memory. Discrete & Continuous Dynamical Systems - A, 2015, 35 (11) : 5221-5237. doi: 10.3934/dcds.2015.35.5221 [16] Mohammed Al Horani, Angelo Favini. First-order inverse evolution equations. Evolution Equations & Control Theory, 2014, 3 (3) : 355-361. doi: 10.3934/eect.2014.3.355 [17] Dirk Hartmann, Isabella von Sivers. Structured first order conservation models for pedestrian dynamics. Networks & Heterogeneous Media, 2013, 8 (4) : 985-1007. doi: 10.3934/nhm.2013.8.985 [18] Alexander Veretennikov. On large deviations in the averaging principle for SDE's with a "full dependence,'' revisited. Discrete & Continuous Dynamical Systems - B, 2013, 18 (2) : 523-549. doi: 10.3934/dcdsb.2013.18.523 [19] Takayoshi Ogawa, Kento Seraku. Logarithmic Sobolev and Shannon's inequalities and an application to the uncertainty principle. Communications on Pure & Applied Analysis, 2018, 17 (4) : 1651-1669. doi: 10.3934/cpaa.2018079 [20] Yong Xu, Rong Guo, Di Liu, Huiqing Zhang, Jinqiao Duan. Stochastic averaging principle for dynamical systems with fractional Brownian motion. Discrete & Continuous Dynamical Systems - B, 2014, 19 (4) : 1197-1212. doi: 10.3934/dcdsb.2014.19.1197

2017 Impact Factor: 0.542