Relationship between Maximum Principle and Dynamic Programming in Stochastic Differential Games and Applications

This paper is concerned with the relationship between maximum principle and dynamic programming in zero-sum stochastic differential games. Under the assumption that the value function is smooth enough, relations among the adjoint processes, the generalized Hamiltonian function and the value function are given. A portfolio optimization problem under model uncertainty in the financial market is discussed to show the applications of our result.


Introduction
Game theory has been an active area of research and a useful tool in many applications, particularly in biology and economic.Among others, there are two main approaches to study differential game problems.One approach is Bellman's dynamic programming, which relates the saddle points or Nash equilibrium points to some partial differential Equations (PDEs) which are known as the Hamilton-Jacobi-Bellman-Isaacs (HJBI) Equations (see Elliott [1], Fleming and Souganidis [2], Buckdahn et al. [3], Mataramvura and Oksendal [4]).The other approach is Pontryagin's maximum principle, which finds solutions to the differential games via some Hamiltonian function and adjoint processes (see Tang and Li [5], An and Oksendal [6]).
Hence, a natural question arises: Are there any relations between these two methods?For stochastic control problems, such a topic has been discussed by many authors (see Bensoussan [7], Zhou [8], Yong and Zhou [9], Framstad et al. [10], Shi and Wu [11], Donnelly [12], etc.)However, to the best of our knowledge, the study on the relationship between maximum principle and dynamic programming for stochastic differential games is quite lacking in literature.
In this paper, we consider one kind of zero-sum stochastic differential game problem within the frame work of Mataramvura and Oksendal [4] and An and Oksendal [6].However, we don't consider jumps.This more general case will appear in our forthcoming paper.For our problem in this paper, [4] related its saddle point to some HJBI Equation and obtained the stochastic verification theorem.[6] proves both sufficient and necessary maximum principles, which state some conditions of optimality via the Hamiltonian function and adjoint Equation.
The main contribution of this paper is that we connect the maximum principle of [6] with the dynamic programming of [4], and obtain relations among the adjoint processes, the generalized Hamiltonian function and the value function under the assumption that the value function is enough smooth.As applications, we discuss a portfolio optimization problem under model uncertainty in the financial market.In this problem, the optimal portfolio strategies for the trader (representative agent) and the "worst case scenarios" (see Peskir and Shorish [13], Korn and Menkens [14]) for the market, derived from both maximum principle and dynamic programming approaches independently, coincide.The relation that we obtained in our main result is illustrated.
The rest of this paper is organized as follows.In Section 2, we state our zero-sum stochastic differential game problem.Under suitable assumptions, we reformulate the sufficient maximum principle of [6] by adjoint Equation and Hamiltonian function, and the stochastic verification theorem [4] by HJBI Equation.In Section 3, we prove the relationship between maximum principle and dynamic programming for our zero-sum stochastic differential game problem, under the assumption that the value function is smooth enough.A portfolio optimization problem under model uncertainty in the financial market is discussed in Section 4, to show the applications of our result.
Notations: throughout this paper, we denote by n R the space of n-dimensional Euclidean space, by n d R  the space of n d  matrices, by n S the space of n n  symmetric matrices. ,and |.| denote the scalar product and norm in the Euclidean space, respectively. appearing in the superscripts denotes the transpose of a matrix.

Problem Statement and Preliminaries
Let 0 T  be given, suppose that the dynamics of a stochastic system is described by a stochastic differential Equation (SDE) on a complete probability space   , , we suppose the filtra- where  contains all P-null sets in  and For any . ., .
as an admissible pair.
We make the following assumption.
be continuous functions.For any     , 0, , we define the following performance functional Now suppose that the control process has the form where  and π are valued in two sets 1 K and 2 K , respectively.We let   for given    , 0, Such a control process (pair)   . .
The intuitive idea is that there are two players, I and II.
Player I controls  and Player II controls  .The actions of the two players are antagonistic, which means that between Players I and II there is a payoff   , ; ,π J t x  which is a cost for player I and a reward for Player II.
We now define the Hamiltonian function

H s x p q b s x p tr
In addition, we need the following assumption.
(H2) B, σ, f are continuously differentiable in   , ,π x  and g is continuously differentiable in x .Moreover, , x x b  are bounded and there exists a constant The adjoint Equation in the unknown s  -adapted processes , .
, under (H2), we know that BSDE (6) admits a unique s  -adapted solution . We now can state the following sufficient maximum principle which is Corollary 2.1 in An and Oksendal [6].
, respectively.Suppose that there exists a solution       ˆ. , .p q to the corresponding adjoint Equation (6).Moreover, suppose that for all , the following minimum/maximum conditions hold: 1) Suppose that for all and 2) Suppose that for all 3) If both Cases (1) and ( 2) hold (which implies, in particular, that g is an affine function), then  is an optimal control (saddle point) and Next, when the control process where .,. 0, ; The following result is a stochastic verification theorem of optimality, which is an immediate corollary of Theorem 3.2 in Mataramvura and Oksendal [4].Lemma 2.2 Let (H1), (H2) hold and     , 0, and a Markovian control process  is an optimal Markovian control.

Main Result
In this section, we investigate the relationship between maximum principle and dynamic programming for our zero-sum stochastic differential game problem.The main contribution is that we find the connection between the value function V , the adjoint processes , p q and the following generalized Hamiltonian function Our main result is the following.
 is an optimal Markovian control, and is the corresponding optimal state.Suppose that the value function .,. 0, ;   , , , .., s t T P a s   π , , , .., s t T P a s     and (3)   , , . .
and sx V is also continuous.For any . , .p q solves the adjoint Equation (6).Proof.(13), (15) can be obtained from the HJBI Equation (10), by the definitions of the generator ,π A  in (9) and the generalized Hamiltonian function G in (12).
We proceed to prove the second part.If and sx V is also continuous, then from (15), we have x Hence, by the uniqueness of the solutions to (6), we obtain (16).The proof is complete.□

Applications
In this section, we will discuss a portfolio optimization problem under model uncertainty in the financial market, where the problem is put into the framework of a zero-sum stochastic differential game.The optimal portfolio strategies for the investor and the "worst case scenarios" for the market, derived both from maximum principle and dynamic programming approaches independently, coincide.The relation that we obtained in our main result Theorem 3.1 is illustrated.
Suppose that the investors have two kinds of securities in the market for possible investment choice: (1) a risk-free security (e.g. a bond), where the price   0 S t at time t is given by Let   π t be a portfolio for the investors in the market, which is the proportion of the wealth invested in the risky security at time t .
Given the initial wealth   π 0 0 Y y   , we assume that   π . is self-financing, which means that the corresponding wealth process The family of admissible portfolios is denoted by  .Now, we introduce a family Q of measures  Q parameterized by processes where We assume that is an equivalent local martingale measure.But here we do not assume that (22) holds.
All  satisfying (20) and ( 21) are called admissible controls of the market.The family of admissible controls where     is a given utility function, which is increasing, concave and twice continuously differentiable on   0,  .We can consider this problem as a zero-sum stochastic differential game between the agent and the market.The agent wants to maximize his/her expected discounted utility over all portfolios π and the market wants to minimize the maximal expected utility of the agent over all "scenarios", represented by all probability measures   To put the problem in a Markovian framework so that we can apply the dynamic programming, define where denote the initial time of the investment, and which is a 2-dimensional process combined the Radon-Nikodym process with the wealth process

Maximum Principle Approach
To solve our problem by maximum principle approach, that is, applying Lemma 2.1, we write down the Hamiltonian function (5) as H s x x p p q q x q x s s s p The adjoint Equations ( 6) are be the corresponding state process, with corresponding solution ˆˆˆˆˆ. ., ., . ., .p p p q q q   to the adjoint Equations.By (7) in Lemma 2.1, we first maximize the Hamiltonian function H over all π   .This gives the following condition for a maximum point π : Then, we minimize H over all    , and get the Following condition for a minimum point  : We try a process with a deterministic differential function f .Differentiating (31) and using (17), we get Comparing this with the adjoint Equation ( 27) by equating the s W s coefficients respectively, we get and Substituting (33) into (30), we have Differentiating (37) and using (17), (36), we get Comparing this with the adjoint Equation (28), we have and Substituting (39) into (29), we have From (40), we get Let 0 t  , we have proved the following theorem.Theorem 4.1 The optimal portfolio strategy π   for the agent is The optimal "scenario", that is, the optimal probability measure for the market is to choose    such that That is, the market minimize the maximal expected utility of the agent by choosing a scenario (represented by a probability law ), which is an equivalent martingale measure for the market (see ( 22)).
In this case, the optimal portfolio strategy for the agent is to place all the money in the risk-free security, i.e., to choose   π 0 t  for all t .This result is the counterpart of Theorem 4.1 in An and Oksendal [6] without jumps with complete information.

Dynamic Programming Approach
To solve our problem by dynamic programming approach, that is, applying Lemma 2.2, we write down the generator ,π A  of the diffusion system (25) as .,.,.0 Applying to our setting, the HJBI Equation ( 10) gets the following form We try a V of the form for some deterministic function f with   1 f T  .Note that conditions (i), (ii), (iii) in (47) can be rewritten as , , A V t x x  over all π gives the following first-order condition for a maximum point π : We then minimize , , A V t x x  over all  and get the following first-order condition for a minimum point  : From (52) we conclude that which substituted into (51) gives And the HJBI Equation (iii) , , 0 A V t x x   states that with these values of π and  , we should have (i.e., to put all the wealth in the risk-free security) and the optimal "scenario" for the market is to choose    such that (i.e., the market chooses an equivalent martingale measure or risk-free measure  Q for the market).This result is the counterpart of Theorem 2.2 in Oksendal and Sulem [15] without jumps.

Conclusions and Future Works
In this paper, we have discussed the relationship between maximum principle and dynamic programming in zerosum stochastic differential games.Under the assumption that the value function is smooth, relations among the adjoint processes, the generalized Hamiltonian function and the value function are given.A portfolio optimization problem under model uncertainty in the financial market is discussed to show the applications of our result.
Many interesting and challenging problems remain open.For example, what is the relationship between maximum principle and dynamic programming for stochastic differential games without the illusory assumption that the value function is smooth?This problem may be solved in the framework of viscosity solution theory (Yong and Zhou [9]).Another topic is that we can continue to investigate the relationship between maximum principle and dynamic programming for forward and backward stochastic differential games, and then study its applications to stochastic recursive utility optimization problem under model uncertainty (Oksendal and Sulem [16]).Such topics will be studied in our future work.