Optimal Production Control of Hybrid Manufacturing/remanufacturing Failure-prone Systems under Diffusion-type Demand

The problem of production control for a hybrid manufacturing/remanufacturing system under uncertainty is analyzed. Two sources of uncertainty are considered: machines are subject to random breakdowns and repairs, and demand level is modeled as a diffusion type stochastic process. Contrary to most of studies where the demand level is considered constant and fewer results where the demand is modeled as a Poisson process with few discrete levels and exponentially distributed switching time, the demand is modeled here as a diffusion type process. In particular Wiener and Ornstein-Uhlenbeck processes for cumulative demands are analyzed. We formulate the stochastic control problem and develop optimality conditions for it in the form of Hamilton-Jacobi-Bellman (HJB) partial differential equations (PDEs). We demonstrate that HJB equations are of the second order contrary to the case of constant demand rate (corresponding to the average demand in our case), where HJB equations are linear PDEs. We apply the Kushner-type finite difference scheme and the policy improvement procedure to solve HJB equations numerically and show that the optimal production policy is of hedging-point type for both demand models we have introduced, similarly to the known case of a constant demand. Obtained results allow to compute numerically the optimal production policy in hybrid manufacturing/ remanufacturing systems taking into account the demand variability, and also show that Kushner-type discrete scheme can be successfully applied for solving underlying second order HJB equations.


Introduction
In recent years, the reverse logistics framework allowing the unified analysis of manufacturing planning and the inventory management has gained a substantial interest among the researchers working in the field. In a book [1] author described the quantitative models to represent the activities of remanufacturing and recycling in the context of reverse logistics emphasizing three issues: namely: distribution planning, inventory management and production planning. In the survey [2] authors analyzed more than sixty case studies in reverse logistics published between 1984 and 2002 and discussed network structures and activities related to the recovery of products up to the end of life. Various optimization models for supply chains with a recovery of returned products have been proposed with special attention to the production control and inventory management using both, deterministic and stochastic approaches. In the majority of previous studies discrete time (as opposed to continuous time) settings is used. In [3] authors present an effective approach to determine the discrete policy of the optimal control for a system with product recovery, taking into account the uncertainty in the demand of new and returned products. They model the demand and the return as discrete independent random variables. In [4] a new discrete stochastic inventory model for a hybrid system is proposed: new and returned products are manufactured separately, the demands are independent but production policies are synchronized. Authors of [5] develop a periodic inventory model of on a finite planning horizon with consideration of production remanufacturing and disposal activities. In [6] authors propose a model of discrete time stochastic optimization for a hybrid system taking into account, the production, subcontracting, remanufacturing of returned products, return market of poor quality products the production line and disposal activities. The Manufacturing systems subject to random breakdowns and repairs were systematically analyzed in [8][9][10] in continuous time using stochastic optimization technique. Recently in [11] this methodology has been extended to address the global performances of the manufacturing system with the supply chain in closed loop. A stochastic dynamic system consisted of two machines dedicated respectively to manufacturing and remanufacturing; the random phenomena are breakdowns and repairs of the machines, the demand of new products was considered deterministic and known, the returned product was a portion of this demand.
The constant demand is a prevailing assumption in the large body of the research devoted to stochastic continuous time optimization of production management in failure-prone systems. Some papers develop optimality conditions and use them for searching numerical solutions [12]; others present analytical solutions as the recent article [13]. In much fewer studies where the random demand is analyzed-it is most often modeled as a Poisson process. This approach allows to keep the usual framework of random discrete events changing the state of the system for both machine breakdowns and demand jumps [14]. Poisson-type demand is used more systematically in inventory optimization problems [15]. A combined model: Poisson process coupled with the diffusion process has been recently proposed in [16] for modeling the demand in inventory problem. In fact diffusion-type processes were used for modeling the demand in the classical paper [8] were optimality conditions have been obtained, however it was the only source of random behavior since the machine breakdowns were not considered.
The system considered in this paper contains reverse logistics loop with manufacturing and remanufacturing branches revisiting the model proposed in [14]. We use continuous time stochastic control approach and adopt the diffusion-type component into the demand model merging this source of random behavior with random machine breakdowns described by Poisson process as in [9,10]. As a direct consequence of an adopted demand model the optimality conditions lead to the Hamilton-Jacobi-Bellman (HJB) equation of the second order. Second order HJB is often met in option price modeling, but for stochastic control in manufacturing systems the HJB is usually of the first order [9,10,14]. Analyzing the second order HJB we use the Kushner finite difference approximations and the policy improvement algorithm [17].
The paper is structured as follows. In Section 2 we describe the model of the hybrid system consisting of 2 machines. The first machine uses primary product, and the second-returned product; both are subject to breakdowns and repairs constituting the first source of uncertainty. We describe in details our demand model using diffusion type random processes constructed as an output of shaping filter excited by the white noise. We study 2 versions of such model simple Brownian motion and first order Markovian process. Latter version seems more realistically fit the real world situations. In Section 3 we derive optimality conditions in the form of Hamilton-Jacobi-Bellman (HJB) equations which are second order partial differential equations (PDEs) for the chosen demand model. In Section 4 we describe the numerical method based on finite difference approximations and policy improvement approach following the methodology proposed in [17] and also in [12]. In Section 5 we apply the developed methodology to the manufacturing system described in Section 2, compute the optimal production policy and show that it is of classical hedging point type. In conclusion we discuss the proposed methodology and obtained results, and outline the possible directions for future works.

Model of a Hybrid Manufacturing System Suitable for Stochastic Control
We consider a hybrid manufacturing/remanufacturing system consisting of two parallel machines denoted M 1 and M 2 respectively, producing the same type of product. Stochastic phenomena are demand level and machine breakdowns/repairs. We take into account the activity of production in forward direction and the activity of reutilization of returned products in reverse logistics. The demand must be satisfied by inventory for serviceable items. This inventory will be built by the products manufactured or reused. The returned products will be in the second inventory namely recovery, they can be remanufactured, or be hold on stock for future remanufacturing. In our problem, we assume that the maximal production rates for each machine are known and the machine M 2 is producing at average supply for return rate, which is also its maximal rate. This situation is illustrated in Figure 1.
State of the machine i M with is modeled as a Markov process in continuous time with discrete state  -machine is out of order). We may the define State transition diagram is shown in Figure 2.
Hybrid system is in production while in modes 1, 2 and 3. Transition probabilities from state  to state  for machine State equations can be written in the simplified form Since the demand   d t and return   R t rates are considered as stochastic processes the more rigorous Ito form of Equations (3) will be used later. Namely let d(t) be a stationary Gaussian process with the constant mean Below we further specify   t  in one of two ways: either an increment of a standard Brownian motion, or an increment of the first order Markov process defined later using the shaping filter.
For the return (remanufacturing) rate, an assumption is made that it is proportional to the customer demand rate with r is a percentage of return.
Stochastic state differential Equations (3) can be rewritten in Ito form using notation   t t z Equations (4) will be also used in the following generic form: , , , , , , (4) is specified as an increment of the shaping filter output (Ornstein-Uhlenbeck process) is a first order Markovian, its correla- Additional insight to the proposed demand model can be given by considering the cumulative demand: Let the cost rate function to be defined as follows: , c   : inventory holding and backlog costs for manufactured product (per time unit); uct (per time unit); , r p c c : production costs for manufacturing and remanufacturing processes (per unit); c  : maintenance cost for nonoperational state of the machines: The objective is to determine the production rates in order to minimize the expected discounted cost (  is the discount rate): Defined hybrid system is said to be meeting feasibility condition if where i  et max are limiting probabilities and maximal productions rates. We recall that the vector of limiting probabilities is defined as an eigenvector of the transition matrix Q(.)

Optimality Conditions for Stochastic Control Problem
Let us define the value function is defined as a minimum (infimum) of expression (9) over all possible control inputs: According to Bellman optimality principle for cost function at t t Using Taylor expansion for the term e t   and the value function over last 3 arguments, and keeping linear terms over  and up to second order terms over x  we get: Second order terms over x  are kept for further analysis because of diffusion-type processes affecting system dynamics. One more technical step consists of computing the value function using Markov chain-type machine dynamics (2) defined through transition probabilities Merging Equations (16) and (17) Copyright © 2013 SciRes. AM Averaging over random realizations of the demand driven by the Brownian input , w  using Equations (5) and applying Ito's lemma we get: Now neglecting all terms of order higher than 1 over  ,

Numerical Method-Policy Improvement
A numerical approach proposed by Kushner in [17] and successfully used in the series of works [9,12] consists of introducing the grid in the state space   approximating the first derivatives by "up wind" finite differences, then use policy improvement-discrete analog of a gradient descent in policy (control) space. Use of "up-wind" derivatives results in conditional computations but greatly improve convergence of the numerical.

Computations of First Derivative
To describe conditional computations of the derivatives let us introduce the following notation: The first derivatives of the value function with respect to 1 x and 2 x are: 1   1  2   1  1  2  1  2  1   1  2  1  1  2  1   , ,   1  ,  ,  , ,   1 , , , , It worth emphasizing that there is no conditional computation for  all the time due to assumption described in Section 2.

Optimal Production Policy for Hybrid System-Simulation Results
The first case we have analyzed corresponds to the hybrid system with manufacturing costs set relatively high in order to enforce production in remanufacturing loop. The (cumulative) demand is modeled as a Brownian process. The results are shown in Figures 3-6. Figure 3 illustrates the shape of the value function   x is below the threshold, zero production above the threshold and production "on demand" at the threshold-level. Comparing Figures 3 and 4 one can observe that the threshold level in mode 2 when machine 2 is in failure is higher than in mode 1. Figure 6 illustrates the optimal policy for the machine   2 (remanufacturing) in modes 1 when both machines are in operation (optimal policy of the machine 2 in mode 3 is identical). Machine 2 is must produce at average supply (proportional to demand) rate-which is also its maximal rate as explained in the section 2. Figures 7 and 8 show the realizations of Brownian and Markov-type (filtered) demand rates respectively (the variance D  is set to the same value). Comparing two graphs one can see that in Brownian case (Figure 7) the variation rate is much faster than in Markov case ( Figure  8). Figures 9 and 10 illustrate the optimal policy for the machine 1 (manufacturing) in mode 1 (both machines in operation) and mode 2 (remanufacturing machine 2 in failure) respectively. The results are to be compared with those shown in Figures 4 and 5. One can observe that the threshold values for the case of slower varying Markov demand are lower as compared to the case of Brownian demand. Parameters used for simulations are summarized in Table 1.
In the second case of Markov-type (filtered) demand the parameters of the filter may be used to fit the model   to the characteristics observed in the real life applications. A classical assumption of the constant demand in this context means that the variability of the demand is ignored and only its average rate is taken into account. A   second order terms in HJB equations reflecting demand variability are in that case neglected and the optimal policy is found using the first order approximation of HJB equation. Optimal policy for the main (manufacturing) machine is of hedging point in both studied (Brownian and Markov) cases-as it is for the constant demand. According to Figures 4, 5, 9 and 10 one can see that more the demand varies rapidly in time, more the hedging stock level increases in order to respond to the demand variability. In addition, the average total cost also increases from 1939 (Markov) to 2236.5 (Brownian) as the demand variability increases.

Conclusion and Future Work
We have shown that the problem of stochastic control corresponding to optimization of production planning in failure prone hybrid manufacturing/remanufacturing systems with random demand can be successfully analyzed for diffusion-type demands. We investigate this problem in continuous time which seems to be the most natural setting. We develop optimality conditions in the form of HJB equations and show that due to the Brownian component in the demand the HJB equations are the second where they are of the first order. We use finite difference approximations for HJB equations reducing a continuous time optimization problem to the discrete time, discrete state, infinite horizon dynamic programming problem, and use policy improvement technique [17] for solving it. Value functions of the stochastic optimization problems are usually non-smooth and corresponding HJB equations have to be addressed using generalized approaches such as viscosity solutions [18]. Theoretical studies of the convergence of discrete approximations to an exact (viscosity-type) solution of HJB equations when the size of the grid tends to zero is addressed in [19,20]. Such theoretical analysis is out of the scope of this paper where we propose a numerical approach targeting the new model for the uncertain demand that allows addressing more naturally the growing number of industrial applications. Considering possible extensions of the study presented in this paper we count to explore a compound demand model of Poisson and diffusion-type process thus allowing both the jumps and continuous random variation of the demand.
order PDEs, contrary to the case of a constant demand