Sensitivity Analysis of the Replacement Problem

The replacement problem can be modeled as a finite, irreducible, homogeneous Markov Chain. In our proposal, we modeled the problem using a Markov decision process and then, the instance is optimized using linear programming. Our goal is to analyze the sensitivity and robustness of the optimal solution across the perturbation of the optimal basis ( ) ∗ B obtained from the simplex algorithm. The perturbation ( )  B can be approximated by a given matrix H such that  ∗ = + B kB H . Some algebraic relations between the optimal solution and the perturbed instance are obtained and discussed.


Introduction
Machine replacement problem has been studied by a lot of researchers and is also an important topic in operations research, industrial engineering and management sciences.Items which are under constant usage, need replacement at an appropriate time as the efficiency of the operating system using such items suffer a lot.
In the real-world, the equipment replacement problem involves the selection of two or more machines of one or more types from a set of several possible alternative machines with different capacities and cost of purchase and operation.When the problem involves a single machine, it is common to find two well-defined forms to do so: the quantity-based replacement, and the time-based replacement.In the quantity-based replacement model, a machine is replaced when an accumulated product of size q is produced.In this model, one has to determine the optimal production size q.While in a time-based replacement model, a machine is replaced in every period of T with a profit maximizing.When the problem involves two or more machines it is named the parallel machine replacement problem [1], and the time-based replacement model consists of finding a minimum cost replacement policy for a finite population of economically interdependent machines.
A replacement policy is a specification of "keep" or "replace" actions, one for each period.Two simple examples are the policy of replacing the equipment every time period and the policy of keeping the first machine until the end of a period N.An optimal policy is a policy that achieves the smallest total net cost of ownership over the entire planning horizon and it has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.In practice, the replacement problem can be easily addressed using dynamic programming and Markov decision processes.
The dynamic programming uses the following idea: The system is observed over a finite or infinite horizon split up into periods or stages.At each stage the system is observed and a decision or action concerning the system has to be made.The decision influences (deterministically or stochastically) the state to be observed at the next stage, and depending on the state and the decision made, an immediate reward is gained.The expected total reward ( ) U t + from the present stage and the one of the following states is expressed by the functional equation.Optimal decisions depending on stage and state are determined backwards step by step as those maximizing the right hand side of the functional equation where ( ) u t + is the expected total reward in stage 1 t + , this reward depends on the action (keep or replace) in the last stage of operation, ( ) t + , T is the total number of stages, K and R are the actions associated with the equipment Keep and Replace, d is the set of actions , K R .
The Markov Decision process concept has been stated by Howard [3] combining the dynamic programming technique with the mathematical notion of a Markov Chain.The concept has been used to develop the solution of infinite stage problems such as in Sernik and Marcus [4], Kristensen [5], Sethi et al. [6], and Childress and Durango-Cohen [1].The policy iteration method was created as an alternative to the stepwise backward contraction methods.The policy iteration was a result of the application of the Markov chain environment and it was an important contribution to the development of optimization techniques [5].
In the other hand, a Markov decision process is a discrete time stochastic control process.At each time step, the process is in state s and the decision maker may choose any action a that is available in state s.The process responds at the next time step moving into a new state s′ , and giving the decision maker a corresponding reward ( ) , a R s s′ .The probability that the process chooses s′ as its new state is influenced by the chosen ac- tion.Specifically, it is given by the state transition function ( ) , a P s s′ .Thus, the next state s′ depends on the current state s and the decision maker's action a .But given s and a , it is conditionally independent of all previous states and actions; in other words, the state transitions of an MDP possess the Markov property.
Finally, it is important to note that linear programming was early identified as an optimization technique to be applied to Markov decision process as described by, for instance [5] and [7].In this document, we consider a stochastic machine replacement model.The system consists of a single machine; it is assumed that this machine operates continuously and efficiently over N periods.In each period, the quality of the machine deteriorates due to its use, and therefore, it can be in any of the N stages, denoted 1, 2, , N  .We modeled the replacement problem using a Markov decision process and then, the instance is optimized using linear programming.Our goal is to analyze the sensitivity and robustness of the optimal solution across the perturbation of the optimal basis.Specifically, the methodology used is to model the replacement problem through a Markov decision process, optimize the instance obtained using linear programming, analyzing the sensitivity and robustness of the solution obtained by the perturbation of the optimal basis from the simplex algorithm, and finally, obtain algebraic relations between the initial optimal solution and the perturbed solution.
In this work, we assume that for each new machine it state can become worse or may stay unchanged, and that the transition probabilities ij p are known, where ij p are defined as

{ }
next state will be current state is 0, if P j i j i = < also be assumed that the state of the machine is known at the start of each period, and we must choose one of the following two options: 1) Let the machine operate one more period in the state it currently is, 2) Replace the machine by a new one, where every new machine for replacement is assumed to be identical.The importance of this study is that the result of the objective function of the replacement problem depends on the probabilities of the transition matrices, which is why by perturbing these matrices, it is unknown if the solution and the decisions associated with the replacement problem will change.Few authors in the literature have studied the sensitivity of the replacement problem by perturbing the transition matrices (see for example [8] and [9]).Studying such problems is important not only to solve this particular model, but to assess if the objective function to solve is still convex.
The original contributions of this work are the perturbation of the optimal basis obtained with the simplex algorithm.It was concluded that by perturbing this optimal basis directly affects the transition matrices associated with the replacement problem, and it found a region of feasibility for perturbation, in which, the objective function and the decisions will not change.
The rest of the paper is organized as follows.The next section presents the literature review for determining the optimal replacement policy.Also we present the problem formulation to the replacement problem with discrete-time Markov decision process and using the equivalent linear programming.Section Properties of the perturbed optimal basis associated with the replacement problem shows some algebraic relations between the optimal basis and the perturbed instance.The following section presents numerical results for a specific example.Finally our conclusions and future research directions are given.

Literature Review
There are several theoretical models for determining the optimal replacement policy.The basic model considers maintenance cost and resale value, which have their standard behavior as per the same cost during earlier period and also partly having an exponential grown pattern as per passage of time.Similarly the scrap value for the item under usage can be considered to have a similar type of recurrent behavior.
In relation to stochastic models the available literature on discrete time maintenance models predominantly treats an equipment deterioration process as a Markov chain.Sernik and Marcus [4] obtained the optimal policy and its associated cost for the two-dimensional Markov replacement problem with partial observations.They demonstrated that in the infinite horizon, the optimal discounted cost function is piecewise linear, and also provide formulas for computing the cost and the policy.In [6], the authors assume that the deterioration of the machine is not a discrete process but it can be modeled as a continuous time Markov process, therefore, the only way to improve the quality is by replacing the machine by one new.They derive some stability conditions of the system under a simple class of real-time scheduling/replacement policy.
Some models are approached to evaluate the inspection intervals for a phased deterioration monitored complex components in a system with severe down time costs using a Markov model, see for example [10].
In [11] the problem is approached from the perspective of the reliability engineering developing replacement strategies based on predictive maintenance.Moreover, in [1] the authors formulated a stochastic version of the parallel machine replacement problem.They analyzed the structure of optimal policies under general classes of replacement cost functions.
Another important approach that has received the problem is the geometric programming [12].In its proposal, the author discusses the application of this technique to solving replacement problem with an infinite horizon and under certain circumstances he obtains a closed-form solution to the optimization problem.
A treatment to the problem when there are budget constraints can be found in [13].In their work, the authors propose a dual heuristic for dealing with large, realistically sized problems through the initial relaxation of budget constraints.
Compared with simulation techniques, some authors [14] proposed a technique based on obtaining the first two moments of the discounted cost distribution, and then, they approximate the underlying distribution function by three theoretical distributions using Monte Carlo simulation.
The most important pioneers in applying dynamic programming models replacement problems are: Bellman [15], White [16], Davidson [17], Walker [18] and Bertsekas [19] Recently the Markov decision process has been applied successfully to the animal replacement problem as a productive unit, see for example [20]- [22].
Although the modeling and optimization of the replacement problem using Markov decision processes is a topic widely known [23].However, there is a significant amount about the theory of stochastic perturbation ma-trices [24]- [27].In literature there are hardly results concerning the perturbation and robustness of the optimal solution of a replacement problem modeled via a Markov decision process and optimized using linear programming.In this paper we are interested in addressing this issue with a stochastic perspective.

Problem Formulation
We start by defining a discrete-time Markov decision process with a finite state space Z with states For every policy ( ) , the corresponding Markov chain is ergodic, then the steady state probabilities of this chain are given by ( ) and the problem is to find a policy ϑ for which the expected payoff is maximum.In this system, the time interval between two transitions is called a stage.An optimal policy is defined as a policy that maximizes (or minimizes) some predefined objective function.The optimization technique (i.e. the method to obtain an optimal policy) depends on the form of the objective function and it can result in different alternative objective function.The choice of criterion depends on whether the planning horizon is finite or infinite (Kristensen, 1996).
In our proposal we consider a single machine and regular times intervals whether it should be kept for an additional period or it should be replaced by a new.By the above, the state space is defined by , and having observed the state, action should be taken concerning the machine about to keep it for at least an additional stage or to replace it at the end of the stage.The economic returns from the system will depend on its evolution and whether the machine is kept or replaced, in this proposal this is represented by a reward depending on state and action specified in advance.If the action replace is taken, we assume that the replacement takes place at the end of the stage at a known cost, the planning horizon is unknown and it is regarded infinite, also, all the stages are of equal length .
The optimal criterion used in this document is the maximization of the expected average reward per unit of time given by ( ) , where i ϑ π is the limiting state probability under the policy ϑ , and the op- timization technique used is the linear programming.Thus, we may maximize the problem (1) using the equivalent linear programming given by [28].
where zk x is the steady-state unconditional probability that the system is in state z , and the decision k is made; similarly zk r is the reward obtained when the system is in state z , and the decision k is made.In this sense, k is optimal in state z if and only if, the optimal solution of (1) satisfy the unconditional probabilities zk x that the system visit the state z , when making the decision k are strictly positive.Note that, the optimal value of the objective function is equal to the average rewards per stage under an optimal policy.The optimal value of is equal to the limiting state probability i π under an optimal policy.Model (1) contains ( ) functional constrains and ( ) decision variables.In [9] is showed that the problem (1) has a degenerate basic feasible solution.In the remainder of this document, we are interested in the optimal basis associated with the solution of the problem (1) when it is solved via the Simplex Method.

Properties of the Perturbed Optimal Basis Associated with the Replacement Problem
In the LP model ( 1), the number of basic solutions ρ is less than or equal to the number of combinations ( ) * ∈ the optimal basis associated to problem (2), and B  the perturbed matrix of B * defined by where 1 k = and H is a matrix with the same order than B * .The optimal solution is ( ) and any perturbed solution is ( ) . From these assumptions we state and prove the next propositions and theorems.

Proposition 4.2:
The matrix H is defined by: [ ] where ij h are the entries of H that could be perturbed.The columns of the optimal basis * B and the perturbed basis B  must sum 1.
Proof: The proof is trivial.The optimal basis B * is composed by the transition probability matrix P , considering the properties of the Markov chain we have 0 , 0,1, , where lim .This property is valid also for B  .

Numerical Example
Consider the following transition probabilities matrices d zj p in [5], which represented a Markovian decision process with In order to maximize the objective function the cost coefficients are shown in Table 1.
The corresponding LP problem is: The optimal inverse basis ( ) The optimal solution and the basic variables of the inverse basis are (presented in order): ( ) ( ) , , , 0.1875, 0, 0.4375, 0.375 . The optimal objective function is 12,187.50.The basis * B that will be perturbed is formed by the columns ( ) , , , Note that B satisfies the Proposition 4.2 that corresponds with the Equation ( 7), this property must be conserved for B  .
Suppose that we are interested to perturb 12 x This decision variable has associated the transition probability ( ) Simplifying the restriction of the state 1 in the LP model (19), the value for this variable is ( ) ( ) Continuing with the process, the restrictions of the states 2 and 3 are respectively: ( ) ( ) Because the restrictions of the LP model (1) the probability is affected by a minus sign.In * B , the variable 12 x is associated with the vector ( ) , and the positions that could be perturbed are − , considering the Equation (7).Note that the first element of the vector does not have any perturbation, because it corresponds to the first restriction of the LP model (1).Suppose also, that the column vector ( ) of the matrix * B that corresponds to the variable 12 x will be perturbed in the second position, from 2 3 to 2 3+ ∈ .The perturbed vector is So, the H matrix is: therefore the perturbed matrix is : Every value of ( ) ( )

H h h h h
is associated with the decision (replace) and the state 1 z = (the variable associated with this column vector is ), because of this, any perturbation in 1 H will affect the R matrix in the first column The R matrix is now ( ) ( ) ( ) ( ) ( ) The K matrix has no changes.
Considering the Equation ( 17) of the Theorem 4.5, x  is obtained Solving the inequality associated with the first element 1 0 8 13 10 15 20 , an interval ( ) ∞ , considering that the probabilities are between 0 and 1, the extent to perturb ∈ in this particular case are ( ]  2 and Ta-ble 3), Proposition 4.1 2 (Table 4 and Table 5), Theorem 4.3 (Table 6 and Table 7), and Proposition 4.4 (Table 8 and Table 9).

Conclusions and Future Work
In this document, we considered a stochastic machine replacement problem with a single machine that operates continuously and efficiently over N periods.We were interested in the matrix perturbation procedure from a probabilistic point of view.A perturbation in a Markov chain can be referred as a slight change in the entries of the corresponding transition stochastic matrix.We perturbed the optimal basis * B , but this kind of perturbation also changes the transition stochastic matrices.
s =  the analyst should made a decision d between ξ possible.Denote by ( ) z n z = and ( ) i d n d= the state and the decision made in stage n respectively, then, the system moves at the next stage 1 n + in to the next state j with a known probability given by the transition occurs, it is followed by the reward k the state z after the decision k d is made.

2 3, 1 −
to conserve the feasibility of the perturb solution x  .Consi- dering this perturbation interval we calculate the numerical comparative of the Proposition 4.1 1 (Table
−of the LP problem associated to this solution is:
a This value is obtained directly from the LP model.b This value is obtained doing the matrix operations.
a This value is obtained directly from the LP model.b This value is obtained doing the matrix operations.
a This value is obtained directly from the LP model.b This value is obtained doing the matrix operations.
a This value is obtained directly from the LP model.b This value is obtained doing the matrix operations.