Discrete Time Markov Reward Processes a Motor Car Insurance Example *

Guglielmo D’Amico, Jacques Janssen, Raimondo Manca Università “G. D’Annunzio” di Chieti, Dip. di Scienze del Farmaco, via dei Vestini, Chieti, Italy Jacan &, EURIA, Université de Bretagne Occidentale, 6 avenue le Gorgeu, Brest, France Università “La Sapienza”, Dip. di Matematica per le Decisioni Economiche, Finanziarie ed Assicurative, via del Castro Laurenziano, Roma, Italy E-mail: g.damico@unich.it, jacques.janssen@skynet.be, raimondo.manca@uniroma1.it Received December 11, 2009; revised December 20, 2009; accepted December 22, 2009


Introduction
In the sixties and seventies, Markov reward processes were developed, mainly in the engineering fields in discrete and continuous time [1].In [2] an application of Continuous Time Markov Reward Processes in life insurance was presented.
In this paper, we present the Discrete Time Markov Reward Processes (DTMRWP) as given in [3].The evolution equation of the expected value of the DTMRWP is presented with different reward structures.Furthermore, the relations useful for the computation of the higher order moments of the Markov reward process are presented and they are given in matrix form too. To the authors' knowledge, it is the first time that higher moments of a discrete time Markov reward process and the matrix approach for the first n moments are given.The matrix approach facilitates the algorithm construction as for example it is explained in [4] for semi-Markov reward processes.
We believe that DTMRWP can describe any kind of premiums or benefits involved in a generic insurance contract then they represent tool to approach in a general way actuarial problems.
In the last section an example on the application of DTMRWP in the motor car insurance is given using real data applied to the bonus-malus Italian rules.

Reward Structure, Classifications and Notation
The association of a sum of money to a state of the system and to a state transition assumes great relevance in the study of financial phenomena.This can be done by linking a reward structure to a stochastic process.This structure can be thought of as a function associated with the state occupancies and transitions [1].
In this paper the rewards are considered as amounts of money.These amounts can be positive, if they are benefits for the system and negative if they are costs.
A classification scheme of different kinds of DTM RWP is reported in [5] page 150.

Discounting Factors
The following notations will be used: (1), (2), , ( ), r r r t   0 , , , for the discrete time homogeneous interest rates and for the discrete time discount factors.See [6] or [7] for further details on this topic.

Reward Notation
, ( ), denote the reward that is given for the per-* Work supported by a MURST grant.
manence in the i-th state; it is also called rate reward, see [8]; the first is paid in the cases in which the period amount in state i is constant in time, the second when the payment is a function of the state and the time of payment. represents the vector of these rewards.
denote the reward that is given for the transition from the ith state to the jth one (impulse reward). is the matrix of the transition rewards.Γ The different kinds of  rewards represent an annuity that is paid because of remaining in a state.This flow is to be discounted at starting time.In the immediate case, the reward will be paid at the end of the period before the transition; in the due case the reward will be paid at the beginning of the period.On the other hand,  represents lump sums that are paid at the instant of transition.
As far as the impulse reward  is concerned, it is only necessary to compute the present value of the lump sum paid at the moment of the related transition.
Reward structure can be considered a very general structure linked to the problem being studied.The reward process evolves together with the evolution of the Markov process which it is linked.When the studied stochastic system is in a state then a reward of type  is paid; once there is a transition an impulse reward of  type is paid.This behaviour is particularly efficient at constructing models which are useful to follow, for example, the dynamic evolution of insurance problems e.g.[9] and [10].

Matrix Operations
We give some matrix operation notation useful to describe the equations of the moments of the Markov reward processes in matrix form.
Given the two matrices A, B with the notations and are denoted, respectively, the usual row column product and the element by element product.Definition 2.1 Given two matrices that have row order equal to m and column order equal to n, the following operation is defined: where c is the m elements vector in which the i-th component is obtained in the following way: 1 ( )


the discounted rewards accumulated in n periods given that at time 0 the system was in the state i and the reward are paid in the immediate case.It is defined recursively as follows: where (0) 0 i   Similar relations can be easily written for discounted homogeneous due cases.We denote by: ( ) ( ), ( ),..., ( ) the mean present value of the rewards paid in the investigated horizon time in the due cases is represented.In this case, in the definition of the ( i  process we put (0) For the sake of understanding, first we present the simplest case in immediate and due hypotheses after only the general relations in the discrete time environment will be given.
The immediate homogeneous Markov formula in the case of fixed permanence and without transition rewards is the first relation presented.The DTMRWP present value after one payment is: (2) (1 ) ( ) and in general, taking into account the recursive nature of relations, at n-th period it is: ( The general case with variable permanence, transition rewards and interest rates is presented.The present value , (2) and in general, taking into account the recursive nature of relation, at n-th period it is: ( 1) This relation can be written in matrix notation in the following way: .
In the case of payment due the permanence reward is paid at beginning of the period and the transition reward at the end.It results: , (2) (2) .
In this section, general formulas were presented.In the construction of the algorithms the dif-ferences between the possible cases should be taken into account.For example in the non-discounting case the following can be stated

The higher Order Moments of Markov Reward Processes
In [11] relations for higher order moments of the integral of a generic function that evolves following a semi-Markov process were given.In more recent works (see [4] and [12]), the relations for higher moments of rewards associated to a semi-Markov backward system were presented.In this section, following the methodology used in the last two quoted papers, the recursive relations useful for computing the higher moments in a Markov reward environment are provided.
It should be stated that the equations of this paper are different from that of [4] and [12] because we consider the conditioning on the starting state but also on the arriving state.
We will give only the discounted case.
According to Section 3 let us define the following stochastic process: Definition 2: Let denote by ( ) ij n  the accumulated discounted rewards in n periods given that at time 0 the system was in the state i and at time it will be in state j: ( ) ( ), ( ),..., ( ) and the higher order moments are defined as ( ) ( ), ( ),..., ( ) and it results for all r that ( ) Similar relations can be easily written for non discounted cases Theorem 4.1 The moments of ( ) ij n  in the discounted immediate case, in matrix form, are given by: where: Proof From (4.1) it results: X n  and by measurability of with respect to the information set ,0 ; , 0 ( 1) that in matrix form gives (5) Now since , by direct computation we get By means of similar procedures, the following corollaries can be obtained.
Corollary 4.2 The higher moments of ( ) ij n  in the discounted due case, in matrix form, are given by the following relation: where: The possibility of computing the second order moments permits the obtaining of the variance and the sigma square, having in this way the opportunity to have a risk measure.

Motorcar Insurance Application
As it is well known, the bonus-malus motor car insurance model can be studied by means of Markov chains, see [13] for a complete description of bonus-malus systems.As far as the authors know, the premiums received and the benefits paid by the insurance company have never been studied simultaneously inside the evolution equation of the model as we propose here.In this way it is possible to have information on the future evolution of cash flows of the insurer and the possibility of computing higher order moments permits the obtaining of risk measures.
In order to apply DTMRWP we will construct a bonus-malus Markov reward model.
It should be noted that, as explained in [14], motor car insurance premiums could be a function of many factors such as type of car, mileage, age of the driver, region, sex and so on.
In Italy the only official distinctions are the province in which the car is insured and the power of its engine.This example will use a transition matrix related to the motor car bonus-malus insurance rules that apply in Italy.In this case, the Markov model fits quite well because: 1) the position of each insured person is given at the beginning of each year, 2) there are precise rules that give the change of states in function of the behaviour of the policyholder person during the year, 3) the future state depends only on the present one.
The range of values of T is expressing the classes of risk in which all drivers are classified.The stochastic process  1, 2,....,18 ( ) X t describing the rating risk class evolution of the policyholder is assumed to be a Markov chain with state space .This choice is determined by the fact that the next risk class is determined through rule (5.1) as a function of the current risk class and the number of accidents the policyholder carried out in the current year.

 
,...,18 The authors are in possession of the history of 105627 insured persons over a period of three years.This means that it was possible consider 316881 real or virtual transitions.The data are related to the years 1998, 1999 and 2000.The estimated Markov transition matrix obtained from the available data taking into account the bonus-malus Italian rules is given in the Table 1.In this table we report only the transition probability that are possible to be observed, the remaining are impossible due to the Italian BMS rules.Then for example, in one step, from state 1 it is possible to migrate only towards state 1 (0 accident), to state 3 (1 accident), to state 6 (2 accident), to state 9 (3 accident) and to state 12 (4 or more accident).The other transitions are not allowable and then their probabilities are zero and then not reported in the table.
The payment of a claim by the insurance company can be seen as a lump sum (impulse or transition reward) paid by the insurer to the insured person.
In Figure 1 the premiums (they can be seen as permanence rewards) that are paid in Naples for a car of 2300 c.c. and in Oristano (a small Sardinian province) for a small car (about 1000 c.c.) are reported.The example is constructed from the point of view of the insurance company and premiums are an entrance for the company.It is to precise that these values correspond to the real premiums (that is loaded premiums covering costs and risk) paid by an insured in the year 2001 and officially given in the internet site of Assicurazioni Generali for that year.
In the example we suppose that the rewards are fixed in the time.Furthermore we suppose to have a yearly fixed discount factor of 1/1.03.
2 gives the mean values of the expenses that the insurance company should pay for the claims made by the insured person.
More clearly stated, the element -7772.51represents the expenses that, on average, the company has to pay for the two accidents that an insured person that was in the state 1 (lowest bonus-malus class) had and which then took him to state 6.
This table was constructed starting from the observed data in the authors' possession.
From the point of view of the model, the elements of this table are transition rewards.More precisely, as already mentioned, they can be seen as lump sums (impulse rewards) paid by the company at the time of the accident.In this case, being expenses for the company, they result negative.

Corollary 4 . 1
The evolution equation of the higher order moment of the ( ) i n  process in the discounted immediate case, in matrix form is: