^{1}

^{*}

^{1}

^{*}

Invariant measures of Markov chains in discrete or continuous time with a countable set of states are characterized by its steady state recurrence relations. Exemplarily, we consider transition matrices and Q-matrices with upper bandwidth n and lower bandwidth 1 where the invariant measures satisfy an (n + 1)-order linear difference equation. Markov chains of this type arise from applications to queueing problems and population dynamics. It is the purpose of this paper to point out that the forward use of this difference equation is subject to some hitherto unobserved aspects. By means of the concept of generalized continued fractions (GCFs), we prove that each invariant measure is a dominated solution of the difference equation such that forward computation becomes numerically unstable. Furthermore, the GCF-based approach provides a decoupled recursion in which the phenomenon of numerical instability does not appear. The procedure results in an iteration scheme for successively computing approximants of the desired invariant measure depending on some truncation level N. Increasing N leads to the desired solution. A comparison study of forward computation and the GCF-based approach is given for Q-matrices with upper bandwidth 1 and 2.

This paper is dedicated to discrete-time Markov chains X = ( X k ) k ≥ 0 with a countably infinite set of states (labelled as 0,1,2, ⋯ ) and a stationary one-step transition matrix P = ( P i k ) 0 ≤ i , k < ∞ of the form

P = [ P 00 P 01 P 02 … P 0 n O O … … P 10 P 11 P 12 … P 1 n P 1 , n + 1 O … … O P 21 P 22 … P 2 n P 2 , n + 1 P 2 , n + 2 O … O O P 32 … P 3 n P 3 , n + 1 P 3 , n + 2 P 3 , n + 3 ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ] (1)

with n ≥ 1 and P k , k + n ≠ 0 for at least one k ∈ ℕ 0 = { 0,1,2, ⋯ } . In addition we assume that X is irreducible and recurrent. Notice that the irreducibility of X implies

P k , k − 1 ≠ 0 for k = 1 , 2 , ⋯ (2)

For a recurrent irreducible Markov chain, the system

{ x k = ∑ i = 0 ∞ x i P i k ( k = 0 , 1 , 2 , ⋯ ) x 0 = 1 , x k ≥ 0 ( k = 1 , 2 , ⋯ ) (3)

has a unique solution which is strictly positive, see Karlin and Taylor ( [

Substituting (1) into (3) and rearranging yields the (n + 1)th-order homogeneous linear recurrence relation

P k + n + 1 , k + n x k + n + 1 + ( P k + n , k + n − 1 ) x k + n + ⋯ + P k , k + n x k = 0 ( k = 0 , 1 , 2 , ⋯ ) (4)

coupled to the side conditions

{ P 00 x 0 + P 10 x 1 = x 0 P 01 x 0 + P 11 x 1 + P 21 x 2 = x 1 ⋮ P 0 , n − 1 x 0 + P 1 , n − 1 x 1 + ⋯ + P n , n − 1 x n = x n − 1 (5)

From the theory of linear difference equations (see Miller [

Since P k + n + 1, k + n ≠ 0 for all k the desired solution ( x k ) k = 0 ∞ is uniquely determined by its initial values x 0 , x 1 , ⋯ , x n and (5). An application of the recurrence relation (4) then should directly lead to the desired solution x. But unfortunately forward computation of x by means of the homogeneous equation (4) is not a meaningful procedure. In the sequel it is shown that the solution space S of the recurrence relation (4) is the direct sum

S = S 1 ⊕ S 2

of two subspaces S 1 and S 2 with the property, that every solution y ∈ S 2 dominates over every solution x ∈ S 1 , i.e.

lim k → ∞ x k y k = 0.

In addition, we have dim S 1 = n , dim S 2 = 1 and x ∈ S 1 .

As pointed out by Cash [

The phenomenon of inherent instability has been first recognized in connection with the computation of higher transcendental functions such as Bessel and associated Legendre functions (see Gautschi [

Remark. Most of our considerations concerning the computation of invariant measures of Markov chains may be extended to more general infinite systems of equations. In Section 3, we will point out the properties of invariant measures which we explicitly used.

This chapter deals with the computation of dominated solutions of homogeneous linear difference equations by means of generalized continued fractions (GCFs). The theory of GCFs was initiated by Jacobi [

Defintion 1. A generalized continued fraction (GCF) of dimension n is an ( n + 1 ) -tuple ( a ( 1 ) , ⋯ , a ( n ) , b ) of real-valued sequences and a convergence structure as follows. Let A ( 1 ) , ⋯ , A ( n ) , B be the principal solutions of the corresponding homogeneous linear difference equation

x k + n + 1 = b k x k + n + a k ( n ) x k + n − 1 + ⋯ + a k ( 1 ) x k ( k = 0 , 1 , 2 , ⋯ ) (6)

satisfying

{ A j ( i ) = δ i , j + 1 = { 1 for i = j + 1 0 for i ≠ j + 1 ( i = 1 , ⋯ , n ; j = 0 , ⋯ , n ) B j = 0 for j = 0 , ⋯ , n − 1 and B n = 1. (7)

The GCF is said to converge iff all the limits

ξ ( i ) = lim k → ∞ A k ( i ) / B k ( i = 1 , ⋯ , n ) (8)

do exist.

Convergence of a GCF is indicated by the notation

[ ξ ( 1 ) ⋮ ξ ( n ) ] = k [ a 0 ( 1 ) a 1 ( 1 ) a 2 ( 1 ) ⋯ ⋮ ⋮ ⋮ a 0 ( n ) a 1 ( n ) a 2 ( n ) ⋯ b 0 b 1 b 2 ⋯ ] .

Terminating a GCF after the N-th column we get the so-called N-th approximants

[ A N + n + 1 ( 1 ) B N + n + 1 ⋮ A N + n + 1 ( n ) B N + n + 1 ] = [ a 0 ( 1 ) a 1 ( 1 ) ⋯ a N ( 1 ) ⋮ ⋮ ⋱ ⋮ a 0 ( n ) a 1 ( n ) ⋯ a N ( n ) b 0 b 1 ⋯ b N ] .

The term “GCF” becomes obvious if forward computation of the N-th approximant of a GCF is replaced by the equivalent backward algorithm. It is known that the recurrence relation (6) can be converted into a first order vector recursion of the form

u k + 1 = W k u k ( k = 0 , 1 , 2 , ⋯ ) (9)

by putting

u k = [ x k x k + 1 ⋮ x k + n − 1 x k + n ] and W k = [ 0 1 0 ⋯ 0 0 0 0 1 ⋯ 0 0 ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ 0 0 ⋯ ⋯ 0 1 a k ( 1 ) a k ( 2 ) ⋯ ⋯ a k ( n ) b k ] ( k = 0 , 1 , 2 , ⋯ ) .

Comparing (6) and (9) it is seen, that the numerators A N + n + 1 ( 1 ) , ⋯ , A N + n + 1 ( n ) and the denominators B N + n + 1 of the GCF appear in the last row of the matrix

W N W N − 1 ⋯ W 0 .

Consider now the backward recurrence scheme

v ( N ) k = v ( N ) k + 1 W k ( k = N , N − 1 , ⋯ , 0 ) (10)

with initial vector v ( N ) N + 1 = ( 0, ⋯ ,0,1 ) . Then

v ( N ) 0 = v ( N ) N + 1 W N W N − 1 ⋯ W 0 = ( A N + n + 1 ( 1 ) , ⋯ , A N + n + 1 ( n ) , B N + n + 1 ) .

Writing (10) in expanded form and inserting the structure of W k , we get

v ( N ) k ( 1 ) = v ( N ) k + 1 ( n + 1 ) a k ( 1 ) ( k = N , N − 1 , ⋯ , 0 ) ,

v ( N ) k ( i ) = v ( N ) k + 1 ( i − 1 ) + v ( N ) k + 1 ( n + 1 ) a k ( i ) ( i = 2 , ⋯ , n , k = N , N − 1 , ⋯ , 0 ) ,

v ( N ) k ( n + 1 ) = v ( N ) k + 1 ( n ) + v ( N ) k + 1 ( n + 1 ) b k ( k = N , N − 1 , ⋯ , 0 )

or, equivalently

{ r ( N ) k ( 1 ) : = v ( N ) k ( 1 ) v ( N ) k ( n + 1 ) = a k ( 1 ) b k + r ( N ) k + 1 ( n ) ( k = N , N − 1 , ⋯ , 0 ) r ( N ) k ( i ) : = v ( N ) k ( i ) v ( N ) k ( n + 1 ) = a k ( i ) + r ( N ) k + 1 ( i − 1 ) b k + r ( N ) k + 1 ( n ) ( i = 2 , ⋯ , n ; k = N , N − 1 , ⋯ , 0 ) (11)

with initial values r ( N ) N + 1 ( i ) = 0 for i = 1 , ⋯ , n . Hence

A N + n + 1 ( i ) B N + n + 1 = r ( N ) 0 ( i ) ( i = 1 , ⋯ , n )

and

ξ ( i ) = lim N → ∞ r ( N ) 0 ( i ) ( i = 1 , ⋯ , n ) . (12)

Alternative procedures for computing GCFs are described in [

The relations between GCFs and linear difference equations have been first recognized by Perron [

Theorem 1. A GCF ( a ( 1 ) , ⋯ , a ( n ) , b ) converges iff there are n + 1 linearly independent solutions x ( 1 ) , ⋯ , x ( n ) , y of the recurrence relation (6) satisfying

lim k → ∞ x k ( i ) y k = 0 ( i = 1 , ⋯ , n ) (13)

and

| x 0 ( 1 ) ⋯ x n − 1 ( 1 ) ⋮ ⋮ x 0 ( n ) ⋯ x n − 1 ( n ) | ≠ 0. (14)

As pointed out by Gautschi [

[ ξ l ( 1 ) ⋮ ξ l ( n ) ] = [ a l ( 1 ) a l + 1 ( 1 ) a l + 2 ( 1 ) ⋯ ⋮ ⋮ ⋮ a l ( n ) a l + 1 ( n ) a l + 2 ( n ) ⋯ b l b l + 1 b l + 2 ⋯ ] (15)

exist for all l ≥ 0 , then x ∈ span ( x ( 1 ) , ⋯ , x ( n ) ) iff

x l + n = − ξ l ( n ) x l + n − 1 − ⋯ − ξ l ( 1 ) x l ( l = 0 , 1 , 2 , ⋯ ) . (16)

Combining (11), (12) and (16), one obtains an efficient algorithm (which we will refer to as Van der Cruyssen’s algorithm) for approximating the first L + 1 components x 0 , ⋯ , x L of an element x ∈ span ( x ( 1 ) , ⋯ , x ( n ) ) with prescribed values x 0 , ⋯ , x n − 1 :

Step 1: Select N > L − n and define r ( N ) l ( i ) for i = 1 , ⋯ , n ; l = 0 , 1 , ⋯ , N + 1 by

r ( N ) N + 1 ( 1 ) = ⋯ = r ( N ) N + 1 ( n ) = 0 ,

r ( N ) l ( 1 ) = a l ( 1 ) b l + r ( N ) l + 1 ( n ) ( l = N , N − 1 , ⋯ , 0 ) ,

r ( N ) l ( i ) = a l ( i ) + r ( N ) l + 1 ( i − 1 ) b l + r ( N ) l + 1 ( n ) ( i = 2 , ⋯ , n ; l = N , N − 1 , ⋯ , 0 ) .

Step 2: Set x ( N ) 0 = x 0 , ⋯ , x ( N ) n − 1 = x n − 1 and

x ( N ) l + n = − r ( N ) l ( n ) x ( N ) l + n − 1 − ⋯ − r ( N ) l ( 1 ) x ( N ) l

for l = 0 , 1 , ⋯ , L − n .

Step 3: Increase N until the accuracy of the iterates is within prescribed limits.

For any l , the vector ( r ( N ) l ( 1 ) , ⋯ , r ( N ) l ( n ) ) T is an approximant of a GCF. Hence, convergence of the algorithm is related to convergence of GCFs. For the latter, we cite two helpful results.

Theorem 2. (Levrie [

[ a 0 ( 1 ) a 1 ( 1 ) a 2 ( 1 ) ⋯ ⋮ ⋮ ⋮ a 0 ( n ) a 1 ( n ) a 2 ( n ) ⋯ b 0 b 1 b 2 ⋯ ]

converges if it satisfies the so-called dominance condition, i.e.

| a k ( 1 ) | + | a k ( 2 ) | + ⋯ + | a k ( n ) | + 1 ≤ | b k | for all k ≥ 0.

Theorem 3. (De Bruin [

[ a 0 ( 1 ) ρ 0 ρ − 1 ⋯ ρ − n a 1 ( 1 ) ρ 1 ρ 0 ⋯ ρ 1 − n ⋯ a ν ( 1 ) ρ ν ρ ν − 1 ⋯ ρ ν − n ⋯ a 0 ( 2 ) ρ 0 ρ − 1 ⋯ ρ 1 − n a 1 ( 2 ) ρ 1 ρ 0 ⋯ ρ 2 − n ⋯ a ν ( 2 ) ρ ν ρ ν − 1 ⋯ ρ ν − n + 1 ⋮ ⋮ ⋮ a 0 ( n ) ρ 0 ρ − 1 a 1 ( n ) ρ 1 ρ 0 a ν ( n ) ρ ν ρ ν − 1 b 0 ρ 0 b 1 ρ 1 b ν ρ ν ] (17)

with nth numerators A ˜ ν + n + 1 ( i ) ( i = 1, ⋯ , n ) and denominators B ˜ ν + n + 1 where the real numbers ρ − n , ρ − n + 1 , ⋯ , ρ 0 , ρ 1 , ⋯ are all different from zero. Then

A ˜ ν + n + 1 ( i ) = ρ ν ρ ν − 1 ⋯ ρ 1 ρ 0 ⋯ ρ − n + i − 1 A ν + n + 1 ( i ) ( i = 1 , ⋯ , n ) B ˜ ν + n + 1 = ρ ν ρ ν − 1 ⋯ ρ 0 B ν + n + 1 } ν ∈ ℕ 0

In other words, if one of these two GCFs converges so does the other.

Notice that the GCF (17) may be interpreted as an equivalence transformation of the GCF (12).

To make (4) congruent with (6) we put

a k ( i ) = − P k + i − 1 , k + n P k + n + 1 , k + n ( i = 1 , ⋯ , n , k = 0 , 1 , 2 , ⋯ ) , (18)

b k = − P k + n , k + n − 1 P k + n + 1 , k + n ( k = 0 , 1 , 2 , ⋯ ) . (19)

Theorem 4. Let a k ( i ) , b k be defined as in (18) and (19). The limits

[ ξ l ( 1 ) ⋮ ξ l ( n ) ] = [ a l ( 1 ) a l + 1 ( 1 ) a l + 2 ( 1 ) ⋯ ⋮ ⋮ ⋮ a l ( n ) a l + 1 ( n ) a l + 2 ( n ) ⋯ b l b l + 1 b l + 2 ⋯ ] (20)

exist for all l ≥ 0 if the corresponding Markov chain is irreducible and recurrent. In case of convergence the invariant measure is a dominated solution of (6) and satisfies the decoupled recursion

x l + n = − ξ l ( n ) x l + n − 1 − ⋯ − ξ l ( 1 ) x l ( l = 0 , 1 , 2 , ⋯ ) . (21)

Proof. Denote by P ( N ) = ( P i k ) 0 ≤ i , k ≤ N the ( N + 1 ) × ( N + 1 ) northwest corner truncation of P . Since P is assumed to be irreducible and recurrent we conclude from Seneta [

h ( N ) ( I ( N ) − P ( N ) ) = e 0 (22)

where I ( N ) is the unit matrix and e ( N ) 0 is the vector with unity in the first position and zeros elsewhere has a unique solution h ( N ) = ( h ( N ) 0 , h ( N ) 1 , ⋯ , h ( N ) N ) satisfying

l i m N → ∞ h ( N ) k h ( N ) 0 = x k x 0 ( k = 0,1,2, ⋯ ) .

The vector h ( N ) satisfies the homogeneous linear difference Equation (6) for k = 0 , 1 , ⋯ , N − n and is subject to the boundary conditions

( 1 − P 00 ) h ( N ) 0 − P 10 h ( N ) 1 = 1 − P 01 h ( N ) 0 + ( 1 − P 11 ) h ( N ) 1 − P 21 h ( N ) 2 = 0 ⋮ − P 0 , n − 1 h ( N ) 0 − P 1 , n − 1 h ( N ) 1 − ⋯ − P n , n − 1 h ( N ) n = 0 } (23)

and

h ( N ) N + 1 = 0. (24)

Notice that the numerators A l ( 1 ) , ⋯ , A l ( n ) and the denumerators B l of the GCF (20) satisfy

x k + n + 1 = b k + l x k + n + a k + l ( n ) x k + n − 1 + ⋯ + a k + l ( 1 ) x k ( k = 0 , 1 , 2 , ⋯ ) (25)

and build up a fundamental system of solutions to Equation (6) for k ≥ l . Hence any solution of the recurrence relation (6) can be expressed in terms of these functions. In view of their initial values

{ A l , j ( i ) = δ i , j + 1 = { 1 for i = j + 1 0 for i ≠ j + 1 ( i = 1 , ⋯ , n ; j = 0 , ⋯ , n ) B l , j = 0 for j = 0 , ⋯ , n − 1 and B l , l + n = 1 ; (26)

we get

h ( N ) k = h ( N ) l A l , k − l ( 1 ) + ⋯ + h ( N ) l + n − 1 A l , k − l ( n ) + h ( N ) l + n B l , k − l ( k = l , l + 1 , ⋯ ) . (27)

Choosing k = N + 1 and utilizing (24) equation (27) reduces to

h ( N ) l + n B l , N − l + 1 = − h ( N ) l A l , N − l + 1 ( 1 ) − ⋯ − h ( N ) l + n − 1 A l , N − l + 1 ( n ) . (28)

Dividing both sides of (28) by B l , N − l + 1 and h ( N ) 0 we formally arrive at

h ( N ) l + n h ( N ) 0 = − h ( N ) l h ( N ) 0 A l , N − l + 1 ( 1 ) B l , N − l + 1 − ⋯ − h ( N ) l + n − 1 h ( N ) 0 A l , N − l + 1 ( n ) B l , N − l + 1 . (29)

Passing to the limit N → ∞ we get the decoupled recursion (21) provided that the limits l i m N → ∞ A l , N − l + 1 ( i ) / B l , N − l + 1 do exist for i = 0 , 1 , ⋯ , n .

To prove our assertion we utilize that the invariant measure ( x k ) k ≥ 0 of the irreducible and recurrent Markov chain (1) is strictly positive and satisfies the recurrence relation (4) which can be rewritten as

P k , k + n ⋅ x k P k + n + 1 , k + n ⋅ x k + n + 1 + P k + 1 , k + n ⋅ x k + 1 P k + n + 1 , k + n ⋅ x k + n + 1 + ⋯ + P k + n − 1 , k + n ⋅ x k + n − 1 P k + n + 1 , k + n ⋅ x k + n + 1 + 1 = ( 1 − P k + n , k + n ) x k + n P k + n + 1 , k + n ⋅ x k + n + 1 ( k = 0 , 1 , 2 , ⋯ ) . (30)

Define ρ k = x k + n x k + n + 1 for k = 0 , 1 , 2 , ⋯ Then (30) becomes equivalent to

| a k ( 1 ) ρ k ρ k − 1 ⋯ ρ k − n | + | a k ( 2 ) ρ k ρ k − 1 ⋯ ρ k − n + 1 | + ⋯ + | a k ( n ) ρ k ρ k − 1 | + 1 = | b k ρ k | ( k = 0 , 1 , 2 , ⋯ )

implying that the GCFs

[ a l ( 1 ) ρ l ρ l − 1 ⋯ ρ l − n a l + 1 ( 1 ) ρ l + 1 ρ l ⋯ ρ l − n + 1 ⋯ a l ( 2 ) ρ l ρ l − 1 ⋯ ρ l − n + 1 a l + 1 ( 2 ) ρ l + 1 ρ l ⋯ ρ l − n + 2 ⋯ ⋮ ⋮ a l ( n ) ρ l ρ l − 1 a l + 1 ( n ) ρ l + 1 ρ l ⋯ b l ρ l b l + 1 ρ l ⋯ ]

converge for all l ≥ 0 by means of Theorem 2. From Theorem 3, we then conclude that the same is true for the GCFs (8). ,

Theorem 2 says that the numerical calculation of the invariant measures of X can be reduced to Van der Cruyssen’s algorithm with (23) as initial conditions.

Remark. It is important that the statement of Theorem 4 consists of two parts:

・ The invariant measure is dominated by other solutions of the difference Equation (4).

・ By means of GCFs, we are able to construct a decoupled recursion which is not affected to numerical instability.

Our main goal was the derivation of the first statement. Although the existence of dominating solutions has been pointed out in specific examples (e.g. ( [

The observation of inherent numerical instability of (4) is essential since the transition structure under consideration arises in many practical applications, e.g. state-dependent queueing systems with bulk arrivals, and it is also valid in a more general context, e.g. as an approximation to state-dependent variants of the traditional G/M/1 queueing model.

Basically, we have exploited that the solutions of the truncated system (22) converge to invariant measures as N → ∞ . For Markov chains with a general transition structure, there is no way of solving x = x P directly (in particular, forward computation cannot be applied if P k l > 0 for some k < l − 1 ). In this situation, Seneta [

The results of the preceding chapter can easily be extended to continuous-time Markov chains Y = ( Y t ) t ≥ 0 generated by a conservative, irreducible and regular Q-matrix of the form

Q = [ q 00 q 01 q 02 ⋯ q 0 n O O ⋯ ⋯ q 10 q 11 q 12 ⋯ q 1 n q 1 , n + 1 O ⋯ ⋯ O q 21 q 22 ⋯ q 2 n q 2 , n + 1 q 2 , n + 2 O ⋯ O O q 32 ⋯ q 3 n q 3 , n + 1 q 3 , n + 2 q 3 , n + 3 ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ] (31)

where n ≥ 1 and q k , k + n ≠ 0 for at least one k ∈ ℕ 0 . Notice that the irreducibility of Y implies

q k , k − 1 ≠ 0 for k = 1 , 2 , ⋯

If Y is positive recurrent, the limits

π k = lim t → ∞ P ( Y t = k | Y 0 = i ) ( k = 0 , 1 , 2 , ⋯ )

exist independently of the initial state and build up a probability distribution which is strictly positive. For the construction of a continuous-time Markov process from its infinitesimal properties the reader is referred to Reuter [

z ( N ) Q ( N ) = e ( N ) 0 (32)

has a unique solution z ( N ) = ( z ( N ) 0 , z ( N ) 1 , ⋯ , z ( N ) N ) satisfying

lim N → ∞ z ( N ) k z ( N ) 0 = π k π 0 ( k = 0 , 1 , 2 , ⋯ ) ,

see Tweedie [

c k ( i ) = − q k + i − 1 , k + n q k + n + 1 , k + n ( i = 1 , ⋯ , n , k = 0 , 1 , 2 , ⋯ ) , (33)

d k = − q k + n , k + n q k + n + 1 , k + n ( k = 0 , 1 , 2 , ⋯ ) . (34)

By substituting (31) into (32) it is seen that z ( N ) satisfies the ( n + 1 ) th order homogeneous linear difference equation

π k + n + 1 = d k π k + n + c k ( n ) π k + n − 1 + ⋯ + c k ( 1 ) π k ( k = 0 , 1 , 2 , ⋯ ) (35)

for k = 0 , 1 , ⋯ , N − n and the boundary conditions

q 00 z ( N ) 0 + q 10 z ( N ) 1 = 1 q 01 z ( N ) 0 + q 11 z ( N ) 1 + q 21 z ( N ) 2 = 0 ⋮ q 0 , n − 1 z ( N ) 0 + q 1 , n − 1 z ( N ) 1 + ⋯ + q n , n − 1 z ( N ) n = 0 } (36)

z ( N ) N + 1 = 0. (37)

By replacing (33) with (18) and (34) with (19), it is readily seen that the scheme (35)-(37) formally coincides with that of the discrete time case. Hence (26)-(29) also hold for z ( N ) l . To prove convergence of the associated GCFs

[ Λ l ( 1 ) ⋮ Λ l ( n ) ] = [ c l ( 1 ) c l + 1 ( 1 ) c l + 2 ( 1 ) ⋯ ⋮ ⋮ ⋮ c l ( n ) c l + 1 ( n ) c l + 2 ( n ) ⋯ d l d l + 1 d l + 2 ⋯ ] ( l = 0 , 1 , 2 , ⋯ ) (38)

we use the same arguments as in the proof of Theorem 4. Notice that ( π k ) k = 0 ∞ satisfies (35). Rearranging yields

− c k ( 1 ) π k π k + n + 1 − ⋯ − c k ( n ) π k + n − 1 π k + n + 1 + 1 = d k π k + n π k + n + 1 ( k = 0 , 1 , 2 , ⋯ ) . (39)

Define γ k = π k + n π k + n + 1 for k = 0 , 1 , 2 , ⋯ . Since − c k ( i ) > 0 for i = 1 , ⋯ , n , k ≥ 0 and d k > 0 for k ≥ 0 , (39) is equivalent to

| c k ( i ) γ k γ k − 1 ⋯ γ k − n | + | c k ( 2 ) γ k γ k − 1 ⋯ γ k − n + 1 | + ⋯ + | c k ( n ) γ k γ k − 1 | + 1 = | d k γ k | ( k = 0,1,2, ⋯ ) .

By Theorem 3, the GCF

[ c l ( 1 ) γ l γ l − 1 ⋯ γ l − n c l + 1 ( 1 ) γ l + 1 γ l ⋯ γ l − n + 1 … c l ( 2 ) γ l γ l − 1 ⋯ γ l − n + 1 c l + 1 ( 2 ) γ l + 1 γ l ⋯ γ l − n + 2 … ⋮ ⋮ c l ( n ) γ l γ l − 1 c l + 1 ( n ) γ l + 1 γ l … d l γ l d l + 1 γ l ⋯ ]

then converges for all l ≥ 0 . Hence the same is true for the GCFs (38). Summarizing we arrive at the following theorem.

Theorem 5. Let c k ( i ) , d k be defined as in (33) and (34), respectively. The limits (38) exist for all l ≥ 0 if the corresponding Markov process is irreducible, regular and positive recurrent. In case of convergence the limiting distribution ( π k ) k = 0 ∞ is a dominated solution of (35) and satisfies the decoupled recursion.

π l + n = − Λ l ( n ) π l + n − 1 − ⋯ − Λ l ( 1 ) π l ( l = 0 , 1 , 2 , ⋯ ) . (40)

For executing (40), we make use of Van der Cruyssen’s algorithm with (36) as initial conditions. The resulting sequence ( z ( N ) k ) k = 0 N can be normalized such that ∑ k = 0 N z ( N ) k = 1 .

Remark. As for discrete-time Markov chains, the key statement of Theorem 5 is that the invariant measure is a dominated solution of the difference equation (35). Again, the GCFs additionally combine the two-step method consisting of considering the truncated system (32) and applying Gaussian elimination or LU decomposition.

As an example we consider the continuous-time Markov chain Y = ( Y t ) t ≥ 0 generated by the Q-matrix Q = ( q i j ) 0 ≤ i , k < ∞ , where

q i k = { a 0 a 1 γ i k = i − 1 , i ≥ 1 a 1 − a 0 − 1 γ i , k = i + 1 1 γ i , k = i + 2 − a 1 − a 0 γ i , k = i = 0 − a 0 a 1 + a 1 − a 0 γ i , k = i ≥ 1 0 , elsewhere (41)

with parameters a 0 > γ > 1 and a 1 ≥ a 0 + 1 . Such a Markov chain may be interpreted as a model for a single-server queueing system with state-dependent arrival and service rates, and with customers arriving in groups of size 1 or 2.

It is easy to check that Q is irreducible. To prove that Y is regular and positive recurrent we make use of Tweedie’s [

∑ k = 0 ∞ q i k z k ≤ − ε (42)

for almost all i. By substituting (41) into (42) we get

1 γ i z i + 2 + a 1 − a 0 − 1 γ i z i + 1 − a 1 − a 0 γ i z i + a 0 a 1 γ i z i − 1 ≤ − ε

for almost all i. The substitution z i = γ i for i = 0 , 1 , 2 , ⋯ implies the condition

( γ − 1 ) ( γ − a 0 ) ( γ + a 1 ) γ ≤ − ε

for almost all i which is true because of a 0 > γ > 1 .

Under these conditions there exists a unique invariant measure of Y (up to multiplication by a constant), which satisfies the following set of equations:

a 0 a 1 γ π 1 − ( a 1 − a 0 ) π 0 = 0 , (43)

a 0 a 1 γ 2 π 2 − a 0 a 1 + a 1 − a 0 γ π 1 + ( a 1 − a 0 − 1 ) π 0 = 0 (44)

and

a 0 a 1 γ k + 3 π k + 3 − a 0 a 1 + a 1 − a 0 γ k + 2 π k + 2 + a 1 − a 0 − 1 γ k + 1 π k + 1 + 1 γ k π k = 0

or, equivalently,

a 0 a 1 π k + 3 − ( a 0 a 1 + a 1 − a 0 ) γ π k + 2 + ( a 1 − a 0 − 1 ) γ 2 π k + 1 + γ 3 π k = 0 (45)

for k = 0 , 1 , 2 , ⋯ . (45) leads to the recursion

π k + 3 = γ ( a 0 a 1 + a 1 − a 0 ) π k + 2 − γ 2 ( a 1 − a 0 − 1 ) π k + 1 − γ 3 π k a 0 a 1 (46)

which is used for forward computation.

Since (45) is a homogenous linear difference equation with constant coefficients a fundamental set of solutions can easily be derived from the roots of its characteristic equation. A simple manipulation yields the following set of linearly independent solutions:

( ( γ a 0 ) k ) k = 0 ∞ , ( ( − γ a 1 ) k ) k = 0 ∞ , ( γ k ) k = 0 ∞ .

The desired invariant measure ( π k ) k = 0 ∞ of Y with initial value π 0 = 1 is a linear combination of the first two solutions, that is:

π k = a 1 a 0 + a 1 ( γ a 0 ) k + a 0 a 0 + a 1 ( − γ a 1 ) k ( k = 0 , 1 , 2 , ⋯ ) .

Therefore, choosing γ > 1 yields that there is a geometrically increasing solution of (46), which explains why forward computation cannot cope with this problem, whereas the GCF-based algorithm becomes a stable procedure. This effect is reflected in

k | Forward computation | GCF based solution | Exact solution |
---|---|---|---|

0 | 1 | 1 | 1 |

1 | 0.171429 | 0.171429 | 0.171429 |

2 | 0.286531 | 0.286531 | 0.286531 |

3 | 0.0932012 | 0.0932012 | 0.0932012 |

4 | 0.0896566 | 0.0896566 | 0.0896566 |

5 | 0.0393357 | 0.0393357 | 0.0393357 |

17 | 9.85074 × 10 − 5 | 9.85075 × 10 − 5 | 9.85075 × 10 − 5 |

18 | 5.93420 × 10 − 5 | 5.93424 × 10 − 5 | 5.93424 × 10 − 5 |

19 | 3.55023 × 10 − 5 | 3.55035 × 10 − 5 | 3.55035 × 10 − 5 |

20 | 2.13423 × 10 − 5 | 2.13458 × 10 − 5 | 2.13458 × 10 − 5 |

21 | 1.27782 × 10 − 5 | 1.27888 × 10 − 5 | 1.27888 × 10 − 5 |

22 | 7.64963 × 10 − 6 | 7.68128 × 10 − 6 | 7.68128 × 10 − 6 |

23 | 4.51039 × 10 − 6 | 4.60533 × 10 − 6 | 4.60533 × 10 − 6 |

24 | 2.47987 × 10 − 6 | 2.76467 × 10 − 6 | 2.76467 × 10 − 6 |

25 | 8.03766 × 10 − 7 | 1.65817 × 10 − 6 | 1.65817 × 10 − 6 |

26 | − 1.56804 × 10 − 6 | 9.95173 × 10 − 7 | 9.95173 × 10 − 7 |

27 | − 7.09265 × 10 − 6 | 5.96988 × 10 − 7 | 5.96988 × 10 − 7 |

28 | − 2.27107 × 10 − 5 | 3.58242 × 10 − 7 | 3.58242 × 10 − 7 |

29 | − 6.89918 × 10 − 5 | 2.14924 × 10 − 7 | 2.14924 × 10 − 7 |

30 | −0.000207491 | 1.28964 × 10 − 7 | 1.28964 × 10 − 7 |

40 | −12.2598 | 7.79771 × 10 − 10 | 7.79771 × 10 − 10 |

50 | −723,927 | 4.71497 × 10 − 12 | 4.71497 × 10 − 12 |

k | Forward computation | GCF based solution | Exact solution |
---|---|---|---|

0 | 1 | 1 | 1 |

1 | 0.15 | 0.15 | 0.15 |

2 | 0.6975 | 0.6975 | 0.6975 |

3 | 0.205875 | 0.205875 | 0.205875 |

4 | 0.501694 | 0.501694 | 0.501694 |

5 | 0.21422 | 0.21422 | 0.21422 |

13 | 0.127849 | 0.127848 | 0.127848 |

14 | 0.132884 | 0.132882 | 0.132882 |

15 | 0.106256 | 0.10623 | 0.10623 |

16 | 0.105863 | 0.10563 | 0.10563 |

17 | 0.0896527 | 0.0875497 | 0.0875497 |

18 | 0.10336 | 0.0844324 | 0.0844324 |

19 | 0.242111 | 0.0717609 | 0.0717609 |

20 | 1.60091 | 0.067756 | 0.067756 |

21 | 13.857 | 0.058602 | 0.058602 |

22 | 124.24 | 0.0545256 | 0.0545256 |

23 | 1117.72 | 0.0477352 | 0.0477352 |

24 | 10,059.1 | 0.0439651 | 0.0439651 |

25 | 90,531.2 | 0.038816 | 0.038816 |

Consider next the M/M/1 queueing system, in which the customers arrive according to a Poisson stream with rate λ > 0 and in which the successive service times are stochastically independent and exponentially distributed with parameter μ > 0 . For this system it is known that the queueing process L = ( L ( t ) ) t ≥ 0 which counts the number of customers in line, forms a homogeneous continuous time Markov chain with state space ℕ 0 = { 0,1,2, ⋯ }

being regular and positive recurrent for ρ = λ μ < 1 , see [

The invariant measure ( π k ) k = 0 ∞ of L meets the second order linear difference equation

λ π k − 1 − ( λ + μ ) π k + μ π k + 1 = 0 ( k = 1 , 2 , ⋯ ) (47)

subject to the side conditions

π 0 = 1 and − λ π 0 + μ π 1 = 0.

In principle, ( π k ) k = 0 ∞ could be computed by forward computation, that is

π 0 = 1 ,

π 1 = ρ π 0 ,

π k + 1 = ( ρ + 1 ) π k − ρ π k − 1 ( k = 1 , 2 , ⋯ ) ,

where ρ = λ μ < 1 .

A simple induction argument shows that π k = ρ k for k = 0 , 1 , 2 , ⋯ . Another solution of (47) is given by π ˜ k = 1 for k = 0 , 1 , 2 , ⋯ , implying that ( π k ) k = 0 ∞ becomes a dominated solution of (47), as claimed. Therefore ( π k ) k = 0 ∞ can also be characterized by the decoupled recursion (40), i.e.

π k + 1 = − Λ k ( 1 ) π k ( k = 0 , 1 , 2 , ⋯ ) . (48)

Substituting of (48) into (47) leads to

Λ k − 1 ( 1 ) = − 1 ( 1 + ρ − 1 ) + ρ − 1 Λ k ( 1 ) ( k = 1 , 2 , ⋯ ) . (49)

By truncating this infinite continued fraction scheme at a certain level k = N , we get approximants for the desired coefficients Λ k ( 1 ) . A numerical example is given in

For Markov chains with a specific transition structure, we have proven that the

k | Forward computation | GCF based solution | Exact solution |
---|---|---|---|

0 | 1 | 1 | 1 |

1 | 0.333333 | 0.3333 | 0.3333 |

2 | 0.111111 | 0.1111 | 0.1111 |

3 | 0.03704 | 0.03704 | 0.03704 |

4 | 0.01235 | 0.01235 | 0.01235 |

5 | 0.004115 | 0.004115 | 0.004115 |

6 | 0.001372 | 0.001372 | 0.001372 |

7 | 4.572 × 10 − 4 | 4.572 × 10 − 4 | 4.572 × 10 − 4 |

8 | 1.524 × 10 − 4 | 1.524 × 10 − 4 | 1.524 × 10 − 4 |

10 | 1.694 × 10 − 5 | 1.694 × 10 − 5 | 1.694 × 10 − 5 |

12 | 1.882 × 10 − 6 | 1.882 × 10 − 6 | 1.882 × 10 − 6 |

15 | 6.969 × 10 − 8 | 6.969 × 10 − 8 | 6.969 × 10 − 8 |

20 | 2.868 × 10 − 10 | 2.868 × 10 − 10 | 2.868 × 10 − 10 |

25 | 1.180 × 10 − 12 | 1.180 × 10 − 12 | 1.180 × 10 − 12 |

30 | 4.870 × 10 − 15 | 4.857 × 10 − 15 | 4.857 × 10 − 15 |

40 | 1.312 × 10 − 17 | 8.225 × 10 − 20 | 8.225 × 10 − 20 |

50 | 1.304 × 10 − 17 | 1.393 × 10 − 24 | 1.393 × 10 − 24 |

90 | 1.304 × 10 − 17 | 1.146 × 10 − 43 | 1.146 × 10 − 43 |

invariant measure is a dominated solution of the corresponding steady-state recurrence relations, and therefore, it cannot be calculated by forward computation. As a by-product of our considerations, a GCF-based method for computing the invariant measure in a numerically stable way has been established. The effects of numerical instability of the original recurrence relations as well as the stability of the decoupled version have been illustrated by numerical examples. In some way, the GCF-based approach is similar to previously recommended methods (truncate the infinite system and solve the truncated one by Gaussian elimination or LU decomposition), but represents the steps in a combined mathematical form. Note that in previous literature, the generality of the transition structure did not allow forward computation, and hence, the question of instability of this method did not arise.

At no point of our consideration, we used stochasticity of P (or the corresponding property of conservativity of Q) explicitly. Therefore, it seems reasonable to extend our results to more general infinite systems S x = 0 of linear equations, where

S = ( s 00 s 01 s 10 s 11 s 12 s 20 s 21 s 22 s 23 ⋯ ⋯ ⋯ ⋱ ⋱ s n 0 s n 1 ⋯ ⋯ s n n s n , n + 1 s n + 1,0 s n + 1,1 ⋯ ⋯ s n + 1, n + 1 s n + 1, n + 2 ⋱ ⋱ ⋯ ⋯ ⋱ ⋱ ) .

In the context of computing invariant measures of Markov chains, we have S = I − P T or S = Q T , respectively. In the general case, the main steps of the proofs of Theorem 4 or Theorem 5 can be restated in an abstract context as follows: If s k k > 0 and s k , k + 1 < 0 for k = 0 , 1 , 2 , ⋯ , and s k l ≤ 0 for k = 1 , 2 , 3 , ⋯ and l < k , and if there is a strictly positive solution x of S x = 0 , all GCFs involved in the algorithmic procedure converge. If additionally, we have

lim N → ∞ x ( N ) n x ( N ) 0 = x n x 0 for all n = 0 , 1 , 2 , ⋯ for the solutions x ( N ) of x ( N ) ⋅ S ( N ) = e ( N ) 0 ,

then the GCF-based algorithm converges to x as N tends to infinity, see [

・ In the tridiagonal case (that is, n = 1 ), a generalization to block-matrices (that is, the corresponding Markov chain is a quasi-birth-death process) by means of matrix-valued continued fractions has been studied in [

・ Instead of banded matrices, we could consider upper Hessenberg matrices P (or lower Hessenberg matrices S). Then the difference equation (4) of order n is replaced by a difference equation of infinite order (sometimes referred to as sum equation). Furthermore, arbitrary lower bandwidthes for P could be considered (as in [

We acknowledge support by Open Access Publishing Fund of Clausthal University of Technology.

Baumann, H. and Hanschke, T. (2017) Inherent Numerical Instability in Computing Invariant Measures of Markov Chains. Applied Mathematics, 8, 1367-1385. https://doi.org/10.4236/am.2017.89101