Several New Line Search Methods and Their Convergence

In this paper, we propose several new line search rules for solving unconstrained minimization problems. These new line search rules can extend the accepted scope of step sizes to a wider extent than the corresponding original ones and give an adequate initial step size at each iteration. It is proved that the resulting line search algorithms have global convergence under some mild conditions. It is also proved that the search direction plays an important role in line search methods and that the step size approaches mainly guarantee global convergence in general cases. The convergence rate of these methods is also investigated. Some numerical results show that these new line search algorithms are effective in practical computation.


Introduction
Consider an unconstrained minimization problem   min , , where is an n-dimensional Euclidean space, a continuously differentiable function.
Throughout this paper, we use     as the gradient function of   f x .Given an initial point 0 x , line search methods for solving (1) take the form 1 , 0,1,2, where k x is the current iterate, k a search direction, and  On the one hand, the search direction is generally required to satisfy the descent condition which guarantees that k is a descent direction of d   f x at k x .In order to ensure the global convergence of line search methods, we often require that the following condition holds, , where   0,1 c  is a constant.The condition (4) is sometimes called the angle property (e.g., [1,2]).The choice of search direction k plays an important role in designing line search methods (e.g., [3]).There are many techniques for choosing the search direction at the kth iteration (e.g., [2,4,5]).
Remark.The original Armijo line search rule is to set k s s  with s being a constant [8].
Goldstein rule.A fixed scalar 1 0, 2 and k  is chosen to satisfy   It is possible to show that if is bounded below, then there exists an interval of step sizes k f  for which the relationships above are satisfied, and there are fairly simple algorithms for finding such a step size through a finite number of arithmetic operations.
Wolfe Rule.Choose k and where  and  are some scalars with and .

 
,1 For the Wolfe rule, we assume that there is a scalar M such that    and , and assume that .Then there exists an interval  . The above three line search rules can guarantee the existence of k  under some mild conditions.However, how to find k  is still a question.Especially, how to choose the initial step size k s in the Armijo rule is also very important in practical computation.In fact, how to solve the inequalities ( 6), (7), ( 8) is also a problem in computation.Some implementable modified Armijo rules were proposed [10][11][12][13].Moreover, some nonmonotonic line search methods were also investigated [14][15][16][17].However, can we find an approach to solve (6), (7), and (8) easily and economically?Sometimes, we first set an initial step size k s and substitute the test step size k s   into the inequalities ( 6), (7), or (8); if this  satisfies the inequalities, then we stop and find a step size k    ; otherwise, we need to use back-tracking or forward-tracking to adjust the test step size until we find an accepted step size k  .
In order to find k  easily and economically, we need to solve three problems.One problem is how to estimate the initial step size k s .The second problem is how to adjust the test step size when the test step size doesn't satisfy the inequalities.The third problem is whether we can extend the accepted scope of step sizes to a wider extent.Our research is focused on the second and third questions.
In this paper, we propose several line search rules for solving unconstrained minimization problems.The modi-fied line search rules used in the methods can extend the range of acceptable step sizes and give a suitable initial step size at each iteration.It is proved that the resulting line search methods have global convergence under some mild conditions.It is also proved that the search direction plays an important role in line search methods and that the step size rule mainly guarantees the global convergence in general cases.Numerical results show that the resulting algorithms are effective in practical computation.
The remainder of the paper is organized as follows.In Section 2, we describe the modified line search rules and its properties.In Section 3, we analyze the global convergence of resulting line search methods, and in Section 4, we study further the convergence rate of the new line search methods.Numerical results are reported in Section 5.

Modified Line Search Rules
We first assume that (H1).The function has a lower bound on f It is apparent that (H3) implies (H2).
In the following three modified line search rules, we define where , and set 2 .
Modified Goldstein Rule.A fixed scalar 1 0, 2 Modified Wolfe Rule.The step size k and where  and  are some scalars with 1 0, 2 and .
Therefore, if (H1) holds, then the three modified line searches are feasible.As a result, the modified line searches can extend the range of acceptable step sizes k  .
For the above three modified line search rules, we denote the three corresponding algorithms by Algorithm (NA), Algorithm (NG), and Algorithm (NW), respectively.

Global Convergence
In this section, we will prove that if (H1) and (H2) hold,  is chosen so that (11), or (12), or (13) and ( 14 and by (11), we have Since k  is the largest one to satisfy the modified Armijo rule, we will have By the mean value theorem on the left-hand side of the above inequality, we can find By (H1), (3), and (11), it follows that   k f is a nonincreasing number sequence and bounded from below, and it has a limit.Furthermore, we get from (11) that and thus, In order to prove (15), we use contrary proof to absurdity.Assume that there exists an and an infinite subset  is an infinite subset then by ( 21) and ( 20) we have The contradiction shows that 1 3 K K  is not an infinite subset and 2 3 K K  must be an infinite subset.By ( 21) and ( 20), we have By the Cauchy-Schwartz inequality and ( 18), we obtain , and the above inequality, we have 2) which also contradicts (21).The contradiction shows that (15) holds.Theorem 3.2 Assume that (H1), (H2), and (3) hold.Algorithm (NG) with the modified Goldstein line search generates an infinite sequence   k x .Then (15) holds.Proof.By using the mean value theorem on the lefthand side inequality of ( 12), there exists By the right-hand side of ( 12) and (H1), it follows that   k f is a monotone decreasing sequence and bounded below, and thus it has a limit.This shows that Using the contrary proof, if (15) doesn't hold, then there exists an infinite subset and an 0,1, By (25) and (24), we obtain By the Cauchy-Schwartz inequality and ( 23), we have By ( 26) and (H2), and noting that  and the above inequality, we have which contradicts (25).The contradiction shows that (15) holds.Theorem 3.3 Assume that (H1), (H2), and (3) hold.Algorithm (NW) with modified Wolfe rule generates an infinite sequence   k x .Then (15) holds.Proof.Using contrary proof, if (15) doesn't hold, then there exists an infinite subset and an such that (25) holds.By (H1) and (13), it follows that (24) holds, and thus (26) holds.
Proof.Since (H3) implies (H2), the conclusions in Theorems 3.1, 3.2, and 3.3 are also true.We will use these conclusions and notations in the proofs of these theorems to our proof.
For Algorithm (NA), by (H3), the Cauchy-Schwartz inequality, and (18), we have where 2 K and k  are defined in the proof of Theorem 3.1.Thus, By ( 17), (28), and the proof of Theorem 3.1, we obtain where By ( 16) and the proof of Theorem 3.1, we have where 1 K is defined in the proof of Theorem 3.1.Set it follows that (27) holds.

Convergence Rate
In order to analyze the con discussion to the case of uniformly convex objective functions.We further assume that (H4): f is twice continuously differentiable and unifo nv rmly co ex on n R .Lemma 4.1 Ass e that (H4) holds.Then (H1) and (H3) hold,   f x has a unique minimizer * x , and there exists , , ; converges to * f at least -linearly.From the last inequality, we g that By ( 37) and ( 38), we can also obtain from the Cau Schwartz inequality that chy- and Its proof can be seen from (e.g., [18,19], etc.).case, the Lipschitz constant of the gradient function

In this
By the above inequality and setting x converges to at least

Numerical Results
we i retical e he gl ge ted line search methods r some ion, we will stud numehms with the ne rch tively.
ion is R -linearly [18].a ec In the above sections, nvestigated the theo prop rties and analyzed t obal convergence d convernce rate of rel an unde y the w line sea mild conditions.In this sect rical performance of algorit approaches.
First, we choose some numerical examples to test the Algorithms (NA), (NG), and (NW) and make some comparisons to the algorithms with the original line searches.The original line search methods are denoted by OA, OG, and OW, resp The numerical examples are from [12].We use the same symbols to denote the problems.For example, (P5) denotes Problem (P5) in [12].For each problem, the limiting number of functional evaluations is set to 10,000, and the stopping condit 6 10 .
We choose a portable computer with a Pentium 1.2 MHz CPU and Matlab 6.1 to test our algorithms.The parameters are set to 0.38 .Set fied Ar m Step 3

, go to St
For the Goldstein and the modified Goldstein rules, we us e e the following procedur to find the step size.This shows that the modified line searches are effective in practical computation and significantly reduce the number of functional evaluations and iterations when reaching the same precision.Moreover, we found that the new modified line approaches can be used to any descent methods.For example, we can take quasi-Newton direc- in the line search methods and use these modified line search approaches to find a step size.
s and functional evaluations.[20][21][22] to test the new modified line search rules.
The numerical results are reported in Table 3.
For large scale problems, numerical results show that the resulting algorithms with the modified line search rules are more effective than the original ones in many cases.The reason is that step size estimation is more useful for such large scale problems that have sparse Hessian matrix.Larger step size plays an important role in the convergence of resulting algorithms with modified line searches.Therefore, initial step size estimation and extending the range of acceptable step sizes is necessary in line search design and algorithm design.

Conclusions
We proposed several new line search algorithms for solving unconstrained minimization problems.The modified line search rules used in the methods c range of acceptable step sizes a suitable initial ste give that the resulting line search algorithms have global convergence under weak conditions.It is also proved that the search direction plays an important role in these methods and that the step size mainly guarantees global convergence in some cases.The convergence rate of these algorithms is investigated.These theoretical results can help us design new algorithms in practice.Furthermore, we extended the line search methods theoretically in some broad sense.Numerical results show that these new modified line search rules are useful and effective in practical computation.
For the future research, we should study the numerical performance of special line search methods for largescale practical problems.Moreover, we can generalize these modified line search approaches to nonmonotone cases [14,16].We can also investigate step-size in different ways [23][24][25].


a step size.Let * x be a minimizer of (1) and thus be a stationary point that satisfies   .In line search methods, there are two things to do at each iteration.One thing is to find a search direction k , and the other is to choose the step size k On the other hand, we should choose k  to satisfy some line search rules.Line search rules can be classified into two types, exact line search rules and inexact line search rules.This paper is devoted to the case of inexact line search rules.There are three well-known inexact line search rules[6][7][8][9].

Table s and functional evaluations.
even ill-conditioned problems, what is the perfor ance of the resulting algorithms with new modified line search rules?We chose standard test problems from