Global Convergence of Curve Search Methods for Unconstrained Optimization

In this paper we propose a new family of curve search methods for unconstrained optimization problems, which are based on searching a new iterate along a curve through the current iterate at each iteration, while line search methods are based on finding a new iterate on a line starting from the current iterate at each iteration. The global convergence and linear convergence rate of these curve search methods are investigated under some mild conditions. Numerical results show that some curve search methods are stable and effective in solving some large scale minimization problems.


Introduction
Line search method is an important and mature technique in solving an unconstrained minimization problem where n R is an n-dimensional Euclidean space and where k d is a descent direction of ( ) x and k α is a step size to satisfy the descent condition ( ) ( ), 0,1, 2, Z. W. Xu et al.
One hopes that { } k x generated by line search method converges to the minimizer * x of (1) in some sense.Let k x be the current iterate.We denote ( ) At the k-th iteration of line search methods, one first chooses a search direction and then seeks a step size along the search direction and completes one iteration (see [1]).The search direction k d is generally required to satisfy 0, which guarantees that k d is a descent direction of ( ) x (e.g. [2] [3]).In order to guarantee the global convergence, we sometimes require k d to satisfy the sufficient descent condition 2 , where 0 c > is a constant.Moreover, the angle testing condition is commonly used in proving the global convergence of related line search methods, that is cos , , where 1 0 η ≥ > .
In line search methods we try to find an α to reduce ( ) = + > at the kth iteration, while curve search method is to define the next iterate on the curve where [ ) obvious that line search method is a special one of curve search methods.In other words, curve search method is a generalization of line search methods.
McCormick [4] and Israel Zang [5] proposed an arc method for mathematical programming, which is actually a special one of curve search methods.Similarly as in line search methods, how to choose a curve at each iteration is the key to using curve search methods.
Botsaris [6]- [9] studied differential gradient method (abbreviated as ODE method) for unconstrained minimization problems.It is required to solve differential equations at the k-th iteration and ( ) ( ).
The ODE method has been investigated by many researchers (e.g.[10]- [14]) and is essentially a curve search method.
However, it is required to solve some initial-value problems of ordinary differential equations to define the curves in ODE methods.Some other curve search methods with memory gradient have also been investigated and been proved to be a kind of promising methods for large-scale unconstrained optimization problems (see [15] [16]).Other literature on curve search methods have appeared in the literature [17]- [19].To the best of our knowledge, the unified form of curve search methods has rarely been studied in present literature.It is necessary to study the general scheme of curve search methods and its global convergence.
In this paper we present a new family of curve search methods for unconstrained minimization problems and prove their global convergence and linear convergence rate under some mild conditions.These method are based on searching a new iterate along a curve at each iteration, while line search methods are based on finding a new iterate on a line starting from the current iterate at each iteration.Many curve search rules proposed in the paper can guarantee the global convergence and linear convergence rate of these curve search methods.Some implementable version of curve search methods are presented and numerical results show that some curve search methods are stable, useful and efficient in solving large scale minimization problems.
The rest of this paper is organized as follows.In the next section we describe the curve search methods.In Sections 3 and 4 we analyze its global convergence and linear convergence rate respectively.In Section 5 we report some techniques for choosing the curves and conduct some numerical experiments.And finally some conclusion remarks are given in Section 6.

Curve Search Method
We first assume that (H1).The objective function

( )
f x is continuously differentiable on n R and the level set ≤ is bounded, where 0 x is given.
(H1').The gradient ( ) ( ) Definition 2.2.We call the one-dimensional function satisfies (H2).This kind of curves are easy to find.For example, are curves that satisfy (H2) and so are the following curves { } k y α satisfies (H2).As a result, line search method is a special one of curve search methods and its convergence can be derived from the convergence of curve search methods.
In the sequel, we describe the curve search method.Algorithm (A).
Proof.Assumption (H1) and Definition 2.1 imply that ( ) g x is uniformly continuous on B and thus, there exist 0 M and a forcing function ( ) , 0 0 , .
In the following we prove that the curve search rules (e), (f) and (g) are also well-defined.
For the curve search rule (e), (H1) and which shows that the curve search rule (e) is well-defined.
For the curve search rules (f) and (g), (H1) and ( ) must have an intersection point and suppose that ( ) where . Using the mean value theorem, there exists . .
In the case of 0 0 η = , there must exist an infinite subset 1 Therefore, for sufficiently large Using the mean value theorem on the left-hand side of the above inequality, there exists and thus By ( 22), (23) and Lemma 2.1, we have which also contradicts (20).
For the curve search rules (a), (b) and (d), since 0 0 η > for the curve search rule (c), let k α be the step size defined by the three curve search rules (a), (b) and (d), and let k α ′ be the step size defined by the curve search rule (c), then we have , .
In the case of 0 0 η = , there must exist an infinite subset 1 K K ⊆ such that (22) holds.For the curve search rule (e), by the left-hand side inequality of (11) and using the mean value theorem, there exists By ( 22), (24) and Lemma 2.1, we have which contradicts (20).For the curve search rules (f) and (g), by ( 14), ( 22) and Lemma 2.1, we have which also contradicts (20).

Convergence Rate
In order to analyze the rate, we further assume that (H3).The sequence { } k x generated by curve search method converges to * x , ( ) ( ) , , , ; , , , ; and thus By ( 28) and ( 27) we can obtain, from the Cauchy-Schwartz inequality , that , .
Proof.We first prove that (31) holds for the curve search rules (c), (e), (f) and (g), and then we can prove (31) also holds for the curve search rules (a), (b) and (d).
By ( 35), ( 23), ( 24), ( 14), (34) and Lemma 2.1, we have ( The contradiction shows that 0 0 η = does not occur and thus 0 0 η > .By letting we can obtain the conclusion. For the curve search rules (a), (b) and (d), let * k α denote the exact step size and k α denote the step size generated by the curve search rule (c).By the previous proof, we . x at least R-linearly.
Proof.By Lemmas 4.1 and 4.2 we obtain ( ) we can prove that 1.

θ <
In fact, by the definition of η in the proof of Lemma 4.2 and (36), we obtain and knowing 1 ω < , we obtain from the above inequality that (  By Lemma 4.1 and the above inequality we have x at least R-linearly.□

How to Find Curves
In order to find some curves satisfying Definition 2.1 and (H2), we first investigate the slope and curvature of a curve.Given a curve ( ) Generally, we require ( ) where 0 1 c < ≤ and 0 1 c > .Moreover, we expect the curves to satisfy It is worthy to point out that many convergence properties of curve search methods remain hold for line search method.In fact, the line ( ) where m is a positive integer and , 2, 3, , . 1


We can prove that ( ) k y α satisfies Definition 2.1 and (H2) under some mild conditions.Numerical results showed that the curve search method was more efficient than some line search methods [16].
Another curve search method is from [15] with the curve defined by ( ) ( ) This curve also satisfies Definition 2.1 and (H2) with ( ) satisfying (5) under certain conditions and has good numerical performance.
Moreover, many researchers take ( ) [20].Certainly, we can obtain some curves by solving initial problems or boundaryvalue problems of ordinary differential equations and sometimes by using interpolation technique.Lucidi, Ferris and Roma proposed a curvilinear truncated Newton method which uses the curve with k s being the quasi-Newton direction and k d being the steepest descent direction.This method also has good numerical performance [18] [19] because it reduces to quasi-Newton method finally and avoids some disadvantages of quasi-Newton method at the initial iterations.We guess that there may be many curve search methods which are superior to line search methods in numerical performance.
For example, if we take k k d g = − and suppose that { } k q and { } k p are uniformly bounded for k, then the following curve ( ) In the following we shall test some curve search methods.

Numerical Experiments
In this subsection, some numerical reports are prisented for some implementable curve search methods.First of all, we consider some curve search methods with memory gradients.The first curve search method is based on the curve ( ) ( ) The second curve search method is to use the curve and the third curve search method searches along the curve at each iteration ( ) We use respectively the Armijo curve search rule and the Wolfe curve search rule to the above three curves to find a step size at each step.Test problems 21 -35 and their initial iterative points are from the literature [21].For example, Problem 21 stands for the problem 21 in the literature and so on.
In the curve search rules (c) and (g) we set the parameters 0.25, 0.75, 1 L σ ρ = = = and 0.38 β = .Numerical performance of the three curve search methods is reported in Table 1 and a pair of numbers means that the first number denotes the number of iterations and the second number denotes the number of functional evaluations."P" stands for problems, n is the dimension of problems and T denotes total CPU time for solving all the 15 problems.We denote A1, A2 and A3 the curve search methods with the curves (40), (41) and (42) respectively.A1(c) and A1(g) means the A1 algorithm with the curve search rule (c) and the A1 algorithm with  1 that curve search methods with memory gradients converge to the optimal solutions stably and averagely.In addition, curve search methods with the Wolfe curve search rule are superior to the methods with the Armijo curve search rule.This shows that L = 1 seems to be an inadequate choice in the Armijo curve search rule and we can take L variably at each step similarly as in the literature [16].
Moreover, many line search methods may fail to converge when solving some practical problems, especially when solving large scale problems, while curve search methods with memory gradients always converge stably.From this point of view, we guess that some curve search methods are available and promising for optimization problems.

Conclusions
Some curve search methods have good numerical performance and are superior to the line search methods to certain extent.This motivates us to investigate the general convergence properties of these promising methods.
In this paper we presented a class of curve search methods for unconstrained minimization problems and proved its global convergence and convergence rate under some mild conditions.Curve search method is a generalization of line search methods but it has wider choices than line search methods.Several curve search rules were proposed and some approaches to choose the curves were presented.The idea of curve search methods enables us to find some more efficient methods for minimization problems.Furthermore, numerical results showed that some curve search methods were stable, available and efficient in solving some large scale problems.
For the future research, we should investigate more techniques for choosing search curves that contain the information of objective functions and find more curve search rules for the curve search method.
. There are several curve search rules as follows.(a) Exact Curve Search Rule.Select an k α to satisfy

2 .
If (H1) holds and ( ) k y α is defined by Definition 2.1 and ), then k α is well defined in the seven curve search rules.

Theorem 3 . 1 .
Assume that (H1), (H1') and (H2) hold, c > is a constant.If k α is defined by the curve search rules (a), (b), (c) or (d) and Algorithm (A) generates an infinite sequence { } Using reduction to absurdity, suppose that there exist an infinite subset

3 . 2 .
holds for the curve search rules (a), (b) and (d), which contradicts(20).The conclusion is proved.□Theorem Assume that (H1) and (H2) hold, and(19), k α is defined by the curve search rules (e), (f) or (g).Algorithm (A) generates an infinite sequence Using reduction to absurdity, suppose that there exist an infinite subset By Theorems 3.1 and 3.2, we can complete the proof.□

.
We hope that the curve is a descent curve at 0 α = , i.e.

Table 1 .
Iterations and function evaluations.
k g − ≤ It is shown in Table