Three New Hybrid Conjugate Gradient Methods for Optimization *

In this paper, three new hybrid nonlinear conjugate gradient methods are presented, which produce sufficient descent search direction at every iteration. This property is independent of any line search or the convexity of the objective function used. Under suitable conditions, we prove that the proposed methods converge globally for general nonconvex functions. The numerical results show that all these three new hybrid methods are efficient for the given test problems.


Introduction
In this paper, we consider the unconstrained optimization problem: where is continuously differentiable and the gradient of f at x is denoted by   g x .Due to its simplicity and its very low memory requirement, the conjugate gradient (CG) method plays a very important role for solving (1.1).Especially, when the scale is large, the CG method is very efficient.Let 0 n x R  be the initial guess of the solution of problem (1.1).A nonlinear conjugate gradient method is usually designed by the iterative form 1 , 0,1, , where k x is the current iterate point, 0 k   is a steplength which is determined by some line search, and k d is the search direction defined by where k g denotes   k g x , and k  is a scalar.There are some well-known formulas for k  , which are given as follows: , where   and  stands for the Euclidean norm of vectors.
Although the methods above are equivalent [8,9] when f is a strictly convex quadratic function and k  is calculated by the exact line search, their behaviors for general objective functions may be far different.Generally, in the convergence analysis of conjugate gradient methods, one hopes the inexact line search such as the Wolfe conditions, the strong Wolfe conditions or the strong *Wolfe conditions,which are showed respectively as follows: 1) The 2) The strong Wolfe line search is to find k

   
For general functions, Zoutendjk [10] and Al-Baali [11] had proved the global convergence of the FR method with different line searches.And Powell [12] gave a counter example which showed that there exist nonconvex functions such that the PRP method may cycle and does not approach any stationary point even with exact line search.Although one would be satisfied with its global convergence, the FR method performs much worse than the PRP (HS, LS) method in real computations.In other words, in practical computation, the PRP method, the HS method, and the LS method are generally believed to be the most efficient conjugate gradient methods since these methods essentially perform a restart if a bad direction occurs.A similar case happens to the DY method and the CD method.That is to say the convergences of the CD , DY and FR methods are established [1][2][3], however their numerical results are not so well.Resently, some good results on the nonlinear conjugate gradient method are given.Combining the good numerical performance of the PRP and HS methods and the nice global convergence properties of the FR and DY methods, recently, [13] and [14] proposed some hybrid methods which we call the H1 method and the H2 method, respectively, that is, Gilbert and Nocedal [15] extended H1 to the case that Numerical performances show that the H1 and the H2 methods are better than the PRP method [13,14,16].
As we all know, the FR , DY and CD methods are descent methods, but their descent properties depend on the line search such as the strong Wolfe line search (1.5).Similar to the descent three terms PRP method in [17], Zhang et al. [18,19] proposed a modified FR method which we call the MFR method, that is, And [18] also gave an equivalent form to the MFR method.Similarly, Zhang [19] also proposed a modified DY method called the MDY method, that is, It is easy to see that the MFR and MDY methods have an important property that the search directions satisfy 2 , which depends neither on the line search used nor on the convexity of the objective function; moreover these two methods reduce to the FR method and the DY method respectively with exact line search.
[18] has explored the convergence and efficiency of the MFR method for nonconvex functions with the Wolfe line search or Armijo line search.Based on the idea of the H1 and the H2 methods, recently, Zhang-Zhou [20]  1 Obviously, these two new hybrid methods still satisfy 2 , which shows that they are descent and independent of any line search used.[20] proved the global convergence of these two methods and also showed their efficiency in real computations.Similarly,based on the idea of the methods all above, we consider the CD method, and propose three new hy-brid conjugate gradient(CG) methods which we call the H3 method, the MCD method and the NH3 method, respectively, that is From these three methods above, it is not difficult to see that the MCD and the NH3 methods also satisfy 2 , which shows that they are sufficient descent methods.In the next section, the new algorithms are given.The global convergence of the proposed methods are proved in Section 3. We give the numerical experiments in Section 4, and in Section 5, the conclusion is presented.

Algorithm 1 (The H3 Algorithm)
Step 0: Choose an initial point 0 , Step 1: If , k g   then stop; Otherwise go to the next step.
Step 2: Compute step size k  by strong *Wolfe line search rule (1.6).
Step 3: Let Step 4: Calculate the search direction .
Step 5: Set : 1, k k   and go to Step 2.

Algorithm 2 (The MCD (Or the NH3) Algorithm)
Step 0: Choose an initial point 0 , Step 1: If , k g   then stop; otherwise go to the next step.
Step 2: Compute step size k  by Wolfe line search rule (1.4).
Step 5: Set : 1 , k k   and go to Step 2.

The Global Convergence
Assumption A 1) The level set In an open convex set N that contains  , f is continuously differentiable and its gradient g is Lipschitz continuous, namely, there exists a constant 0 x generated by Algorithm 1 and Algorithm 2 is contained in  .In addition, we can get from Assumption A that there exists a constant B and 1 0, In the latter part of the paper, we always suppose that the conditions in Assumption A hold.Then there is an useful lemma, which was originally given in [10,21].We now establish the global convergence theorem for Algorithm 1 and Algorithm 2 .

The Global Convergence of the H3 Method
For simlpicity, here, we list the Theorem 2.3 in [14] as the following Lemma 3.2 without proof.
Lemma 3.2 Suppose that 0 x is an initial point, Consider the method (1.2) and (1.
Further, the method converges in the sense that lim inf 0.
From the Lemma 3.2 above, similar to Corollary 2.4 in [14], we give the global convergence of the H3 method(Algorithm 1).
Theorem 3.3 Suppose that 0 x is an initial point, Consider the Algorithm1, then we have either 0 From the second inequality of (1.6) and the definitions of 3 , Therefore the statement follows lemma 3.2.

The Global Convergence of the MCD Method
Now, we establish the global convergence theorem for the MCD method.Proof Suppose by contradiction that the desired conclusion is not true, that is to say, there exists a constant 0 and then we have Dividing both sides of (3.7) by   The last inequality implies which contradicts (3.4).The proof is then completed.

The Global Convergence of the NH3 Method
The same as the Theorem 3.4, we can establish the following global convergence theorem for the NH3 method.Proof Suppose by contradiction that the desired conclusion is false, that is to say, there exists a constant 0 Similar to (3.7), we get from (1.16) that where 3 1 2 1 .
Dividing both sides of (3.10) by   2 T k k g d , we get, from (3.11), (1.13) and the definition of 3 ,  which contradicts (3.4).This finishes the proof.

Numerical Experiments
In this section, we carry out some numerical experiments.These three algorithms have been tested on some problems from [22].The results are summarized in the following three tables: x the final point, k the number of times of iteration for each problem.
These three tables show the performance of these three methods relative to the iterations, It is easily to see that, for each algorithm, are all very efficient, especially for the problems such as s201, s207, s240, s311.The results for each problem are accurate, and with less number of times of iteration.

Conclusions
We have proposed three new hybrid conjugate gradient (CG) methods, that are, the H3 method, the MCD me-   thod and the NH3 method, where the last two methods produce sufficient descent search direction at every iteration.This property depends neither on the line search used nor on the convexity of the objective function.Under suitable conditions, we proposed the global convergence of these three new methods even for nonconvex minimization.And numerical experiments in section 4 showed that the new three algorithms are all efficient for the given test problems.


in (1.7) and H2 k  in (1.8), respectively, and proposed two new hybrid PRP-FR and HS-DY methods called the NH1 method and the NH2 method, respectively, that is,

Lemma 3 . 1 From
Let   k x be generated by (1.2) and k d is a descent direction.If k  is determined by the Wolfe line search (1.4), then we have Lemma 3.1 and (1.13) that for the MCD and NH3 methods with the Wolfe line search,we can easily obtain the 3), where k  is computed by the Wolfe line search(1.4),and k

Theorem 3 . 4
Let   k x be generated by the MCD method (Algorithm 2, where k d satisfies (1.15)) 13) and the definition of CD k

Theorem 3 . 5
Let   k x be generated by the NH3 method (Algorithm 2, where k d satisfies (1.16) )

Table 1 - 3 .
[22]each test problem, No. is the number of the test problem in[22], 0x is the initial point, k