^{1}

^{2}

^{3}

In this paper we propose a new family of curve search methods for unconstrained optimization problems, which are based on searching a new iterate along a curve through the current iterate at each iteration, while line search methods are based on finding a new iterate on a line starting from the current iterate at each iteration. The global convergence and linear convergence rate of these curve search methods are investigated under some mild conditions. Numerical results show that some curve search methods are stable and effective in solving some large scale minimization problems.

Line search method is an important and mature technique in solving an unconstrained minimization problem

where

where

One hopes that

At the k-th iteration of line search methods, one first chooses a search direction and then seeks a step size along the search direction and completes one iteration (see [

which guarantees that

where

where

In line search methods we try to find an

where

McCormick [

Botsaris [

or to solve

where

However, it is required to solve some initial-value problems of ordinary differential equations to define the curves in ODE methods. Some other curve search methods with memory gradient have also been investigated and been proved to be a kind of promising methods for large-scale unconstrained optimization problems (see [

In this paper we present a new family of curve search methods for unconstrained minimization problems and prove their global convergence and linear convergence rate under some mild conditions. These method are based on searching a new iterate along a curve at each iteration, while line search methods are based on finding a new iterate on a line starting from the current iterate at each iteration. Many curve search rules proposed in the paper can guarantee the global convergence and linear convergence rate of these curve search methods. Some implementable version of curve search methods are presented and numerical results show that some curve search methods are stable, useful and efficient in solving large scale minimization problems.

The rest of this paper is organized as follows. In the next section we describe the curve search methods. In Sections 3 and 4 we analyze its global convergence and linear convergence rate respectively. In Section 5 we report some techniques for choosing the curves and conduct some numerical experiments. And finally some conclusion remarks are given in Section 6.

We first assume that

(H1). The objective function

(H1'). The gradient

Definition 2.1. Let

where

Definition 2.2. We call the one-dimensional function

where

It is obvious that the addition, the multiplication and the composite function of two forcing functions are also forcing functions.

In order to guarantee the global convergence of curve search methods, we suppose that the initial descent direction

(H2). The search curve sequence

where

Remark 1. In fact, if there exist

where

This kind of curves are easy to find. For example,

are curves that satisfy (H2) and so are the following curves

(for

Remark 2. If

cause of

and

Remark 3. In line search methods, if we let

In the sequel, we describe the curve search method.

Algorithm (A).

Step 0. Choose

Step 1. If

Step 2. Let

Step 3. Set

Once the initial descent direction

For convenience, let

(a) Exact Curve Search Rule. Select an

(b) Approximate Exact Curve Search Rule. Select

(c) Armijo-type Curve Search Rule. Set

to be the largest

(d) Limited Exact Curve Search Rule. Set

(e) Goldstein-type Curve Search Rule. Set

(f) Strong Wolfe-type Curve Search Rule. Set

and

(g) Wolfe-type Curve Search Rule. Set

(12) and

Lemma 2.1. Let

where

Proof. Assumption (H1) and Definition 2.1 imply that

By (H2), Definition 2.1 and (15), noting that

□

Lemma 2.2. If (H1) holds and

Proof. Let

implies that there exists

Thus

which shows that the curve search rules (a), (b), (c) and (d) are well-defined.

In the following we prove that the curve search rules (e), (f) and (g) are also well-defined.

For the curve search rule (e), (H1) and

imply that the curve

which shows that the curve search rule (e) is well-defined.

For the curve search rules (f) and (g), (H1) and

imply that the curve

and

where

By (16) we have

and thus,

Therefore,

Obviously, it follows from (17) and (18) that

□

Theorem 3.1. Assume that (H1), (H1') and (H2) hold,

where

Proof. Using reduction to absurdity, suppose that there exist an infinite subset

(H1) implies that

Let

In the case of

because of

By (9) and (21) we have

By (H1) we can obtain

which contradicts (20).

In the case of

Therefore, for sufficiently large

Using the mean value theorem on the left-hand side of the above inequality, there exists

and thus

Hence

By (22), (23) and Lemma 2.1, we have

which also contradicts (20).

In fact, we can prove that

This contradiction shows that

For the curve search rules (a), (b) and (d), since

This and (H1) imply that

holds for the curve search rules (a), (b) and (d), which contradicts (20). The conclusion is proved.

□

Theorem 3.2. Assume that (H1) and (H2) hold,

Proof. Using reduction to absurdity, suppose that there exist an infinite subset

For the curve search rules (e), (f) and (g), in the case of

By (H1) we have

which contradicts (20).

In the case of

Thus

By (22), (24) and Lemma 2.1, we have

which contradicts (20). For the curve search rules (f) and (g), by (14), (22) and Lemma 2.1, we have

which also contradicts (20).

The conclusions are proved.

□

Corollary 3.1. Assume that (H1), (H1') and (H2) hold,

Proof. By Theorems 3.1 and 3.2, we can complete the proof.

□

In order to analyze the convergence rate, we further assume that

(H3). The sequence

positive definite matrix and

Lemma 4.1. Assume that (H3) holds. Then there exist

and thus

By (28) and (27) we can obtain, from the Cauchy-Schwartz inequality , that

and

Its proof can be seen from the book ( [

Lemma 4.2. Assume that (H2) and (H3) hold and

Proof. We first prove that (31) holds for the curve search rules (c), (e), (f) and (g), and then we can prove (31) also holds for the curve search rules (a), (b) and (d).

By (9), (11), (12) and (5), we have

Since

By (4), Cauchy Schwartz inequality and (19), we have

Let

If

Letting

for (23),

for (24) and

for (13), we have

By (35), (23), (24), (14), (34) and Lemma 2.1, we have

The contradiction shows that

we can obtain the conclusion.

For the curve search rules (a), (b) and (d), let

All the conclusions are proved.

□

Theorem 4.1. Assume that (H2) and (H3) hold and

Proof. By Lemmas 4.1 and 4.2 we obtain

By setting

we can prove that

By setting

and knowing

By Lemma 4.1 and the above inequality we have

thus

i.e.,

Therefore,

which shows that

□

In order to find some curves satisfying Definition 2.1 and (H2), we first investigate the slope and curvature of a curve. Given a curve

where

It is worthy to point out that many convergence properties of curve search methods remain hold for line search method. In fact, the line

where m is a positive integer and

We can prove that

Another curve search method is from [

and

This curve also satisfies Definition 2.1 and (H2) with

Moreover, many researchers take

and

with

For example, if we take

satisfies (H2), provided that

In this subsection, some numerical reports are prisented for some implementable curve search methods. First of all, we consider some curve search methods with memory gradients. The first curve search method is based on the curve

The second curve search method is to use the curve

and the third curve search method searches along the curve at each iteration

We use respectively the Armijo curve search rule and the Wolfe curve search rule to the above three curves to find a step size at each step. Test problems 21 - 35 and their initial iterative points are from the literature [

In the curve search rules (c) and (g) we set the parameters

P | n | A1(c) | A1(g) | A2(c) | A2(g) | A3(c) | A3(g) |
---|---|---|---|---|---|---|---|

21 | 10^{4} | 193/1089 | 118/673 | 168/1982 | 132/982 | 145/869 | 156/1421 |

22 | 10^{4} | 254/2736 | 212/1983 | 316/1572 | 247/2195 | 231/1673 | 238/1965 |

23 | 10^{4} | 121/689 | 128/513 | 98/1034 | 122/832 | 117/968 | 146/872 |

24 | 10^{4} | 316/1863 | 235/1493 | 356/1987 | 234/1392 | 198/1326 | 168/1628 |

25 | 10^{4} | 119/628 | 121/892 | 126/916 | 115/1639 | 179/1473 | 105/1034 |

26 | 10^{4} | 178/2134 | 192/2075 | 169/1935 | 142/1432 | 128/1732 | 126/1728 |

27 | 10^{4} | 127/982 | 134/763 | 133/1772 | 152/1827 | 109/913 | 118/1471 |

28 | 10^{4} | 153/918 | 217/1528 | 145/1463 | 143/1367 | 135/1731 | 129/1862 |

29 | 10^{4} | 183/2156 | 137/1985 | 163/3721 | 169/2176 | 165/2191 | 127/1632 |

30 | 10^{4} | 152/962 | 123/1891 | 106/2732 | 136/1472 | 145/1569 | 113/1528 |

31 | 10^{4} | 117/1465 | 109/1394 | 127/1528 | 137/1647 | 134/1841 | 152/1378 |

32 | 10^{4} | 98/1275 | 126/1763 | 129/972 | 104/1166 | 111/1634 | 94/982 |

33 | 10^{4} | 129/863 | 116/1872 | 162/1798 | 181/1744 | 148/1825 | 116/1872 |

34 | 10^{4} | 67/862 | 95/962 | 86/739 | 74/763 | 88/1267 | 85/1621 |

35 | 10^{4} | 432/3721 | 269/2964 | 195/1267 | 342/2374 | 253/1288 | 317/1268 |

T | - | 653s | 436s | 548s | 463s | 553s | 414s |

the curve search rule (g) respectively, and so on. The stop criteria is

It is shown in

Moreover, many line search methods may fail to converge when solving some practical problems, especially when solving large scale problems, while curve search methods with memory gradients always converge stably. From this point of view, we guess that some curve search methods are available and promising for optimization problems.

Some curve search methods have good numerical performance and are superior to the line search methods to certain extent. This motivates us to investigate the general convergence properties of these promising methods.

In this paper we presented a class of curve search methods for unconstrained minimization problems and proved its global convergence and convergence rate under some mild conditions. Curve search method is a generalization of line search methods but it has wider choices than line search methods. Several curve search rules were proposed and some approaches to choose the curves were presented. The idea of curve search methods enables us to find some more efficient methods for minimization problems. Furthermore, numerical results showed that some curve search methods were stable, available and efficient in solving some large scale problems.

For the future research, we should investigate more techniques for choosing search curves that contain the information of objective functions and find more curve search rules for the curve search method.

Zhiwei Xu,Yongning Tang,Zhen-Jun Shi, (2016) Global Convergence of Curve Search Methods for Unconstrained Optimization. Applied Mathematics,07,721-735. doi: 10.4236/am.2016.77066