An Improved Line Search and Trust Region Algorithm

In this paper, we present a new line search and trust region algorithm for unconstrained optimization problems. The trust region center locates at somewhere in the negative gradient direction with the current best iterative point being on the boundary. By doing these, the trust region subproblems are constructed at a new way different with the traditional ones. Then, we test the efficiency of the new line search and trust region algorithm on some standard benchmarking. The computational results reveal that, for most test problems, the number of function and gradient calculations are reduced significantly.


Introduction
Trust region algorithm is one of the most often used methods for solving the unconstrained optimization problem Generally, a trial step is calculated by solving the trust region subproblem Where is the gradient at the current approximate solution, k is a symmetric matrix which approximates the Hessian of the current trust region radius and here  refers to the 2-norm.
Trust region algorithms are blessed with both strong theoretical convergence properties and effectiveness in practice [1][2][3][4].It has been studied by many researchers.The traditional trust region is normally an area centered at the current iterate k x as indicated in model (2).In 2003, based on the traditional trust region algorithm, Ma [5] proposed a new trust region algorithm, whose trust region radius is k k g  and whose trust region center is .
After carefully studying the idea of traditional trust region algorithms and Ma [5], Zhou et al [6] also proposed an improved trust region based on an algorithm depicted in [7].
The improved trust region is a ball centered at the point and whose radius is In this paper, .k  we present another new trust region algorithm, which is based on the idea of Zhou et al [6] and to investigate the performance of the algorithms with different trust region subproblems.
Our aim is to improve the algorithm proposed in [6,7] and to make it more effective in practical implementation.The main difference between the algorithm in [7] and our algorithm is that in the former one the trust region subproblem has the model like equation (2), while in our algorithm the trust region subproblem is reconstructed at different trust region center and radius.
The paper is organized as follows.In Section 2, we describe our algorithm for unconstrained optimization problems which combines the new trust region and line search algorithm.In Section 3, primary numerical results are presented.Finally a short discussion about conclusion and future works is given in section 4.

Traditional Line Search and Trust Region Algorithm
In this subsection, we consider the line search and trust region method for the unconstrained optimization problem where  , f x the objective function, is a real-valued continuously differentiable function.Its solution is approximated by iteratively solving the subproblem (2).
Let k be the solution of (2).The predicted reduction is defined by the reduction of the approximate model, that is, and the actual reduction is defined by the reduction of the objective function, that is, The ratio between the two reductions is defined by It is well known that the ratio plays a key role in the traditional trust region algorithm to determine whether the trial step is acceptable or not and to adjust the new trust region radius.If the trial step is not successful, we will reject it, reduce the trust region radius and resolve the subproblem (2); otherwise, we will accept the trial step and enlarge the trust region radius.
The next iterate 1 k x  is determined by the following formula: where is a small constant.
The trust region radius for the next iteration is chosen by the following formula: where are positive constants that satisfy Next we describe the traditional line search and trust region algorithm.

Algorithm 1
Step 1 given 1 and that satisfy 3 , , c c c Step 2 solve (2) inaccurately so that k d   k , and so that the model has sufficient reduction. Step Step 4; else find the minimum positive integer such that , ; Step 4 set Step 5 compute

B
In the traditional trust region, the trust region subproblem is (2).In Zhou et al [6], the trust region subproblem is as follows. where is the center, k  is the radius of the trust region.
After studying [6], we have test some other radii, such as 0.5 k  , 0.75 k  and 1.5 k  .By doing this, we want to examine the effect by choosing different trust region radius at the negative gradient direction.In the numerical results, we find that the numerical result with parameter 1.5 k  is better than that with and 0. 0.5 k  75 k  .So we select 1.5 k  as the new trust region radius.
Then we give the trust region subproblem as follows.
  where d is the trial step, 1.5 is the center, 1.5 k  is the trust region radius.If we let ~2 3 At iterations of traditional line search and trust region algorithm, a trial step k is generated by solving the trust region subproblem (2).While in each iteration of our line search and trust region algorithm, the trial step k is generated by solving the trust region subproblem (5) or (6).As in [10], we solve (6) inaccurately subject to and where is a constant.
We name our line search and trust region algorithm as "L-NTR1" for convenience.
For completeness, we describe our improved line search and trust region algorithm.It is analogous with Algorithm 1.

Algorithm 2
Step 1 given 1 and that satisfy 3 , , c c c So that ( 7) and ( 8) are satisfied.
Step 3 compute   Step 4; else find the minimum positive integer such that go to Step 5.

Numerical Results
In this section, we implement our new line search and trust region algorithm and compare it both with the traditional line search and trust region algorithm (L-TTR) and the improved one (L-NTR) which is proposed in our earlier work [6].
The test problems are those given by Moré, Garbow and Hillstrom [8], and we use the same numbering system as that in [8].In all the tests, the trial step is calculated approximately by the algorithm given by Nocedal & Yuan in [9].The algorithm is terminated when the norm of the gradient k g is less than , or when the 8 10    number of calculations exceeds 30000.Next we give the results of compared different trust region radii in Table 1.
Where "P" represents the problem, "n" represents the dimension of the problem."nf" and "ng" represent the numbers of function calculations and gradient calculations, respectively.
From the Table 1, we see that the algorithm with trust region radius of 1.5 k   performs better than the other two settings.For most of problems, it needs less computation numbers both on function value and on gradient.Then, we compare our line search and trust region algorithm (L-NTR1) both with L-TTR and L-NTR.The detailed results are listed in Table 3.Furthermore, for depicting the performance of different algorithms, we say that an algorithm wins only if the number of function calculations required to solve a test problem is smaller than or equal to 95% of the one required by another algorithm.For example, the number of L-TTR for calculating the objective function is 26, and the one of L-NTR1 is 27, but we say the two algorithms are balance for the problem 15.The comparison of the experimental results is given in Table 2.  Clearly, our proposed new algorithm L-NTR1 is better than L-TTR and L-NTR.The numbers of wins for the two algorithms L-NTR1 and L-NTR are 10 and 6, respectively.The numbers of wins for the two algorithms L-NTR1 and L-TTR are 10 and 5, respectively.That is to say, in the 17 problems, our algorithm L-NTR1 wins 10 times.So our new line search and trust region method (L-NTR1) performs much better than L-TTR and L-NTR.(Table 3)

Conclusions and Future Works
In this paper, we investigate the performance of a new algorithm where the trust region subproblem is constructed by introducing negative gradient.By introducing the improvement of the trust region subproblem, the line search and trust region algorithm is improved.The numerical test results indicate that the number of function calculations is reduced significantly for most test problems.In general, we think the new line search and trust region algorithm may be more effective in practice.In the near future, we will still devote ourselves to studying the effect with different choices of the trust region radius to investigate when the trust region algorithm performs better.