A Nonmonotone Line Search Method for Regression Analysis *

In this paper, we propose a nonmonotone line search combining with the search direction (G. L. Yuan and Z. X.Wei, New Line Search Methods for Unconstrained Optimization, Journal of the Korean Statistical Society, 38(2009), pp. 29-39.) for regression problems. The global convergence of the given method will be established under suitable conditions. Numerical results show that the presented algorithm is more competitive than the normal methods.


Introduction
It is well known that the regression analysis often arises in economies, finance, trade, law, meteorology, medicine, biology, chemistry, engineering, physics, education, history, sociology, psychology, and so on [1,2,3,4,5,6,7].The classical regression model is defined by Y=h(X 1 , X 2 , …, X p )+ ε where Y is the response variable, Xi is predictor variable, i=1,2, …, p, p＞0 is an integer constant, and ε is the error.The function h(X 1 , X 2 , …, X p ) describes the relation between Y and X=(X 1 , X 2 , …, X p ).If h is linear function, then we can get the following linear regression model Y= 0 which is the most simple regression model, where 0 β , 1 β , …, p β are regression parameters.On the other hand, the regression model is called nonlinear regression.We all know that there are many nonlinear regression could be linearization [8,9,10,11,12,13].Then many authors are devoted to the linear model [14,15,16,17,18,19].Now we will concentrate on the linear model to discuss the following problems.One of the most important work of the regress analysis is to estimate the parameters ) , , , ( The least squares method is an important fitting method to determined the parameters ) , , , ( which is defined by where h i is the data valuation of the ith response variable, X i1 , X i2 , …, X ip are p data valuation of the ith predictor variable, and m is the number of the data.If the dimensionp and the number m is small, then we can obtain the parameters ) , , , ( from extreme value of calculus.From the definition (2), it is not difficult to see that this problem (2) is the same as the following unconstrained optimization problem In this paper, we will concentrate on this problem (3) where f : n ℜ → ℜ is continuously differentiable (linear or nonlinear).For regression problem (3), if the dimension n is large and the function f is complex, then the method of extreme value of calculus will fail.In order to solve this problem, numerical methods are often used, such as steepest descent method, Newton method, and Guass-Newton method [5,6,7].Numerical method, i.e., the iterative method is to generates a sequence of points {x k } which will terminate or converge to a point x * in some sense.The line search method is one of the most effective numerical method, which is defined by where k α is determined by a line search is the steplength, and k d which determines different line search methods [20,21,22,23,24,25,26,27] is a descent direction of f at x k .
Due to its simplicity and its very low memory requirement, the conjugate gradient method is a powerful line search method for solving the large scale optimization problems.This method can avoid, like steepest de-Copyright © 2009 SciRes JSSM scent method, the computation and storage of some matrices associated with the Hessian of objective functions.
Conjugate gradient method has the form is a scalar which determines the different conjugate gradient method [28,29,30,31,32,33,34,35,36,37].Throughout this paper, we denote f(x k ) by f k , ) ( k x f ∇ by g k , and by g k+1 , respectively. .denotes the Euclidian norm of vectors.However, the following sufficiently des cent condition which is very important to insure the global convergence of the optimization problems 2 , 0 and some constant is difficult to be satisfied by nonlinear conjugate gradient method, and this condition may be crucial for conjugate gradient methods [38].At present, the global convergence of the PRP conjugate gradient method is still open when the weak Wolfe-Powell line search rule is used.Considering this case, Yuan and Wei [27] proposed a new direction defined by where , it is easy to see that the search direction d k is the vector sum of the gradient -g k and the former search direction d k -1 , which is similar to conjugate gradient method.Otherwise, the steepest descent method is used as restart condition.Computational features should be effective.It is easy to see that the sufficiently descent condition (6) is true without carrying out any line search technique by this way.The global convergence has been established.Moreover, numerical results of the problems [39] and two regression analysis show that the given method is more competitive than the other similar methods [27].
Normally the steplength k α is generated by the following weak Wolfe-Powell (WWP): Find a steplength where 0＜ 1 σ ＜ 2 σ ＜1.The monotone line search technique is often used to get the stepsize k α , however monotonicity may cause a series of very small steps if the contours of objective function are a family of curves with large curvature [40].More recently, the nonmonotonic line search for solving unconstrained optimization is proposed by Grippo et al. in [40,41,42] and further stud-ied by [43,44] etc. Grippo, Lamparillo, and Lucidi [40] proposed the following nonmonotone line search that they call it GLL line search.GLL line search: Select steplength k k}, H≥0 is an integer constant.Combinng this line search and the normal BFGS formula, Han and Liu [45] established the global convergence of the convex objective function.Numerical results show that this method is more competitive to the normal BFGS method with WWP line search.Yuan and Wei [46] proved the superlinear convergence of the new nonmonotone BFGS algorithm.
Motivated by the above observations, we propose a nonmonotone method on the basic of Yuan and Wei [27] and Grippo, Lamparillo, and Lucidi [40].The major contribution of this paper is an extension of the new direction in [27] to the nonmonotone line search scheme, and to concentrate on the regression analysis problems.Under suitable conditions, we establish the global convergence of the method.The numerical experiments of the proposed method on a set of problems indicate that it is interesting.
This paper is organized as follows.In the next section, the proposed algorithm is given.Under some reasonable conditions, the global convergence of the given method is established in Section 3. Numerical results and a conclusion are presented in Section 4 and in Section 5, respectively.

Algorithms
The proposed algorithm is given as follows.
Step 4: Set k: =k+1 and go to step 1. Yuan and Wei [27] also presented two algorithms; here we stated them as follows.First another line search is given [47]: find a steplength k Step 0: Choose an initial point Step Step 3: Calculate the search direction d k+1 by (7).
Step 4: Let , 1 Step 5: Set k: =k+1 and go to step 1.We will concentrate on the convergent results of NLSA in the following section.
In the following, we assume that 0 ≠ k g for all k, for otherwise a stationary point has been found.The following lemma shows that the search direction dk satisfies the sufficiently descent condition without any line search technique.
Based on Lemma 3.

Numerical Results
In this section, we report some numerical results with NLST, Algorithm 1, and Algorithm 2. All codes were written in MATLAB and run on PC with 2.60GHz CPU processor and 256MB memory and Windows XP operation system.The parameters and the rules are the same to those of [27], we state it as follows: uphill search direction may occur in the numerical experiments.In this case, the line search rule maybe fails.
In order to avoid this case, the stepsize _k will be accepted if the searching number is more than twenty five in the line search.We will stop the program if the condi- is satisfied.We also stop the program if the iteration number is more than one thousand, and the corresponding method is considered to be failed.In this experiment, the direction is defined by: g d g otherwise The parameters of the presented algorithm is chosen as: , 1 .0 , 01 .0 In this section, we will test three practical problems to show the efficiency of the proposed algorithm, where Problem 1 and 2 can be seen from [27].In Table 1 and 2, the initial points are the same to those of paper [27] and the results of Algorithm 1 and Algorithm 2 can also be seen from [27].In order to show the efficiency of these algorithms, the residuals of sum of squares is defined by where n is the number of terms in problems, and p is the number of parameters, if RMS p is smaller, then the corresponding method is better [49].Problem 1.In the following table, there is data of some kind of commodity between year demand and price: The statistical results indicate that the demand will possibly change though the price is inconvenient, and the demand will be possible invariably though the price changes.Overall, the demand will decrease with the increase of the price.Our objective is to find out the approximate function between the demand and the price, namely, we need to find the regression equation of d to the p.
Problem 2. In the following table, there is data of the age x and the average height H of a pine tree: Similar to problem 1, it is easy to see that the age x and the average height H are parabola relations.Denote the regression function by β and 2 β , where n=10.Then the corresponding unconstrained optimization problem is defined by It is well known that the above problems ( 22) and ( 24) can be solved by extreme value of calculus.Here we will solve these two problems by our methods and other two methods, respectively.Problem 3. Supervisor Performance (Chapter 3 in [49]).where Y is overall appraisal to supervisor, X 1 denotes to processes employee's complaining, X 2 refer to do not permit the privilege, X 3 is the opportunity about study, X 4 is promoted based on the work achievement, X 5 refer to too nitpick to the bad performance, and X 6 is the speed of promoting to the better work.The above data can also be found at: http://www.ilr.cornell.edu/%7Ehadi/RABE3/Data/P054.txt.
Assume that the relation between Y and Xi (i=1, 2, …, 6) is linear [49], similar to Problem 1 and 2, the corresponding unconstrained optimization problem is defined by )) ,..., , , where n = 30.The regression equation from one fitting way (see Chapter 3.8 in [49]) is given by Y ˆ=10.787+0.613X 1 −0.These numerical results of Table 4-6 indicate that proposed algorithm is more competitive than those of Algorithm 1 and 2, and the initial points do not influence the results obviously about these three methods.Moreover, the numerical results of NLSA, Algorithm 1, and Algorithm 2 are better than those of these methods from extreme value of calculus or some software.Then we can conclude that the numerical method will outperform the method of extreme value of calculus in some sense, and some software for regression analysis could be further improved in the future.Overall, the direction defined by ( 7) is notable.

Conclusions
The major contribution of this paper is an extension of the direction (7) to a nonmonotone line search technique (GLL line search).The presented method possess global convergence and the numerical results show that the given algorithm is successful for the test problems.These test numerical results further show that the direction defined by ( 7) is notable.We hope the method can be a further topic for the regression analysis.
For further research, we should study other line search methods for regression analysis.
Moreover, more numerical experiments for large practical problems about regression analysis should be done in the future.

Copyright
when the program is stopped or the solution is obtained from one way.Let The columns of the tables 4-6 have the following meaning: * β : the approximate solution from the method of extreme value of calculus or some software.: the solution as the program is terminated.β ( : the initial point.* ε : the relative error between RMS p ( * β ) and RMSp ( ) It is not difficult to see that the price p and the demand d are linear relations.Denote the regression function by p work is to get 0 β and 1 β .By least squares method, we need to solve the following problem ∑ n=10.Then the corresponding unconstrained optimization problem is defined by