Group Variable Selection via a Combination of qL Norm and Correlation-Based Penalty

Considering the problem of feature selection in linear regression model, a new method called LqCP is proposed simultaneously to select variables and favor a grouping effect, where strongly correlated predictors tend to be in or out of the model together. LqCP is based on penalized least squares with a penalty function that combines the ( ) 0 1 q L q < < norm and correlation-based penalty that is CP norm. It can shrink some coefficients to exactly zero and additionally the CP term links strength of penalization to the correlation among the predictors. The simulation studies show the advantages of LqCP with the increase of noise variables and the case of p n > . In addition, a simulation about grouped variable selection is performed. Finally, The model is applied to two real data: US Crime Data and Gasoline Data. In terms of prediction error and estimation error, empirical studies show the efficiency of LqCP.

) norm and correlation-based penalty that is CP norm.It can shrink some coefficients to exactly zero and additionally the CP term links strength of penalization to the correlation among the predictors.The simulation studies show the advantages of LqCP with the increase of noise variables and the case of p n > .In addition, a simulation about grouped variable selection is performed.Finally, The model is applied to two real data: US Crime Data and Gasoline Data.In terms of prediction error and estimation error, empirical studies show the efficiency of LqCP.

Introduction
Here the usual linear regression mode is considered in the paper given by: where 1 n y × are the observations, 1 p β × is a vector of unknown parameters to be estimated, n p X × is an n p × matrix of p predictor vectors of n observations and  is a random error vector with ( ) which often is assumed that it is subject to normal distribution independently.Ordinary least squares (OLS) estimates are very common which can be obtained by minimizing the sum of square residuals.In general, the criteria for evaluating the quality of a model from the following two aspects.One is prediction accuracy on test data and the other is to tend to select a simple model.In other words, less variables would be selected in the condition of same prediction effects.Variable selection is necessary especially when the number of predictors is large.There are many applications using variable selection to solve problems like knowledge discovery with high-dimensional data source [1] and it could greatly enhance the prediction performance of the fitted model.Traditional model selection method is best-subset selection and its step-wise variants.However, best-subset selection is computationally prohibitive when the number of predictors is large.As analyzed by Breiman (1996) [2], subset selection is unstable; thus, the resulting model has poor prediction accuracy.To overcome the drawbacks of subset selection, many variable selection methods are appeared and the most popular recently is regulation method.
In recent years, regularization method has attracted a great attention.It is used in applications such as machine learning, denoising, inpainting, deblurring, compressed sensing, source separation and more.Generally, the square loss function of penalized least squares estimates: ( ) where 0 λ > is a penalty parameter.0 q > is considered in this paper.
. When 1 q = , it becomes Lasso procedure.The procedure of minimizing the objective function is called ridge regression when 2 q = .As a continuous shrinkage method, ridge regression achieves its better prediction performance through a bias-variance trade off.However, ridge regression can not produce parsimonious model, which means all the predictors are kept in the model.Lasso is proposed by [3] in 1996.It imposes the 1 L norm regularization to the loss function and becomes a widely popular regularization method.
Lasso can shrink some coefficients to zero to achieve the effects of variable selection.Due to this reason, lasso has gained popularity in high-dimensional data.
Although lasso is a popular method for variable selection, it still has several drawbacks.The first is the lack of oracle property.The oracle property means the probability of selecting the right set of nonzero coefficients converges to one, and the estimators of the nonzero coefficients have asymptotically normal distribution with the same means and covariances.Fan and Li (2001) [5] first pointed the parameters estimators of lasso are biased and don't have the oracle property.Then adaptive lasso proposed by Zou (2006) [6] overcomes this limitation of lasso.In adaptive lasso, weights are used for regularizing different coefficients in the 1 L norm regularization, which means different coefficients have different shrinkage factors.Secondly, lasso have very poor performance when there are highly correlated variables in high-dimensional data.What's more, lasso only selects n variables at most when the number of variables p is more than the number of sample observation size n , which is a big limitation in application areas.At last, there are grouping variables among genes for the microarray gene data.These genes mostly have a high pairwise correlation with each other while lasso can not select a group of correlated genes.Based on these limitations, Zou and Hastie [7] proposed the elastic net regularization which is a combination of 1 L norm and 2 L norm.Similarly, elastic net (ENET) lacks the oracle property even though it outperforms Lasso.Further, Zou and Zhang [8] proposed adaptive elastic net to achieve the oracle property and good estimation accuracy.In the base of that, Ghosh [9] further studied grouping effects in the adaptive elastic net by using the ordinary least squares as the initial weight in low dimension data for simplicity.However, the estimators of ordinary least squares is very bad in the case of large numbers predictors or high correlated variables.Additionally, elastic net and adaptive elastic net don't take into account the information about the correlations of variables.
In the same spirits, there existed other penalty based on methods for handling grouping effects.Penalizing least squares via combining 1 L and L ∞ named OSCAR is presented by Bondell and Reich (2008) [10].The Oscar forces some coefficients to be identically equal, encouraging correlated variables that have similar effects on the response to form clusters represented by the same coefficients.
However, the computation of Oscar estimation is slow for large p which is based on a sequential quadratic programming.Then, considering the information of correlations of variables, Tutz and Ulbricht (2009) researched the property of correlation-based penalty (CP) and pointed the grouping effect of CP term.In the article of Tutz and Ulbricht, blockwise boosting procedure is applied in the simulations, which updates at each step the coefficient of more than one variable.However, in practical implementation, the step length factor and the stopping number of iterations have to be determined.Sometimes, this may be difficult and affects the sparsity of the solution as well as the speed of convergence of the algorithm.Therefore, El Anbari and Mkhadri (2014) [11] proposed an alternative regularization procedure called L1CP by combing 1 L norm with CP term.The method performs automatic variable selection and has the ability of grouping effects.
In this paper, motivated by the sparsity and grouping effect especially the case of the pairwise correlations are very high, a new regulation procedure called LqCP is proposed in linear regression setting.It combines the ( ) norm and CP penalty.Similar to the L1CP method, LqCP also performs automatic variables selection and allows to select or to remove highly correlated variables together.Section 2 first introduces the property of elastic net and adaptive elastic net and then demonstrates LqCP and its algorithm as well as the method of choosing regularization parameters.In Section 3, simulation studies give the estimation mean squared errors for different circumstances to show the parameters estimation effects among LqCP, elastic net (ENET), adaptive elastic net (AENET) and L1CP.Section 4 is mainly about the application examples to show the prediction accuracy of models.The conclusion of this paper is given in Section 5.

The Elastic Net and Adaptive Elastic Net
Here the form of elastic net described in the above firstly is showed in the following.The naive elastic net estimator ( ) naive β is the minimizer of equation: This method is called the naive elastic net which overcomes the limitations of Lasso in the case of p n > .The penalty of combining In a similar way to Lasso, the elastic net does not enjoy the oracle property.
Combining the property of the adaptive Lasso, Zou and Zhang [8] proposed the adaptive elastic net which combines the adaptive Lasso and 2 L norm.Ad- ditionally, they established the oracle property of the adaptive elastic net in the condition of weak regularity.The form of the adaptive elastic net is as follows where  , and γ is a positive constant.Choosing the initial weight is crucial in the adaptive elastic net.Zou and Zhang [8] proposed using the elastic net as an initial weight either in low-dimensional data or high-dimensional data.In the paper of Ghosh [9], the weights is donated by ( ) ˆols β .Ghosh [9] researched the ability of variable selection and grouping effect problem, proved that the adaptive elastic net also can automatically select the group variables and also had a better performance for prediction accuracy than elastic net.But the estimator of ordinary least squares is very bad in the case of high dimension or high correlations of variables.Therefore, based on the paper of Zou [8], the elastic net estimators is regarded as the initial weight in simulations in this paper.To avoid the invalid values when the estimators ˆ0 j β = , the initial weights are defined by ( ) , where n is the sample size and 0 γ > .For adaptive methods, while as 0 γ → it becomes a biased procedure like lasso.Similarly for γ → ∞ adaptive methods are un- biased.A good choice of γ is a compromise between shrinkage and bias and can be obtained via cross-validation.As described in Ghosh, γ should not be N. Mao, W. Z. Ye chosen too big a quantity and typically first should be given a range of values.
Then for each fixed γ , the cross-validation is applied to select other tuning parameters.But the elastic net and adaptive elastic net does not take into account the correlations structure of variables.Therefore, according to the CP term proposed by Tutz and Ulbricht (2009) [12], proposed new method is as follows.

Introduction of Proposed Model
In the context of linear regression problems, the following penalty function based on residual sum squares is considered.The q L penalty on f is defined as When 0 q = , the corresponding penalty is discontinuous at the origin and consequently is not easy to compute.Thus in this paper 0 q > is designed.The least squares subject to the q L penalty with 0 1 q < < is first studied by Frank and Friedman (1993) [13] which is known as bridge regression.Fu (1998) [14] and Knight and Fu (2000) [4] studied asymptotic properties and the computation of bridge estimators.When 2 q = , the solution β never becomes zero unless ˆ0 β = and it is biased.For 1 q ≤ , the bridge estimator tends to shrink small absolute coefficients to exact zeros and hence selects important variables.As pointed out by Theorem 2 in Knight and Fu (2000) [14], when 1 q > the amount of shrinkage towards zero increases with the magnitude of the regression coefficients being estimated.It suggests that if 1 q < , estimate nonzero regression parameters at the usual rate without asymptotic bias while shrinking the estimates of zero regression parameters to 0 with positive probability.In practice, in order to avoid unacceptable large bias for large parameters, the value of q is often chosen not too large.When 0 1 q < < , the q L penalty may achieve better sparsity than 1 L penalty because larger penalty is imposed on small coefficients than 1 L penalty.
So according to better sparsity property of ( ) and the case of high correlations of variables, the model is defined by ( ) where λ are positive constants.ij ρ denotes the (empirical) corre- lation between the ith and jth predictors.Here the penalty ( ) c P β is introduced by Tutz and Ulbricht (2009) [12] and is defined assuming that As introduced in the above, q L penalty in the loss of residual square sum have a better sparsity, while the correlation-based penalty ( ) c P β will en- courage grouping effect for highly correlated variables.In fact, it's easy to see that for strong positive correlation ( 1 ij ρ ≈ ) in (7), the first term becomes the dominant having the effect that estimates for i β and j β are similar ( ˆî For strong negative correlation ( , where ) ,the optimization problem ( 6) is equivalent to Proof of ( 8): In fact, the penalty ( ) ( ) can be written as follows: where which is equivalent to the optimization problem (6).

The Algorithm Procedure
The estimators of β can be computed via the Cyclic Descent Algorithm for q l sparsity penalized linear regression problem [15].The main idea is to transform the LqCP problem into an equivalent problem on augmented data.
Indeed, the optimization problem ( 6) can be written as where W is defined by (8), is a real symmetric positive-define square matrix, assuming that 2 N. Mao, W. Z. Ye 57 The LqCP estimator is defined as Note that the sample size in the augmented problem is n p + and * X has rank p .As described in the paper of Marjanovic and Solo [15] in 2014, ( ) sparsity penalizing linear regression can make a better sparsity result and a quick rate of convergence than 1 L norm.Whats more, they proposed a new algorithm called Cyclic Descent and have proved that minimizing the problem of sparsity penalized q L in linear regression has global minimizers.In the base of that, the algorithm is used to solve the Equation ( 15) and the procedure is as follows: ( ) Here, the related values are defined that ( ) Through the whole procedure, calculating the model ( 12) has been transformed to (15).Then, the algorithm procedure is stated below: (b) Use (a) computing the map: There are two solutions to this equation and is the larger one.It can be computed from the iteration, where where k e has a 1 in the k-th position and 0's in the rest.
(d) Update the residual m r with:

Selection of Tuning Parameters
In practice, it is important to select appropriate tuning parameters in order to obtain a good prediction precision or estimation precision.There are three parameters ( ) , ,q λ λ which need to be chosen.As mentioned in the previous section, how to choose a proper q is important, which depends on the nature N. Mao, W. Z. Ye of data.If sparsity of model is the point of focus, smaller q tends to be proper.The aim of this paper is not only to research the sparsity of variables but also to study grouping effect about strongly correlated variables.Therefore, the best ( ) should be chosen by experiment data and cross-validation.
Firstly, ( ) is given a grid of values to be compared.The choice of ( ) , λ λ will be different for different q .For each fixed q , cross-validation is used to choose the best ( ) , λ λ .Typically, given a grid of values for 2 λ , 5-fold cross-validation is applied to getting the best 1 λ in terms of minimized mean squared error of models or prediction for each fixed 2 λ .In simulation studies, the mean square error are defined by where ( ) T E X X is approximated by the sample covariance matrix of X of out-of-sample data.The formula of MSE is mentioned by Tibshirani [3] can measure the estimation error of model for simulation studies.In real data, because of unknown true parameters, test error defined by ( ) can measure prediction error of model.At last, the chosen value ( ) , , q λ λ is the best to compare with other models.

Simulation Studies
In this section, simulation studies are presented the finite sample performance of LqCP.The results analysis are considered from variable selection ability, the estimation errors, grouping effect.Data is generated from the true model: y is the response variable and X is an n p × matrix with p predictor vectors and n observations. is a random error vector with ( ) 0 E =  .β is p dimension parameters and σ expresses the volatility of y .Three methods in the simulations study: the elastic net (ENET), the adaptive elastic net (10%), 60 (5%), 100 (3%).The number of noise variables are respectively 7, 27, 57, 97.That is to say, this part concerns that how the results change with the increase of noise variables.The simulation results are in Table 1 and Table 2.
Example 2. This example is about the case of p n > .Similar to the example of Wu, Shen and Geyer (2009), corresponding to high dimensional simulation scenarios with correlated groups, large p and small n , the i X are simulated from ( )   ρ ≈ , and there are 25 pure noise features.Also, the data is split as 70 observations for training data and 30 observations for testing data.
From Table 1, It is anticipated that MSE of methods increase with the increase of noise variables.In 30% and 10% levels, AENET is better than other methods.But in in 5% and 3% levels, LqCP has the minimized MSE, which indicates LqCP performs better in low sparsity levels.In addition, LqCP is much better than ENET and L1CP for all levels.Table 2 illustrates that LqCP can estimate all true zero coefficients and the number of incorrect selection for true non-zero coefficients is always 0 with the increase of noise variables.Especially relative to ENET and L1CP, both of them can not select all true zero coefficients in four circumstances.
In the case of p n > showed in Table 3, LqCP performs best and are more stable.As expressed in the example, the number of true noise variables is 180 and the result from LqCP is 179, which achieves a better selection of zero coefficients than other methods.From Table 4, LqCP performs with the minimized MSE and the minimized standard error which also has a better variable selection ability for true non-zero coefficients and zero coefficients in case of high correlations 0.96 ρ ≈ .It states better estimation effect of parameters of LqCP for grouping effect than other methods.

Real Data Sets Experiment
This part is about the performances of LqCP for two real world data sets: the US Crime and Gasoline described by 15 p = and 401 p = explanatory predictors respectively.The dimension p of US Crime data set is smaller than the sample  size ( 47 n = ), while the number of variables of Gaoline exceeds largely the sample size 69 n = .Because the true parameters in application is unknown and the concern is prediction accuracy of response variable.Test Error is mentioned in the above as the criterion comparing among models.The selection of q is also based on the minimized test error for 1 1 2 0.1, , , , 0.9  5.

US Crime Data
Clearly, Table 5 shows that LqCP selects 7 variables and has the minimized test error, which performs better than other methods followed by AENET on Crime data.Although the standard error of Test Error is not the lowest, it is very close to the standard error of AENET.The selected variables are percentage of males aged 14 -24 (M), mean years of schooling (Ed), police expenditure in 1960 (Pol), numbers of males per 1000 females (M.F), number of non-whites per 1000 people (NW), income inequality (Ineq), probability of imprisonment (Prob).
These variables can also be selected by other methods.Seeing from prediction accuracy and sparsity effect of models, LqCP is the best.

Gasoline Data
This data set "Gasoline" comes from R package "pls".It is about infrared In the circumstance of high dimension ( 401 p = ) and small sample observations size( 69 n = ), LqCP is the winner in term of test error, which also gets the least number of variables.Significantly, in this application, the correlation of variables is very high and approaches 1.The result shows that AENET, L1CP and LqCP have similar variable selection effect but ENET is the worst.Therefore, this application proved the efficiency of LqCP from the aspect of p n > and highly correlated variables.

Conclusion
In this paper, motivated by variable selection and grouped selection property in linear regression problems, a new method called LqCP is proposed, which is a regularization procedure based on the penalized least squares with a mix of ( ) norm and correlation-based penalty.Firstly, this paper discusses the current models that have grouping effect including elastic net, adaptive elastic net and L1CP.Similar to them, LqCP can also encourage grouping effect, where strongly correlation among predictors tend to be in or out of the model together.Through the simulation studies in the above, LqCP has better performances in terms of variable selection ability with large numbers of noise When 1 q = , the model changes to the combination of 1 L and ( ) c P β , which is the model called L1CP proposed by El Anbari M. and Mkhadri A. (2014)[11].

Initialize with 1 β and calculate the initial residual 1
by m and the coefficient index of an iterate by k .Set 1

(
AENET) and the L1CP are listed to be compared.Because these methods have the ability of grouped variable selection.Data is divided into two data sets: training data and testing data.Training data is used to do model fitting and cross-validation.Testing data is used to evaluate the error of models.For each estimator β , its estimation accuracy is measured by the mean squared error (MSE) in the testing data.The variables selection performance is gauged by Cand IC, where C is the number of zero coefficients that are correctly estimated by zero and IC is the number of non-zero coefficients that are incorrectly estimated by zero.In addition, the algorithm's stopping criterion is these simulation studies, q is given five different values which are results in the following tables are chosen by comparing MSE among different q .ENET and AENET are computed by "glmnet" package and L1CP is computed by "lars" package in R language .Example 1. 100 data sets are generated with sample size 100 observations from the linear regression model levels are considered as below.Smaller value 2 σ = is more proper and can make all results stable through the simulation experiment.The correlation matrix is given by ( ) sample size of training data is 70 and testing data is 30.Furthermore, considering the problem of sparsity, noise variables increase by increasing the number of zero in β to generate four different dimensional data.The dimensions of data are respectively 10 (30%), 30 − − − − − .So there are 20 grouped relevant predictors and 180 noise predictors and 100 n = .Also, 100 data sets are generated and split data 70/30 into two parts for training data and testing data.Example 3.About the grouping effect, 100 sample size described by 40 predictors is considered.The true parameters are

Table 1 .
Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors for different numbers of noise variables.

Table 2 .
Median number of C and median number of IC based on 100 data replications for different numbers of noise variables.The number of true noise variables are 7, 27, 57, 97.

Table 3 .
Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors.The dimension is 200 and the number of true noise variables is 180.

Table 4 .
Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors.The number of true noise variables is 25.
This data set is taken from R package "MASS" which contains 47 observations and 15 variables as well as one response variable.Criminologists are interested in the effect of punishment regimes on crime rates which has been studied using aggregate data on 47 states of the USA for 1960.This data set contained the N. Mao, W. Z. Ye spectrum, which contains 69 observations.Recently, infrared spectrum is based on the function of diffuse reflecting degree measured by interval 2 nm from 900 nm to 1700 nm.Gasoline data have 401 prediction variables and the correlations of variables are very high that are almost 0.99.Similarly, the data set is split 100 times into a training set of 40 observations and a test set of 29.The prediction results are reported in Table6.

Table 5 .
US Crime Data-Median test errors of four methods based on 100 random splits with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 test errors.The median number of selected variables by each method is also reported.

Table 6 .
Gasoline Data-Median test errors of four methods based on 100 random splits with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 test errors.The median number of selected variables by each method is also reported.> and grouped variable selection automatically for high correlations of variables.Additionally, two real data proved LqCP's efficiency from the aspects of p n < and p n > through comparing prediction error with other models.Moreover, the oracle property is important in statistics area.One of future works is to pay attention to prove the oracle property of LqCP and also LqCP can be expended to general regression like logistic regression, quantile regression to solve some regression problems or multi-calss classification problems especially for data with highly correlated variables.