Ordinal Outcome Modeling : The Application of the Adaptive Moment Estimation Optimizer to the Elastic Net Penalized Stereotype Logit

Penalized ordinal outcome models were developed to model high dimensional data with ordinal outcomes. One option is the penalized stereotype logit, which includes nonlinear combinations of parameter estimates. Optimization algorithms assuming linearity and function convexity were applied to fit this model. In this study the application of the adaptive moment estimation (Adam) optimizer, suited for nonlinear optimization, to the elastic net penalized stereotype logit model is proposed. The proposed model is compared to the L1 penalized ordinalgmifs stereotype model. Both methods were applied to simulated and real data, with non-Hodgkin lymphoma (NHL) cancer subtypes as the outcome, with results presented and discussed.


Introduction
Many research studies seek to predict related outcomes given a set of independent variables or to quantify the relationship between them.In certain instances, the outcome of interest is ordinal.Ordinal variables are defined as having distinct ordered levels; however, the distance between the levels cannot be ascertained.An example of an ordinal variable is cancer stage.Take, for instance, testicular seminoma, a germ cell tumor in the sperm of the testes [1].This cancer can be classified according to stage.The stages are: 1) Tumor stage 1, cancer has not spread beyond the testicle.
2) Tumor stage 2, cancer has spread to the blood or lymphatic vessels.The ordering of categories is evident.The aim of statistical and machine learning models is to quantify the relationship between covariates and associated outcome so that one can predict the outcome variable and assess the relationship between the two with statistical significance.The range of ordinal outcome models includes cumulative logit, proportional odds model, adjacent-category logit [2], and stereotype logit [3].These procedures assume there are more observations than independent variables, or covariates.Another assumption is the resulting parameter estimates follow a normal distribution.
In addition, we now live in an era of high dimensional data, and massive amounts of information are being collected [4].These data are used to better understand and analyze related issues.However, this comes at a cost, and traditional methods are ill-equipped to utilize these datasets.These data may have more variables than observations.The distributions of the parameter estimates may not follow a normal distribution.We may collect genetic data, demographic data, and clinical data, resulting in an analysis data set containing thousands of variables with a few hundred observations when evaluating health conditions with associated ordinal outcomes [5]; the distribution of the parameter estimates may not follow a normal, or other known, distribution.
Penalized ordinal outcome models were developed to analyze high dimensional data with ordinal outcomes.Some of these modeling schemes are glmnetcr [6], ordinalgmifs [7], and penalized stereotype logit models [8].Apart from the stereotype logit, these models are linear in that the objective cost functions can be represented as a linear combination of parameters estimates; these models also assume the cost function is convex.For the stereotype logit, which includes nonlinear combinations of parameter estimates, optimization algorithms that assumed linearity, and function convexity, were applied [7] [9].A nonlinear and nonconvex approach to optimize the cost function of the penalized stereotype logit should be explored.
This study investigates the extension of a previously developed elastic net penalized stereotype logit [8].We add an elastic net penalty [10] to the stereotype logit model [3].To optimize the penalized function, we use the Adam optimizer [11] which is suited to nonlinear functions.This, in turn, allows us to evaluate the prediction accuracy of the model, which we were not able to do previously.
The updated modeling procedure is presented with first order derivatives, optimization procedure, and a bootstrap resampling scheme to assess variable importance.Said modeling procedure is applied to simulated and real-world datasets with reported results.The proposed method is compared with the ordinalgmifs implemented L1 penalized stereotype logit [7], an existing method for analyzing high dimensional data with an ordinal outcome.Journal of Data Analysis and Information Processing

Method
For a given observation i (there are a total of n observations), denote the outcome vector i y as ( ) , , , where 1 ij y = if for that observation, the outcome is in the j th category, and all other entries are set to 0. There are J possible outcomes.Denote the vector ( ) as the covariate vector consisting of p values.The log of the information entropy, based on a multinomial distribution, is represented as where ( ) ( ) ) The log of the odds ratio, with level J being the reference level, ij θ , is represented as This representation is known as the stereotype logit [3].For each ordered level, the effect of the independent variables is equal to an overall effect, i ′ x β , multip- lied by a value j φ , which is referred to as the intensity parameter.Primarily, as we are concerned with modeling the log of the odds ratios, ij θ , the log of the information entropy can now be written as

Elastic Net Penalized Stereotype Logit
We take the log of the information entropy, with a stereotype logit parameterization, and add an elastic net penalty [10].A penalty on the sum of the squared and absolute values of the parameters is enforced [12] [13].For a set of parameters, represented in a p length vector β , the elastic net penalty is defined as where 0 λ < < ∞ can vary and n is the sample size of the dataset.For this study, ς was set to 0.5.The goal of the elastic net is to penalize large values of the pa- rameter estimates, forcing their magnitude to decrease in proportion to their size.
During optimization, the system is forced shrink the parameter estimates' size Journal of Data Analysis and Information Processing when finding an optimal solution.
Based on the log of the information entropy, we are concerned with finding estimates for parameters, β , α , and φ such that where α denotes the vector of length J − 1 containing the intercepts for the J − 1 logits and φ denotes the vector on length J − 1 containing the intensity parameters.In addition, minimizing the negative log entropy is equivalent to maximizing the log entropy, and we will work with the negative representation.
Therefore, after imposing the elastic net penalty, we are concerned with finding parameter estimates such that: In machine learning, for any given model there is usually a hyperparameter set, a set of parameters that is not optimized over but whose choice of values affects the final solution.In machine learning, if there are multiple hyperparameters for any optimization procedure, there is still no established method to select these to optimize the function with respect to the parameters [14].For the purposes of this manuscript, the hyperparameters were set to given values (a small range of values was considered with the optimal set being selected); λ was set to 0.001.The partial first derivatives with respect to α j , β k , and ϕ j are presented.
( ) ( ) Denote the full parameter set β, α, and ϕ as ψ.The partial derivatives are vectorized (placed into one vector) and are represented by a derivative vector, denoted ( )

Adam Optimization
The implemented Adam algorithm [11] attempts to find a parameter set that will minimize model Equation (8).The approach uses non-linear programs to find optimal solutions given a hyperparameter set.The Adam optimizer combines the idea of momentum optimization [15] and RMSProp [16].Adam keeps track of an exponentially decaying average of past gradients from previous iterations (momentum optimization).Adam also tracks the exponentially decaying average of past squared gradients from previous iterations (RMSProp).The applied algorithm is listed below.
1) Initialize m and s to have all zero entries; these vectors are of length Journal of Data Analysis and Information Processing ( ) 2) Initialize ψ using He Initialization [17].
3) Compute ( ) , where i references the iteration number, or until a prespecified number of iterations are reached.
The vectors m and s contain the exponentially decaying averages of ( ) For the He initialization [17], the parameter estimate vector ψ is initialized using the random normal function of the form ( ) where ( ) are randomly generated values from a normal distribution with a mean 0 and standard deviation 1 and p is the number of covariates in the dataset.The hyperparameter set consists of ν 1 , τ 2 , η, ε, ζ, and ς.For this study, after considering a small range of candidate values, ν 1 was set to 0.5, τ 2 was set to 0.8, η was set to 0.008, ε was set to 1E-7, ζ was set to 1E-5, λ was set to 0.001 and, ς was set to 0.5.Steps three through six are repeated until a specified number of iterations is reached (800 for this study) or until ( ) ( ) . In applying this algorithm, we need to include an adjustment for the elastic net penalty.When taking derivatives with respect to β, we adjust these functions by subtracting the derivatives of the elastic net penalty.This is not done for α or ϕ.As a result, when computing the derivatives for the β k , where k references the iteration, we subtract from that derivative term ( ) ( ) ( ) ( ) which leads to the derivatives for each β subject to the elastic net penalty.For each iteration, in addition to modifying β by subtracting a function of its derivative we also shrink the parameters by a factor of ( ) ( ) ( ) + − .This method was implemented in the R programming environment [18].Functions from the MASS [19] and matrixcalc [20] R packages were used to implement the proposed model.

Applied Bootstrap Resampling Procedure
For the proposed model, the standard errors of our parameter estimates are computed using a bootstrapping pairs design [21].Denote B as the number of resamples without replacement.For this study, B is set to 200 [21].For each bootstrap resample, we resample n tuples with replacement, which gives us the Having these B parameter estimates allows us to estimate their standard errors and construct confidence intervals.
The bootstrap-t confidence interval method is used to construct confidence Journal of Data Analysis and Information Processing intervals.The bootstrap-t confidence intervals are of the form where ( ) ( ) with ( ) where 1, 2, , where The R programming environment [18] was used to implement this procedure.

Application to Simulated Data
The simulation procedure used is the same as previously presented [8] with a few noted changes.In that study, one dataset was simulated using a compound symmetric correlation structure for the covariates with ρ = 0.01.In addition, we simulated three additional dataset types.The second dataset type was simulated using a first order autoregressive, AR(1), correlation structure with ρ set to 0.1.The third dataset type has a Toeplitz correlation structure with each ρ generated randomly using a uniform distribution U (0, 0.4).The fourth dataset type has an unstructured correlation structure with each ~U (0, 0.4).For all simulated datatypes, 20 covariates (10 are significant) were generated for 1000 observations.Among the 10 significant parameter estimates, 5 were randomly set at 0.5 and 5 at −0.5 for all datasets.Each dataset was centered and scaled before the proposed model was fit.For each datatype, 100 datasets were simulated.The described bootstrap resampling technique was used to provide 95% confidence intervals; B = 200 resamples were used.In the model fitting process, the data were split into training: test data in the ratio 8:2.The model was developed using the training data.The final model was applied to the test dataset (independent data not used to build the model).The main criteria examined were the number of significant covariates that have non-zero parameter estimates, the number of non-significant Journal of Data Analysis and Information Processing coefficients that have estimates close to zero within a threshold, the accuracy of predictions when the model is applied to the test data set, and execution times.

Results
The proposed methodology, and ordinalgmifs method with the option probability, model = "Stereotype", was applied to the simulated data.The goal was to compare two implementations of penalized stereotype logit models.Table 1 presents the mean, and standard deviations, for prediction accuracy (determined by the test data) and execution times for both methods.Two-sided, two sample Welch's t tests, with significance level of 0.05, were used to compare mean accuracy and execution times for both methods.For the proposed method, the average prediction accuracy for the compound symmetric simulated data is 96.1%; for AR(1) correlated data, 96.52% of the observations were correctly classified.This rate was 96.24% for the Toeplitz correlated data and 96.49% for the unstructured correlated data.Regarding classification, the proposed method outperformed the ordinalgmifs method on all datasets as determined by the t tests (all p values < 0.001).Regarding execution times, the proposed method executed faster for all datasets considered (all p values < 0.001).The average execution times for the proposed method range from 17.21 to 23.63 seconds, for the ordinalgmifs method the range is 166.98 to 200.06 seconds.The proposed method executed ~10-fold faster on average.Tables 2-5 present the parameter estimates for the 10 significant parameters based on the proposed method.For all simulated datasets, the 10 non-significant parameters (not shown in the tables) had a maximum absolute value of 0.04; these values were close to 0. For the ordinalgmifs method, all non-significant parameters had estimates of 0. The proposed method selects the significant parameters that are truly related to the outcome while setting estimates of the non-significant parameters close to 0. In comparison, the ordinalgmifs methods set these values to 0. The confidence intervals are somewhat narrow (~0.5) for parameter estimates of significant covariates.
Figure 1 presents the percent of observations correctly classified per iteration for the training data of the simulated datasets.The method performance never decreases for all simulated datasets.Once the method maximizes the proportion that it can correctly estimate, it oscillates around that value.This could be due in part to the use of the Adam optimization algorithm [11].The results indicate that the proposed model framework is adept at variable selection and classification capabilities when applied to independent datasets.The proposed model outperforms the ordinalgmifs implementation with regards to prediction accuracy and execution times.Computationally, the proposed model executes faster than the ordinalgmifs implementation by ~10-fold.The analysis was performed in the R programming environment [18].names are also presented.The names of the genes are provided by the HUGO Gene Nomenclature Committee (https://www.genenames.org/)[31].As with the results in the simulation section, the 95% confidence intervals are narrow, and prediction accuracy was 73%.When applying the stereotype logit-based ordinalgmifs function to the NHL data, parameter estimates could not be obtained due to optimization issues with that implementation.The following error was reported by the R programming environment "Error in optim(c(alpha, phi), fn.stereo, w = w, x = x, beta = beta, y = y: L-BFGS-B needs finite values of 'fn'".
The above error relates to the optimization function being passed infinite values during the optimization process.All covariate values were centered and scaled prior to analysis.To correct the error, multiple values of the hyperparameters were passed to the ordinalgmifs function in R with no success.The data was checked for missing values; there were none.In addition, all the data were numeric.As a result, no comparison could be made with the ordinalgmifs implementation of the L1 penalized stereotype logit.

Discussion
There are multiple hyperparameters in the elastic net constrained stereotype logit, optimal values for these must be explored as this has the potential to increase variable selection and classification capabilities.This is usually accomplished with a grid search and is an open problem in machine learning for multiple hyperparameters [14].For this study, a small selection of values was considered for each hyperparameter.The hyperparameter of interest is λ but the choice for the remaining hyperparameters is also very important in determining the optimal solution.
A bootstrap resampling procedure was used to estimate the 95% confidence intervals.The main drawback is the computational time required to produce the confidence intervals with 200 additional models being fit.It may be advisable to perform a closed form estimate of the parameter variance matrix [2].
Although the stereotype logit is considered by many a generalized linear model, it is not.As such, an optimal solution may not exist, or there may be inflexion points.As a result, different starting values may yield different solutions.In this study, applying the method to a given dataset does not exhibit a great deal of variation in results, and the results of the applied bootstrap procedure confirm this.To address this, we applied a variable initialization scheme proposed by He [17].In addition, the Adam optimization function is well suited to dealing with non-convex functions [11].The combination of these two factors addresses this issue.

Conclusion
A proposed model for the elastic net penalized stereotype logit model, with optimization provided by the Adam optimizer, to analyze ordinal outcome data was presented.The proposed method was applied to simulated and NHL data with reported results.For the simulated data, variable selection was perfect, and only significant variables had parameter estimates not close to 0. The classifications ranged from 96.1% to 96.52% on the test datasets.For the NHL data, 73% were correctly classified.The 20 topmost genes in terms of absolute value of the coefficient of variation were presented.Our evaluation study shows that the proposed method outperforms the ordinalgmifs penalized stereotype logit model; no comparison could be made with the NHL data analysis as the ordinalgmifs implemented stereotype logit was not able to produce parameter estimates.This manuscript is an extension of previous work [8].In the previous study, variable selection was adequate, but the classification capabilities were lacking.This work improves the prediction accuracy when applied to simulated and NHL data (ranging from 73% to 96.52%).In addition, the variable importance also improved with only the significant parameters having non-zero estimates.This study demonstrates, with success, the application of the Adam optimizer to the elastic net penalized stereotype logit model to analyze ordinal outcome data with promising results, as demonstrated on the simulated and NHL datasets.

A 4 )
. A. A. Williams DOI: 10.4236/jdaip.2019.7100215 Journal of Data Analysis and Information Processing 3) Tumor stage 3, cancer has spread beyond the lymphatic and blood vessels nodes to the spermatic cord.Tumor stage 4, cancer has spread beyond previously mentioned areas to other parts of the body.
proposed model is then fit to each resampled data set.Once the B models are fit, the corresponding parameter estimates are obtained.Denote the bth bootstrap parameter estimates as ( )

Figure 1 .
Figure 1.Plots presenting the percent correctly classified per iteration for the four simulated datasets."The black line demonstrates the progress (% correctly classified) for the training data".

Table 1 .
Average accuracy (% correctly classified) and execution times for proposed and ordinalgmifs methods, along with standard deviations.
For the two methods, accuracy and executions times were compared using a two-sided Welch's two sample t test, with significance level of 0.05."*" indicates a statistically significant difference.

Table 2 .
Parameter estimates and 95% confidence intervals for truly important variables included in the final model of the compound symmetric correlated data.

Table 3 .
Parameter estimates and 95% confidence intervals for truly important variables included in the final model of the AR(1) correlated data.

Table 4 .
Parameter estimates and 95% confidence intervals for truly important variables included in the final model of the Toeplitz correlated data.

Table 5 .
Parameter estimates and 95% confidence intervals for truly important variables included in the final model of the unstructured correlated data.

Table 6 .
Variable importance based on the application of the proposed model to the NHL dataset.The topmost 20 genes, in terms of variable importance, are presented.The model achieved a prediction accuracy of 73%.