Predicting ERP User Satisfaction ― an Adaptive Neuro Fuzzy Inference System (ANFIS) Approach

ERP projects’ failing to meet user expectations is a serious problem. This research develops an Adaptive Neuro Fuzzy Inference System (ANFIS) model, to predict the key ERP outcome “User Satisfaction” using causal factors present during an implementation as predictors. Data for training and testing the models was from a cross section of firms that had implemented ERPs. ANFIS is compared with other prediction techniques, ANN and MLRA. The results establish that ANFIS is able to predict outcome well with an error (RMSE) of 0.277 and outperforms ANN and MLRA with errors of 0.85 and 0.86 respectively. This study is expected to provide guidelines to managers and academia to predict ERP outcomes ex ante, and thereby enable corrective actions to redirect ailing projects.


Introduction
The track record of successful IT projects of which Enterprise Resource Planning (ERP) is a subset projects remains poor.The latest CHAOS study of the Standish Group reports a marked decrease in IT project success rates, with only 32% succeeding in "on time" and "on budget" delivery with required features and functions.44% were delivered late or over budget, and/or with less than the required features and functions.24% were cancelled prior to completion or delivered and never used [1].This is worse than the figures of about decade back as observed by Robey in 2002: "About half of ERP projects fail to achieve anticipated benefits" [2].
Information systems (IS) project failures often encounter project "escalation" defined as a continued commitment to a failing course of action despite "uncertainty surrounding the likelihood of goal attainment" [3].Escalation research lists issues that cause escalation and suggests strategies for de-escalation which includes abandoning or "redirecting" the project [4,5].While these are acceptable as reactive steps a proactive approach of predicting impending failures, would be invaluable as one could then attempt to forestall or at least redirect the project far better.
The essence of proactive control is having predictive capabilities.The challenge is to move from the diagnosis of the source of past problems to the prediction and forecasting of potential problems in new projects [6].Can a robust, easy to use and reliable predictor be developed that would "red flag" impending failures in ERP implementations?This is the research question we seek to answer in this paper.This research has developed a method of predicting User Satisfaction, a key measure of ERP project success using ex ante causal factors as predictors.
This study consolidates and extends an earlier study which gathered data from a cross section of business organizations that had implemented ERP systems in the last three years and developed and tested a measurement model for causal factors for success [7].Data was collected, using a structured questionnaire, on Critical Success Factors (CSFs), identified in literature as being causal for the success of an ERP implementation [8] and overall User Satisfaction , a key indicator of the success [9,10].Respondents to our questionnaire represented different user cohorts: Strategic Users, Technical Users and Operational Users.The validity and reliability of the measurement model and its innate value as a predictor of ERP success was established using Structural Equation Modeling (SEM) with LISREL 8.7.
In the present study the data from the earlier study was used to develop predictive models for ERP implementation outcomes measured in terms of User Satisfaction.
Three prediction techniques, Multiple Linear Regression Analysis (MLRA), Artificial Neural Networks (ANN) and Adaptive Neuro Fuzzy Inference System (ANFIS) were tested.Of the three ANFIS was found to be significantly better in predicting User Satisfaction of an ERP project.
This paper is organized as follows: Section 2 presents the literature review and establishes the need and relevance of this research work.Section 3 outlines the method used in the research.This section also explains different prediction techniques with specific emphasis on ANFIS.Section 4 presents the results of the modeling and compares the results of the various techniques used.Section 5 concludes the paper with the direction for continuing research.

Literature Review
We looked at the several studies on the Critical Success Factors (CSFs) of an ERP implementation.CSFs can best be defined in the words of Somers and Nelson [8] as "key players and activities...playing a pivotal role in (determining) an organization's experience with the ERP implementation".By their very definition then, CSFs should be good predictors or causal factors as key determinants of outcome-"the organization's experience" or User Satisfaction.In this study, therefore, we have used CSFs and Causal Factors interchangeably.
One of the earliest researchers on CSFs, Davenport [11] identifies six factors mostly relating to the directional or strategic aspects of an ERP project.These include factors related to top management support, use of a cross functional steering committee, communication, cross functional implementation teams etc.
Parr and Shanks [12] focus on the process management aspects of an implementation.The key factors identified include management support, an organizational commitment to change and appropriate definition of scope.Hong and Kim [13] stress on the organization preparedness aspects of the implementation namely organizational fit, system adaptation levels, resistance to change, etc.
Somers and Nelson [8] work considered the most comprehensive [14] identifies a set of 22 CSFs.They used the Cooper and Zmud's [15] six stage model of IT implementation to track the importance of different factors across the six stages of: initiation, adoption, adaptation, acceptance, routinization & infusion.
CSFs have been listed extensively but not too many attempts are evident to group and measure CSFs present in an implementation.Attempts at grouping in literature are: Parr et al. who suggested into four categories [12]: 1) Management 2) Software 3) Project & 4) People.Wixom and Watson [16] suggest three groups: Organizational, Project & Technical.Holland and Light [17] two groups: The other research gap we found is the lack of studies that systematically addressed the issue of ERP project risk detection.In his doctoral desertaion Marbach [18] suggests the following nomological network to categorise IT risk literature: describe, identifiy, detect, assess & address.While there has been a fair amount of work relating to the first two, not too much work is evident in the areas of detect and assess and much less in the area of address.These areas seem to be dominated by practitioner approaches and case studies with not too many empirical studies.Finally, we could not find research that combined empirical data with a dynamic modeling approach which would address both issues of detection and assessing and provide through simulation, a means to address the third-"addressing the risk".
In addition, there have been several studies over the last two and a half decades on measuring IS success with one of the earliest studies being the work of Davis [19] who cites "Lack of user acceptance" as the main impediment of success.Other researchers who studied enterprise systems success and corrective measures include Markus and Tannis [20], Shang and Seddon [21] and Myers et al. [22].The often cited work on IS systems success of DeLone and McLean [9,10] posit six major dimensions of Information success and identify User Satisfaction as "probably the most widely used single measure of I/S success" [9].In conclusion, study of literature establishes the validity of causal factors in impacting I/S project success and confirms the use of overall user satisfaction as a good and acceptable omnibus measure for determining success or failure of an I/S implementation project.

The Method
The conceptual model underlying the present study is given in Figure 1.

Table 1. Construct and measures.
Predictors used for Modelling earlier empirical study [7], cted from a cross section of around 60 organizations and 156 respondents representing three user cohorts: Strategic, Technical & Operational responded to a pretested and validated (for content validity) structured questionnaire.Respondents rated the CSFs present in their organizations during ERP implementation.The CSFs list used for this research was drawn from prior research, and confirmed by an expert panel as relevant for the current context.The CSFs were also validated as relevant as per Structuration, Expectations-Confirmation, Lewins Change and Agency theories.Responses were captured on a Likert scale with end values of 5 = Completely Agree and 1 = Completely Disagree.From the same set of respondents their overall satisfaction, a measure of Success of the ERP project was also captured on a seven point Likert Scale with end values of 7 = Completely Satisfied and 1 = Completely Dissatisfied.
The data collected was checked and 142 respons ere found to be complete and used for further analysis.Exploratory Factor Analysis (EFA) with SPSS 7.0 using Principle Component Analysis and Varimax rotation, revealed the presence of three distinct constructs, that logically mapped to Process Factors, Strategic Factors, and Vendor (related) factors.Confirmatory Factor Analysis (CFA) using Structural Equation Modeling (SEM) using LISREL 8.7 confirmed the construct validity, and discriminant validity of the constructs in addition to confirming good overall model fit.Table 1 gives the complete set of measures and the constructs that they map to.
Scale reliability tests were conducted-Table 2. High factor loadings in excess of 0.7 and high Croanbach Alpha values in excess of 0.8 confirmed the additivity of the measures [23].This allowed for the creation of summated factors scores using the Bartlet method which was found to give better reliability when compared to the Andersen-Rubin method.These factors scores for Strategic, Process and Vendor factors were used as the predictors and User Satisfaction as the dependent variable.
The overall dataset consisted of 142 responses.This was used for the model building and testing exercise.A sample of the data set showing the independent variables as well as the dependent variable for the model is given in Table 3.

o lt iscussio
Three different prediction techniques were used: 1) Mutip g A (MLRA), eural Networks (ANN) and 3) Adaptive Neuro-Fuzzy nce System (ANFIS).In all cases about 70% of the data was used to build/train the model.The balance 30% of the data was used for testing the model.Each of these is discussed in the following paragraphs.

Multiple Linear Regression Analysis
ity"-the error ariances of the predictor variables are constant across use of ML pecially when modeling issues reted to human jugdement where multicollinearity and cognition is important, and precise computational ane principle f an adaptive learning algorithm and uses an informahts multiply (i.e., amplify or attenuat the desired outputs.One algorithm that perfo

M deling-Resu s and D ns
le Linear Re ression nalysis 2) Artificial N Infere

(MLRA)
Linear least squares regression analysis is still the most common technique used, as observed in the literature [24].Being a pure statistical technique MLRA has a few important underlying assumptions.These are 1) "linearity"-the assumption that the predictor variable is linearly related to the dependent variable, 2) no "multicolloinearity"-the individual predictors are not correlated to each other, 3) no "heteroscadac v the range of data.These conditions make the RA restrictive es la heteroscadacity are sometimes unavoidable [25].However, despite its limitations MLRA is an established technique and this study compares the results of MLRA with results obtained from other prediction techniques.

Artificial Neural Networks (ANN)
The most common model-building technique identified in the literature as an alternative to MLRA is backpropagation trained feed-forward neural networks [24] often referred to simply as back-propagation networks.ANNs are complex and flexible nonlinear systems with the ability to deal with noisy or incomplete input patterns, high fault tolerance, and the ability to generalize from the input data.[26] Neural networks excel at applications where pattern re swers are not required [24].ANN works on th o tion processing system composed of a large number of interconnected processing elements (neurons) working in tandem.
Neural networks are made of basic units arranged in layers.A unit collects information provided by other units (or by the external world) to which it is connected with weighted connections called synapses.These weights, called synaptic weig e) the input information.A positive weight is considered excitatory, a negative weight inhibitory.One of the most popular architectures in neural networks is the multi-layer perceptron which is illustrated in Figure 2.
Learning happens through a methodology of continuously altering the weights to achieve closer and closer values to rms this is known as the back propogation algorithm.The back propagation algorithm is a generalization of the Copyright © 2010 SciRes.IIM ng, inse th not always follow a strict "yes-no" rule ut could have a range of responses across a continuum.hybrid technique which combines the adaptive learning capability of ANN along with the intuitive fuzzy logic of human reasoning formulated as a feed-forward neural network.Hence, the advantages of a fuzzy system can be combined with a learning algorithm [28].Fusion of Artiorks (ANN) and Fuzzy Inference Systems (FIS) is used by researchers in various scientific e model developed.

Adaptive Neuro-Fuzzy Inference Systems (ANFIS)
While ANN is a good technique that emulates the way a human brain makes a judgement, a limitation is the way t handles the input data.In the case of human reasoni i input data need not always be crisp but could have l guistic labels like "small", "high"etc.Also, the respon e data need to b Such a pattern of responses is referred to as the membership function and such reasoning is called "fuzzy" reasoning.A fuzzy inference system using fuzzy rules can model qualitative aspects of human behavior.This was first explored by Takagi and Sugeno [27] and has since been used in numerous applications involving predictions [28].
Fuzzy inference systems are composed of five functional blocks as given in Figure 3.These are 1) a rule base containing a number of if-then rules 2) a database which defines the membership function, 3) a decision making interface that operates the given rules 4) a fuzzification interface that converts the crisp inputs into "degree of match "with the linguistic values like high or low etc., and 5) a de fuzzification interface that reconverts to a crisp output [28].
Adaptive Neuro Fuzzy Inference system (ANFIS) is a and engineering areas due to the growing need of adaptive intelligent systems to solve the real world problems.ANN learns by adjusting the weights of interconnections between layers.FIS uses fuzzy set theory, fuzzy if-then rules, and fuzzy reasoning.Given in the following section are details of th A typical ANFIS consists of five layers, which perform different actions in the ANFIS are detailed below.For simplicity, we have illustrated a system that has two inputs x and y and one output Z.The rule base, for illustrative purposes consists of two if-then rules of the Takagi-Sugeno type.
Layer 1: All the nodes in this layer are adaptive nodes.They generate membership grades of the inputs.The node function is given by: where x and y are inputs and A i and B embership functions (MF's), which can be triangular, trapezoidal, Gaussian functions or other shapes.In our study, the Gaussian MF's has been utilized and three input parameters are: Process, Strategic and Vendor.
Layer 2: The nodes in this layer are fixed nodes which multiply the Layer 3: The nodes in this layer are also fixed nodes.It calculates the ratio of a rule's firing strength to sum of the firing strengths of all the rules.This action is represented as follws: This is called normalized firing strength.
ts of this layer are given by: Layer 4: Each node in this layer is an adaptive node, whose output is simply the product of the normalized firing strength and a first-order polynomial (for a first order Sugeno model).
Thus, the outpu Parameters in this layer are referred to as consequent parameters.
Layer 5: The single node in this layer computes the overall output as the summation of all incoming signals, i.e. representation of the eight rules is given where the ov   predicted data were compared with the actual data are given in Figure 6.
The contour diagrams of the output and its predictor variables are given in Figures 7(a

Prediction with MLRA
The predictors given in Table 4 above were regressed against User Satisfaction which is a measure of ERP success.The resulting regression equation is given in Equation (6).Table 4 lists  The model is tested for assumption violations of multiple regressions.Plotting the residuals versus the preicted variable is a basic method of identifying d lations.An assumption often non-normality.A diagnostic for the same is a histogram of the residuals as well as a normality probability plot of residuals.A normal distribution makes a straight diagonal line and the residual line closely following the diagonal indicates that the distribution is normal [10].Figure 10 gives the residual plots and as can be seen the specified model does not violate the core assumptions of multiple regression.where a is actual value, p is predicted value and n t is the number of testing samples.The results show that ANFIS has a significantly better prediction capability.The RMSE of ANFIS is almost three times better than the other two methods.The MAPE results are also much better for ANFIS.This establishes that ANFIS as a modelling too is an outcomes.

1 .
Survey instrument got responses on a scale of 5-1 2.Construct confirmed through EFA followed by CFA 3. Summated scale created using Factor scores by Bartlet method

Figure 2 .
Figure 2. ANN-multilayered perceptron.least mean square algorithm.The network weights are modified to minimize the mean squared error between the desired and actual outputs of the network.The network is trained using a training data set where the input and output values are known.After the training is completed, the weights are frozen and the m r prediction of outputs for new sets of input values.
j are appropriate m inputs and send the product out.The outputs of this layer are represented as: odel can be used ficial Neural Netw fo

1 .
Prediction with ANNThe prediction was carried out using ANN.The ANN structure used in our study is given in Figure 8.The same training set of 99 used for the ANFIS model was used for the ANN modelling as well.The testing set consisted of the same 44 that were used to test the ANFIS model.The model developed is able to predict User Satisfac tion with a Root E) of 0.85 an Mean Averag of 0.195.The rediction results of the ANN model for the test data is as given in Figure 9.

6 )
the coefficients and the results of the significance tests.User Satisfaction = 4.86 + 0.812 Process + 0.496 Strategic -0.010 Vendor (Process and Strategic predictors have a very strong influence on the p faction with P < such vioencountered violation in d Mean Square Error (RMS e Percentage Error (MAPE) a p rediction of User Satis 0.05, but the Vendor predictor (with P > 0.05), does not contribute much to the prediction of User Satisfaction.

Table 5 . Comparitive results.
User Sa ftio r study that developed and tested a measurement model for assessing Linear Regression Analysis (MLRA), Artificial Neural Network (ANN) and Adaptive Neuro Fuzzy Inference System diction models using part of the dataset (99 responses) and tested using the balance (43 responses).Of the three techniques ANFIS outperformed ANN terms RMSE and MAPE.The model developed can be used by the management to asses the predicted User Satisfaction levels well in advance and thereby take appropriate corrective measures.The utility would be further enhanced if the model developed could be supplemented with a decision support system (DSS) that would help practioners simulate the outcome of the implementations dynamically by RP imData for the modeling is from a prio the causal factors for ERP implementation outcomes using Structural Equation Modeling (SEM).About 150 respondents from about 60 business organizations representing different user cohorts, Strategic, Operational and Technical, responded to a structured questionnaire.This generated the data on the causal factors and also User Satisfaction.This data was used for further analysis in this study.