Relative Performance Evaluation of Competing Crude Oil Prices’ Volatility Forecasting Models: A Slacks-Based Super-Efficiency DEA Model

Abstract

With the increasing number of quantitative models available to forecast the volatility of crude oil prices, the assessment of the relative performance of competing models becomes a critical task. Our survey of the literature revealed that most studies tend to use several performance criteria to evaluate the performance of competing forecasting models; however, models are compared to each other using a single criterion at a time, which often leads to different rankings for different criteria—A situation where one cannot make an informed decision as to which model performs best when taking all criteria into account. In order to overcome this methodological problem, Xu and Ouenniche [1] proposed a multidimensional framework based on an input-oriented radial super-efficiency Data Envelopment Analysis (DEA) model to rank order competing forecasting models of crude oil prices’ volatility. However, their approach suffers from a number of issues. In this paper, we overcome such issues by proposing an alternative framework.

Share and Cite:

Ouenniche, J. , Xu, B. and Tone, K. (2014) Relative Performance Evaluation of Competing Crude Oil Prices’ Volatility Forecasting Models: A Slacks-Based Super-Efficiency DEA Model. American Journal of Operations Research, 4, 235-245. doi: 10.4236/ajor.2014.44023.

ficiency analysis and output-oriented super efficiency analysis only take account of technical efficiency, whereas orientation-free super efficiency analysis takes account of an additional performance component; namely, slacks. Notice that the efficient model SMA20 maintained its best position in the rankings regardless of whether the DEA analysis is input-oriented, output-oriented or orientation-free, because it is always on the efficient frontier and has zero slacks regardless of the performance measures used.

With respect to orientation-free super efficiency analysis, a close look at Table 3 reveals that whether one measures biasedness by ME or MVolScE and measures goodness-of-fit by MAE or MAVolScE, the ranks of the best models (e.g., SMA20, SES, and AR(5)) and the worst models (e.g., RW, HM and AR(1)) remain the same; i.e., they are robust to changes in measures. On the other hand, whether one measures biasedness by ME or MVolScE and measures goodness-of-fit by MSE or MSVolScE, the ranks of the best models (e.g., SMA20, SES, and CGARCH(1, 1)) and the worst models (e.g., RW, HM, and AR(1)) remain the same. These rankings suggest that, for our data set, AR(5) tends to produce large errors and CGARCH(1, 1) tends to produce small errors, as their ranks are sensitive to whether one penalizes large errors more than small ones or not. Finally, whether one measures biasedness by ME or MVolScE and measures goodness-of-fit by MMEU (respectively, MMEO), the ranks of the best models such as SMA20 and CGARCH(1, 1) (respectively, RW, HM, and SMA20) and the worst models such as RW, HM and AR(1) (respectively, SMA60 and PARCH(1, 1)) remain the same. Notice that the rankings under MMEU and MMEO differ significantly, which suggest for example that the performance of models such as RW, HM, and CGARCH(1, 1) is very sensitive to whether one penalizes negative errors more than positive ones (i.e., decision maker prefers models that under-estimate the forecasts) or vice versa. In general

Table 1. Unidimensional rankings of competing forecasting models―ranking in descending order of performance.

Table 2. Super efficiency DEA scores-based multidimensional rankings of volatility forecasting models.

1RW; 2HM; 3SMA20; 4SMA60; 5SES; 6ARMA (1, 1); 7AR (1); 8AR (5); 9GARCH (1, 1); 10GARCH-M (1, 1); 11EGARCH (1, 1); 12TGARCH (1, 1); 13PARCH (1, 1); 14CGARCH (1, 1).

however, when under-estimated forecasts are penalized, most GARCH types of models tend to perform well ―suggesting that they often produce forecasts that are over-estimated. On the other hand, when over-estimated forecasts are penalized, averaging models such as RW, HM, SES tend to perform very well―suggesting that these models often produce forecasts that are under-estimated.

Last, but not least, given our data set and the measures under consideration, numerical results suggest that, with the exception of CGARCH, the family of GARCH models have an average performance as compared to smoothing models such as SMA20 and SES―this suggests that the data generation process has a relatively long memory, which obviously gives advantage to models such as SMA20 and SES as compared to GARCH (1, 1), GARCH-M (1, 1), EGARCH (1, 1), TGARCH (1, 1) and PARCH (1, 1), which are short memory models―similar findings on the GARCH type of models were reported by Kang et al. [12] .

Table 3. Slack-based super efficiency DEA scores-based multidimensional rankings of volatility forecasting models.       

1RW; 2HM; 3SMA20; 4SMA60; 5SES; 6ARMA (1, 1); 7AR (1); 8AR (5); 9GARCH (1, 1); 10GARCH-M (1, 1); 11EGARCH (1, 1); 12TGARCH (1, 1); 13PARCH (1, 1); 14CGARCH (1, 1).

5. Conclusion

Nowadays, forecasts play a crucial role in driving our decisions and shaping our future plans in many application areas such as economics, finance and investment, marketing, and design and operational management of supply chains, among others. Obviously, forecasting problems differ with respect to many dimensions; however, regardless of how one defines the forecasting problem, a common issue faced by both academics and professionals is related to the performance evaluation of competing forecasting models. Although most studies tend to use several performance criteria, and for each criterion, one or several metrics to measure each criterion, the assessment exercise of the relative performance of competing forecasting models is generally restricted to their ranking by measure, which usually leads to different unidimensional rankings. Xu and Ouenniche [1] proposed an input-oriented radial super efficiency DEA-based framework to evaluate the performance of competing forecasting models of crude oil prices’ volatility, which delivers a single ranking based on multiple performance criteria. However, such a framework suffers from four main issues. First, under the VRS assumption, inputoriented super-efficiency scores can be different from output-oriented super-efficiency scores, which may lead to different rankings. Second, radial super-efficiency DEA models may be infeasible for some efficient DMUs; therefore, ties would persist in the rankings. Third, radial super-efficiency DEA ignore mix efficiency, which may lead to overestimated efficiency scores. Fourth, in many applications such as ours, the choice of an orientation in DEA is rather superfluous. In this paper, we overcome these issues by proposing an orientation-free super-efficiency DEA framework; namely, a slacks-based super-efficiency DEA framework for assessing the relative performance of competing volatility forecasting models. We assessed the relative performance of fourteen forecasting models of crude oil prices’ volatility based on three criteria which are commonly used in the forecasting community; namely, the goodness-of-fit, biasedness, and correct sign criteria. We have chosen to consider several measures for each criterion to find out about the robustness of multidimensional rankings with respect to different measures. The main conclusions of this research may be summarized as follows. First, models that are on the efficient frontier and have zero slacks regardless of the performance measures used (e.g., SMA20) maintain their ranks regardless of whether the DEA analysis is input-oriented, output-oriented or orientation-free. Second, the multicriteria rankings of the best and the worst models seem to be robust to changes in most performance measures; however, SMA20 seems to be the best across the board. Third, when under-estimated forecasts are penalized, most GARCH types of models tend to perform well―suggesting that they often produce forecasts that are over-estimated. On the other hand, when over-estimated forecasts are penalized, averaging models such as RW, HM, SES tend to perform very well―suggesting that these models often produce forecasts that are under-estimated. Finally, our empirical results seem to suggest that, with the exception of CGARCH, the family of GARCH models have an average performance as compared to smoothing models such as SMA20 and SES, which suggests that the data generation process has a relatively long memory.

NOTES

*Corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Xu, B. and Ouenniche, J. (2012) A Data Envelopment Analysis-Based Framework for the Relative Performance Evaluation of Competing Crude Oil Prices’ Volatility Forecasting Models. Energy Economics, 34, 576-583.
http://dx.doi.org/10.1016/j.eneco.2011.12.005
[2] Hamilton, J.D. (2008) Oil and the Macroeconomy. In: Durlauf, S. and Blume, L., Eds., The New Palgrave Dictionary of Economics, Palgrave MacMillan.
http://dx.doi.org/10.1057/9780230226203.1215
[3] Kilian, L. (2008) The Economic Effects of Energy Price Shocks. Journal of Economic Literature, 46, 871-909.
http://dx.doi.org/10.1257/jel.46.4.871
[4] Sadorsky, P. (2005) Stochastic Volatility Forecasting and Risk Management. Applied Financial Economics, 15, 121-135. http://dx.doi.org/10.1080/0960310042000299926
[5] Sadorsky, P. (2006). Modelling and forecasting petroleum futures volatility. Energy Economics, 28, 467-488.
http://dx.doi.org/10.1016/j.eneco.2006.04.005
[6] Agnolucci, P. (2009) Volatility in Crude Oil Futures: A Comparison of the Predictive Ability of GARCH and Implied Volatility Models. Energy Economics, 31, 316-321.
http://dx.doi.org/10.1016/j.eneco.2008.11.001
[7] Marzo, M. and Zagaglia, P. (2010) Volatility Forecasting for Crude Oil Futures. Applied Economics Letter, 17, 1587-1599.
http://dx.doi.org/10.1080/13504850903084996
[8] Andersen, P. and Petersen, N.C. (1993) A Procedure for Ranking Efficient Units in Data Envelopment Analysis. Management Science, 39, 1261-1294.
http://dx.doi.org/10.1287/mnsc.39.10.1261
[9] Banker, R.D., Charnes, A. and Cooper, W.W. (1984) Models for the Estimation of Technical and Scale Inefficiencies in Data Envelopment Analysis. Management Science, 30, 1078-1092.
http://dx.doi.org/10.1287/mnsc.30.9.1078
[10] Charnes, A., Cooper, W.W. and Rhodes, E. (1978) Measuring the Efficiency of Decision Making Units. European Journal of Operational Research, 2, 429-444.
http://dx.doi.org/10.1016/0377-2217(78)90138-8
[11] Tone, K. (2001) A Slacks-Based Measure of Efficiency in Data Envelopment Analysis. European Journal of Operational Research, 130, 498-509.
http://dx.doi.org/10.1016/S0377-2217(99)00407-5
[12] Kang, S.H., Kang, S.M. and Yoon, S.M. (2009) Forecasting Volatility of Crude Oil Markets. Energy Economics, 31, 119-125.
http://dx.doi.org/10.1016/j.eneco.2008.09.006
[13] Wang, Y.D. and Wu, C.F. (2012) Forecasting Energy Market Volatility Using GARCH Models: Can Multivariate Models Beat Univariate Models? Energy Economics, 34, 2167-2181.
http://dx.doi.org/10.1016/j.eneco.2012.03.010
[14] Day, T.E. and Lewis, C.M. (1993). Forecasting Futures Market Volatility. The Journal of Derivatives, 1, 33-50.
http://dx.doi.org/10.3905/jod.1993.407876
[15] Fong, W.M. and See, K.H. (2002) A Markov Switching Model of the Conditional Volatility of Crude Oil Futures Prices. Energy Economics, 24, 71-95.
http://dx.doi.org/10.1016/S0140-9883(01)00087-1
[16] Nomikos, N.K. and Pouliasis, P.K. (2011) Forecasting Petroleum Futures Markets Volatility: The Role of Regimes and Market Conditions. Energy Economics, 33, 321-337.
http://dx.doi.org/10.1016/j.eneco.2010.11.013
[17] Ghysels, E., Harvey, A. and Renault, E. (1996) Stochastic Volatility. In: Maddala, G.S. and Rao, C.R., Eds., Handbook of Statistics 14: Statistical Methods in Finance, Elsevier Science, Amsterdam.
[18] Poon, S.H. and Granger, C.W.J. (2003) Forecasting Financial Market Volatility: A Review. Journal of Economic Literature, 41, 478-539.
http://dx.doi.org/10.1257/jel.41.2.478
[19] Charnes, A. and Cooper, W.W. (1962) Programming with Linear Fractional Functionals. Naval Research Logistics Quarterly, 15, 333-334.
[20] Johnson, A.L. and Ruggiero, J. (2012) Nonparametric Measurement of Productivity and Efficiency in Education. Annals of Operations Research, 194, 1-14.
http://dx.doi.org/10.1007/s10479-011-0880-9
[21] Korhonen, P.J. and Syrjanen, M.J. (2003) Evaluation of Cost Efficiency in Finnish Electricity Distribution. Annals of Operations Research, 121, 105-122.
http://dx.doi.org/10.1023/A:1023355202795
[22] Ozcan, Y.A., Lins, M.E., Lobo, M.S.C., da Silva, A.C.M., Fiszman, R. and Pereira, B.B. (2010) Evaluating the Performance of Brazilian University Hospitals. Annals of Operations Research, 178, 247-261.
http://dx.doi.org/10.1007/s10479-009-0528-1
[23] Seiford, L.M. (1997) A Bibliography for Data Envelopment Analysis (1978-1996). Annals of Operations Research, 73, 393-438. http://dx.doi.org/10.1023/A:1018949800069
[24] Cooper, W.W., Seiford, L.M. and Tone, K. (2007) Introduction to Data Envelopment Analysis and Its Uses: With DEA-Solver Software and References. 2nd Edition, Springer, New York.
[25] Liu, J.S., Lu, L.Y., Lu, W.M. and Lin, B.J. (2013) A Survey of DEA Applications. Omega, 41, 893-902.
http://dx.doi.org/10.1016/j.omega.2012.11.004
[26] Banker, R.D., Cooper, W.W., Seiford, L.M., Thrall, R.M. and Zhu, J. (2004) Returns to Scale in Different DEA Models. European Journal of Operational Research, 154, 345-362.
http://dx.doi.org/10.1016/S0377-2217(03)00174-7
[27] Tone, K. (2002) A Slacks-Based Measure of Super-Efficiency in Data Envelopment Analysis. European Journal of Operational Research, 143, 32-41.
http://dx.doi.org/10.1016/S0377-2217(01)00324-1
[28] Du, J., Liang, L. and Zhu, J. (2010) A Slacks-Based Measure of Super-Efficiency in Data Envelopment Analysis: A Comment. European Journal of Operational Research, 204, 694-697.
http://dx.doi.org/10.1016/j.ejor.2009.12.007
[29] Andersen, T.G. and Bollerslev, T. (1998) Answering the Skeptics: Yes, Standard Volatility Models Do Provide Accurate Forecasts. International Economic Review, 39, 885-905.
http://dx.doi.org/10.2307/2527343

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.