Skill Assessment of North American Multi-Models Ensemble (NMME) for June-September (JJAS) Seasonal Rainfall over Ethiopia ()
1. Introduction
Seasonal rainfall variability throughout Ethiopia is a subject of recent research, particularly as extreme rainfall events have incurred significant negative impacts on agriculture, water, energy, and other sectors of the country [1].
In recent years, there has been increasing demand for high-resolution weather and climate forecasts at sufficient lead times to allow response planning from users in agriculture, hydrology, disaster management, energy, health, and planning, among others. Skillful seasonal forecasts at lead times of several months can help farmers anticipate the onset of seasonal rainfall to select crop variety and sowing time and to identify the risk of extreme events in the season and hence minimize impacts of such events. However, the performance of at least some seasonal rainfall forecasts has been shown to be poor due to uncertain initial conditions and model errors at global and regional levels [2] [3] [4] [5]. Here, we examine the performance of the models comprising the North American Multi-model Ensemble (NMME) in predicting the June-September (JJAS) seasonal rainfall over Ethiopia.
There are currently several operational systems that provide global seasonal rainfall forecasts, for example, the National Centers for Environmental Prediction (NCEP) Climate Forecast System Version 2 (CFSV2) [6], and the European Centre for Medium-Range Weather Forecasts (ECMWF) [7]. Recently, the NMME was introduced, and the ensemble system was found to reduce forecast uncertainty and to increase the seasonal forecast skill [8]. The preliminary validation of forecast skill in the NMME model exhibited better confidence globally [9] [10] [11] [12]. However, some significant forecast errors were detected at regional or local scales [13]. Generally, the predictability of the seasonal climate signal in the NMME is highest in the tropical latitudes and decreases towards the extra-tropical latitudes; less than 30% of global drought onsets could be detected by the NMME [14] [15]. In spite of this, the NMME has been utilized with some success in global and some regional contexts, in the outlook of precipitation [16], soil moisture [17] and hydro-meteorological variables [18]. The seasonal forecasting systems and techniques used by Ethiopia’s National Meteorological Agency (NMA) have been documented in [19].
The NMA uses indices of sea surface temperatures (SSTs) over the tropical Pacific Ocean, the Southern Oscillation Index (SOI) and North Atlantic Oscillations (NAO) the Multivariate ENSO Index (MEI as described by [20] and the ENSO (El Niño-La Niña) outlook obtained from NOAA/CPC. Historical and current Niño 3.4 SSTs (the Niño 3.4 region is located in the central equatorial tropical Pacific Ocean) are used to select years with ENSO evolution similar to the forecast prediction season/year. Rainfall prediction for the current year is then based on rainfall observed in these analog years. Monthly SSTs are compared for several months in advance of the season to be predicted. Finally, the seasonal forecast is presented in terms of the probability of rainfall being below, near, and above the climatological normal [21] evaluated the skill of February-May (FMAM) and June-September (JJAS) rainy seasons during the period 1999-2011, and the results revealed that the forecasting system is biased toward the near-normal category over Ethiopia. Moreover, there are limitations relating to accuracy, forecast methodology, geographic scale, and interpretation of lead-time. Some studies demonstrate that combining statistical and dynamical methodologies has the potential to improve the seasonal forecast skill [22].
The focus of this paper is to examine the seasonal prediction skill and predictability of North American Multi-Model Ensemble Models (NMME) In the June-September rainy season (JJAS). We explore the general circulation models of NMME, employing canonical correlation analysis (CCA) as a post-processing methodology, and assess the skill of the forecasts with respect to JJAS precipitation.
2. Study area and Data
2.1. Study Area
Ethiopia is located in the Horn of Africa within 3˚N - 15˚N and 33˚E - 48˚E, bordered with Eritrea to the north, Djibouti to the east, Sudan to the west, Kenya to the south, and Somalia to the south and east. It covers an area of about 1.14 million square kilometers (944,000 square miles), the country’s topography is characterized by high and rugged plateaus and peripheral lowlands. Elevation in the country range from 160 meters below sea level (northern exit of the Rift Valley) to over 4600 meters above sea level (of northern mountainous regions) (Figure 1). The highest mountains are concentrated on the northern and southern plateaus of the country [23].
Ethiopia has three climatological rainy seasons, namely February-May (FMAM, known as Belg in Ethiopia), June-September (JJAS, known as Kiremt), and October-January (ONDJ, known as Bega) [24] - [31]. Figure 1 indicates that JJAS is the main rainy season, accounting for 50% - 80% of annual rainfall totals and the largest seasonal rainfall for many portions of the country [19]. The seasonal rainfall is highest over northwestern, western, southwestern, and central portions of Ethiopia (Figure 1). As Cleary document by [19], the National Meteorological Agency (NMA) has issued seasonal forecasts three times a year since 1987 by assessing the time-lag correlations of SSTs with seasonal rainfalls of various regions over Ethiopia. The seasonal probability forecasts of four-month seasonal rainfall totals are provided with typically one-month lead time (e.g. the forecast for June-September [JJAS] is issued in late May).
2.2. Data
2.2.1. Description of NMME Models
The NMME is an ensemble of ensemble predictions across several coupled models from US and Canadian institutions [8]. The NMME multi-model ensemble forecast system has been shown to exhibit better performance than any single
![]()
Figure 1. Digital elevation map of predictand or target area in meter (top left panel), annual total rainfall (top right panel), June-September seasonal total rainfall (bottom left panel) and JJAS % of contribution (bottom right panel).
model ensemble, motivating the NMME undertaking [31] [32] [33].
The NMME forecast variables include SST, precipitation, and T2m, with real time and archived forecast products available online (http://www.cpc.ncep.noaa.gov/products/NMME). The NMME data represents one of the most extensive archives of seasonal predictions made using active seasonal forecast models currently available [8], and are of tremendous interest and value to both the research and forecasting communities. These represent only some of the most recent studies making use of this extensive and unique dataset [8]. The hindcast and forecast data are also archived at the International Research Institute for Climate and Society (IRI) https://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/).
For this study, we used a total of seven individual NMME models (90 ensemble members). Table 1 indicates the models included in NMME (including their expanded names). The first column includes the institution where each model was produced and the name of the model. The models have various ensemble sizes, ranging from 10 members to 24. Further details about the individual models can be found in their reference papers, listed in Table 1. The hindcasts for the NMME models are available from 1982-2010 and all model outputs have 1.0˚ latitude by 1.0˚ longitude horizontal resolution and forecast leads of at least 0.5 - 11.5 months. In this study, we analyzed the rainy season in Ethiopia for the JJAS season using one-month lead time, which means a forecast made from initial conditions at the beginning of one month for the next. For example, the one-month-lead forecast from May’s initial conditions is the forecast for June.
2.2.2. Observation Datasets
The NMME models’ rainfall forecasts are evaluated based on Climate Hazards Infrared Precipitation with Stations (CHIRPS) precipitation datasets. CHIRPS is
![]()
Table 1. Selected NMME models for seasonal forecast of JJAS season.
a gridded analysis provided at 1.0˚ × 1.0˚ resolution [37] [38]. The dataset is accessed from International Research Institute Data Library (IRDL) https://iridl.ldeo.columbia.edu/SOURCES/.UCSB/.CHIRPS/).
3. Methodology
3.1. Canonical Correlation Analysis (CCA) Approaches
In this study, forecasts are derived from the various dynamical model outputs by applying post-processing based on Canonical Correlation Analysis (CCA) [39]. The predictor (i.e. dynamical model) spatial domain is usually designed to be larger than the predictand (forecast) domain, so that relevant features outside of the targeted domain can be used for better model error correction, for example, to correct for systematic errors in model spatial structure of circulation and related variables (Figure 1). In the CCA method used here, preorthogonalization, using empirical orthogonal function (EOF) analysis, is done separately on the model hindcasts (the X variable, or predictor) and on the corresponding observations (the Y variable, or predictand), and a truncated set of the principal component time series from these EOFs is used as input to the CCA. The main purpose of the EOF/PCA analyses is to reduce the potential for over-fitting, through data compaction and noise filtering [40].
3.2. Deterministic and Probabilistic Skill Assessment
This section provides both deterministic and probabilistic skill assessment of the NMME system. We utilize both deterministic and probabilistic methods of assessment as there is evidence that the two approaches together provide a more complete representation of skill [41]. The forecast skill of NMME models is evaluated by using several deterministic skill measures: two alternative forced choice (2AFC), Spearman, Pearson correlation, and ROC [42]. The Spearman rank and Pearson correlation is computed for the NMME mean precipitation forecast at one-month lead-time before the start of JJAS season. The NMME mean represents the ensemble mean of the post-processed individual NMME model forecasts. Both correlations are computed separately for the JJAS season and at one-month lead-time, for the hindcast period of 1982-2010. According to [43] reported that the sign of the correlation coefficient determines whether the correlation is positive or negative and the magnitude of the correlation coefficient determines the strength of the correlation. Moreover [43] suggests the correlation categories 0.00 - 0.19: “very weak”, 0.20 - 0.39: “weak”, 0.40 - 0.59: “moderate”, 0.60 - 0.79: “strong” and 0.80 - 1.0: “very strong”. We applied root mean square error (RMSE) (also called the root mean square deviation [RMSD]) to measure the difference between modeled and observed variables.
3.3. Taylor Diagram
A Taylor diagram summarizes the statistical relationship between model and the observed/reference field [44]. The diagram is useful for evaluating the accuracy of multiple model outputs against a reference data. Further information on the Taylor diagram is provided in the supplemental material.
4. Results and Discussion
4.1. Climatology of JJAS Season
We first analyzed the spatial seasonal rainfall distribution of rainfall climatology for Ethiopia using CHIRPS data. Rainfall over the tropical semi-arid and arid areas was characterized by high inter-annual variability [45]. Therefore, understating the seasonal rainfall performance is important for agriculture, water, energy and for other socio-economic activities. In Ethiopia, June-September is the main rainy season, and the magnitude of rainfall is higher over most parts of the country. The June-September rainy season is very important for its water resources management and agriculture production. Figure 2 shows that northwestern, western, and central parts of the country exhibit the highest rainfall during this season, and the spatial averaged daily rainfall amount varies between 0 to 10 mm/day and exceeds 10 mm over the western portions of Ethiopia. The daily mean spatial rainfall pattern obtained from ensemble NMME models hindcasts shows similarity with CHIRPS over most parts of the country, but the models slightly underestimate rainfall over northern parts of Ethiopia.
4.2. Deterministic and Probabilistic Skill of NMME Models
In this section we summarize the deterministic verifications methods utilized. The accuracy of the predictions can be indicated by their correlation coefficient, standard deviation and root mean square error with respect to the observations, over the 1982-2010 period. The root mean square error is defined as the average, for predictions for a given target month and lead time, of the squared differences between the predictions and their corresponding observations. The skill of the NMME models, in terms of spatially averaged correlation coefficient, root mean square error, and standard deviation metrics, are presented in Figure 3. The
![]()
Figure 2. Comparison of observed and ensemble model mean daily rainfall during JJAS season.
![]()
Figure 3. Taylor diagram displaying statistical comparison of correlation coefficient, root mean squared error (RMSE) and standard deviation of NMME models.
correlation skill of NMME models varies between 0.1 and 0.68 (Figure 3). The GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, GFDL-CM2p5-FLOR-B01 and NASA-GMAO-062012 models exhibit relatively high skill (0.68, 0.58, 0.52 and 0.5, respectively). The COLA-RSMAS-CCSM4, CMC1-CanCM3 and NCEP-CFSv2 models exhibit correlations of <0.3, and these models have limited skill in simulating and predicting JJAS seasonal rainfall over Ethiopia (Figure 3).
The two alternatives forced choice (2AFC) score and relative operating characteristics (ROC) score for above and below are presented in the following sections. The NMME model forecast skill for JJAS seasonal precipitation is not equally distributed over Ethiopia (Figures 4-6). The red colors, which characterize the north, northwest, northeast, central and eastern portions of Ethiopia, depict where NMME skill is higher in capturing JJAS seasonal rainfall, and blue areas indicate the opposite. The 2AFC result of individual NMME models depicts that skill difference over most parts of Ethiopia in predicting the June-September seasonal rainfall (Figure 4). The 2AFC skill of the GFDL-CM2p5-FLOR-A06, CMC2-CanCM4 and GFDL-CM2p5-FLOR-B01 models are skillful over northern, central and southwestern portions of Ethiopia and these models depict that limited skill over western and southeastern portion of Ethiopia. The models NASA-GMAO-062012, COLA-RSMAS-CCSM4, CMC1-CanCM3 and NCEP-CFSv2 show comparably less skill over most parts of Ethiopia. Generally, the 2AFC scores of the NMME results show limited skill in predicting the June-September seasonal rainfall over Ethiopia across the western region of the country. It is also noted that most models predict worse than climatology over
![]()
Figure 4. Comparison of 2AFC plot of individual with ensemble NMME models.
western and southeastern portion of Ethiopia.
4.2.1. Verification Metrics
Figure 5 and Figure 6 shows the ROC skill scores for above and lower tercile forecasts for Jun-September seasonal rainfall. During June-September most of the models show a forecast skill better than the climatological forecast. The relative (or receiver) operating characteristic (ROC) is a representation of the skill of a forecasting system in which the hit rate and the false-alarm rate are compared [38] [46] [47]. The spatial plot of ROC above and ROC below is presented in Figure 5 and Figure 6 for each individual model and for the NMME ensemble. In particular, the GFDL-CM2p5-A06, GFDL-CM2p5-B01, CMC2-CanCM4, and COLA-CCSM4 models exhibit ROC above scores exceeding 0.6 over northwestern,
![]()
Figure 5. Comparison plot of individual ROC above plots of and ensemble of NMME models.
central, northeastern, and northern portions of Ethiopia. The regions with a ROC score between 0.6 and 0.7 indicate skill in predicting the JJAS seasonal rainfall over Ethiopia. The United Kingdom Meteorological Office reported that ROC scores equal to or greater than 0.6 indicate skill that is better than guessing, whereas those between 0.5 and 0.6 are associated with predictions that are little better than guesswork.
But the skill of these models is limited over western, southwestern, and southeastern parts of the country, with ROC scores < 0.4. On the other hand the skill scores of NASA-GEOSS2S, CMC1-CanCM3 and NCEP-CFSV2 depicts < 0.5 over western, northwestern and pocket areas of northeastern parts of Ethiopia (Figure 5). When probability is low, such as 0.10, a false alarm (meaning
![]()
Figure 6. The same as Figure 5 but for ROC below plots of individual and ensemble of NMME models.
that the event does not occur) is more desirable than a hit. With a possible range of 0% - 100%, a 50% rate of correct discrimination (appearing as a ROC area of 0.5) is expected by chance and reflects 0 forecast skill [46]. The spatial ROC result shows promising skill of NMME models in predicting the JJAS rainfall over portions of Ethiopia. And the skill of the multimodel ensemble is comparable or slightly better than any single model.
The spatial plot of ROC below for NASA-GEOSS2S, GFDL-CM2p5-A06, CMC2-CanCM4, and GDL-CM2P5-B01 NMME models show scores > 0.7 over northern, central, and southwestern regions of Ethiopia during the JJAS season (Figure 6). The ROC below skills of COLA-CCSM4, CMC1-Can-CM3, and NCEP/CFSV2 models are decidedly lower over the northern region. Overall, the ROC below skill is higher than ROC above skill.
4.2.2. Correlation Skill of NMME Models
This section compares the cross-validated skill of NMME models at a one-month lead-time based on Pearson and Spearman correlations. Rainfall generally has lower prediction skill than other variables, due to relatively higher spatial variability in rainfall patterns [16]. For the Pearson correlation, an absolute value of 1 indicates that a perfect linear relationship and a correlation close to 0 depicts that no linear relationship between variables.
Our result of Pearson correlation for the models of GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, and GFDL-CM2p5-FLOR-B01 shows promising skill to predict and capturing the JJAS seasonal rainfall over Ethiopia (Figure 7 and Table 2). These models capture most parts of northern, northeastern, central, and south-western portions of Ethiopia with promising Pearson correlation of >0.6 (Figure 7). However, the Pearson correlation skill indicates negative correlation −0.6 over western and southeastern and the models are very limited in capturing the June-September seasonal rainfall (Figure 7). Moreover, the NASA-GEOSS2S, COLA-RSMAS-CCSM4, CMC1-CanCM3, and NCEP-CFSv2 models reveal more limited skill in predicting the JJAS seasonal rainfall over most parts of Ethiopia with the skill of <0.4 (Figure 8 and Table 2). Noticeable skill differences are obtained between individual NMME models based on Pearson correlation. However, the skill of ensemble NMME models of Pearson correlation shows promising improvement compared to individual NMME models (Figure 7).
4.3. Extreme Event Diagnosis in the Case of El Niño 2015
The variability of seasonal rainfall over Ethiopia leads to the recurrence of extreme drought and flood, and the intensity of extreme events affects various socio-economic activities of the country [48]. Ethiopia is one of the largest humanitarian aid beneficiaries in the world and experiences significant climate induced drought and water related stresses on crop and livestock productivity [49]. Every year approximately three million Ethiopians are affected by crop production shortfalls adding to the 7.6 million supported every year by the
![]()
Table 2. Spatially averaged Pearson correlation coefficient of individual NMME models.
![]()
Figure 7. Pearson correlation skill comparison of individual and ensemble NMME models at a 1-month lead-time.
Productive Safety Net Program a social safety net supporting some of Ethiopia’s poorest and most food-insecure families [50] because of extreme drought. Very recently the 2015 El Niño-induced drought has caused food insecurity among 10.2 million people one of the highest on the record [1]. It is thus very important to analyze extreme events [51]. Particular example in Ethiopia is droughts and flood continuously occurred over northeastern, eastern, central and northern parts of the country and recently become an annual problem for loss of lives and properties [48].
Accurate and reliable prediction of the seasonal rainfall prior few months in advance can assist in anticipating extreme events such as drought. We have chosen to focus on the extreme drought year of 2015 in order to examine the skill of the NMME system in capturing the dry extreme event. Figure 9 indicates that
![]()
Figure 8. The same as Figure 7 but for Spearman correlation skill comparison of individual and ensemble NMME models at a 1-month lead-time.
the NMME models predict a below normal probability category rather than above normal. The models capture more than 85% of the area that experienced below normal rainfall over northern, northeastern, central, and eastern parts of Ethiopia during the strong El Niño year of 2015 in the JJAS season. Due to this most parts of Ethiopia affected by drought extremes in 2015. So, NMME based forecasting is skillful to detect the drought extremes over Ethiopia. Previous studies of [52] revealed that NMME models had a higher skill to predict the seasonal drought extremes. Moreover, [15] used seasonal precipitation forecasts from the NMME and other GCMs to examine the predictability of drought onset around the globe.
![]()
Figure 9. Ensemble NMME model rainfall probability category of below and normal (top left and right) above normal (bottom left) and 2AFC deterministic skill (bottom right panel) during 2015 El Niño year.
5. Conclusions and Recommendations
In this paper, we have evaluated the skill of the North American Multi-Model Ensemble (NMME) in forecasting June-September (JJAS) seasonal rainfall over Ethiopia, based on hindcasts spanning 28 years (1982-2009). The performance of seasonal forecast rainfall was evaluated using Pearson and Spearman correlation, ROC, and root mean square error (RMSE), including both deterministic and probabilistic skill scores. Noticeable skill differences in seasonal rainfall prediction were found between individual models at a one-month lead time.
The main findings of our study are the following:
1) The GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, GFDL-CM2p5-FLOR-B01, and NASA-GMAO-062012 models have relatively high skill (correlation of >0.5 and higher) over northern, northwestern, northeastern, and few places in central parts of Ethiopia. The COLA-RSMAS-CCSM4, CMC1-CanCM3, and NCEP-CFSv2 NMME models show lower skill (correlation <0.5), and these models reveal limited skill. GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, GFDL-CM2p5-FLOR-B01, and NASA-GMAO-062012 show better skill performance in predicting the JJAS seasonal rainfall at 1 one-month lead time.
2) These models capture most of the JJAS seasonal rainfall over northern, northeastern, central, southwestern, and northwestern portions of Ethiopia with promising skill. However, the models exhibit limited skill in predicting the JJAS seasonal rainfall over western and southeastern parts of the country. Pearson and Spearman’s correlations are weak or strongly negative over western Ethiopia.
1. The performance of the multi-model ensemble of the NMME model revealed better skill over the northern portion of Ethiopia, and the ensemble skill is very limited over western parts of Ethiopia. It appears that a multimodel ensemble of only the better performing individual models would exhibit additional skill.
3) This assessment of NMME models also shows that the models are more promising for forecasting dry conditions (including extreme drought events) relative to wet conditions over Ethiopia.
Generally, the NMME is a promising tool for seasonal forecasting over Ethiopia during the June-September season. However, both individual and ensemble NMME model skill of rainfall forecasts is limited over the western and southeastern parts of Ethiopia. A more comprehensive skill assessment is a necessary next step to understand the skill of NMME models to predict the seasonal rainfall in different seasons with different lead times over Ethiopia. The results provide promise for significant advancement in operational rainfall forecasting over Ethiopia.
Acknowledgments
This work is undertaken as part of the Columbia World Project, ACToday, Columbia University in the City of New York and by the National Key R&D Program of China (2018YFC1507101). The authors express their appreciation to IRI and Nanjing University of Information Science and Technology (NUIST) for encouraging and supporting the research. The authors are grateful to the International Research Institute of Columbia University (IRI) for providing the NMME model data utilized in this study.
Data Availability
The NMME model data used to support the findings of this study are available from International Research Institute Data Library (IRDL) at https://iridl.ldeo.columbia.edu/SOURCES/Models/NMME/).