Skill Assessment of North American Multi-Models Ensemble (NMME) for June-September (JJAS) Seasonal Rainfall over Ethiopia

In recent years, there has been increasing demand for high-resolution seasonal climate forecasts at sufficient lead times to allow response planning from users in agriculture, hydrology, disaster risk management, and health, among others. This paper examines the forecasting skill of the North American Mul-ti-model Ensemble (NMME) over Ethiopia during the June to September (JJAS) season. The NMME, one of the multi-model seasonal forecasting systems, regularly generates monthly seasonal rainfall forecasts over the globe with 0.5 - 11.5 months lead time. The skill and predictability of seasonal rainfall are assessed using 28 years of hindcast data from the NMME models. The forecast skill is quantified using canonical correlation analysis (CCA) and root mean square error. The results show that the NMME models capture the JJAS seasonal rainfall over central, northern, and northeastern parts of Ethiopia while exhibiting weak or limited skill across western and southwestern Ethiopia. The performance of each model in predicting the JJAS seasonal rainfall is variable, showing greater skill in predicting dry conditions. Overall, the performance of the multi-model ensemble was not consistently better than any single ensemble member. The correlation of observed and predicted seasonal rainfall for the better performing models—GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, GFDL-CM2p5-FLOR-B01 and NASA-GMAO-062012—is 0.68, 0.58, 0.52, and 0.5, respectively. The COLA-RSMAS-CCSM4, CMC1-CanCM3 and NCEP-CFSv2 models exhibit less skill, with correlations less than 0.4. In general, the NMME offers promising skill to predict seasonal rainfall over Ethiopia during the June-September (JJAS) season, motivating further work to assess its performance at longer lead times.


Introduction
Seasonal rainfall variability throughout Ethiopia is a subject of recent research, particularly as extreme rainfall events have incurred significant negative impacts on agriculture, water, energy, and other sectors of the country [1].
In recent years, there has been increasing demand for high-resolution weather and climate forecasts at sufficient lead times to allow response planning from users in agriculture, hydrology, disaster management, energy, health, and planning, among others. Skillful seasonal forecasts at lead times of several months can help farmers anticipate the onset of seasonal rainfall to select crop variety and sowing time and to identify the risk of extreme events in the season and hence minimize impacts of such events. However, the performance of at least some seasonal rainfall forecasts has been shown to be poor due to uncertain initial conditions and model errors at global and regional levels [2] [3] [4] [5].
Here, we examine the performance of the models comprising the North American Multi-model Ensemble (NMME) in predicting the June-September (JJAS) seasonal rainfall over Ethiopia.
There are currently several operational systems that provide global seasonal rainfall forecasts, for example, the National Centers for Environmental Prediction (NCEP) Climate Forecast System Version 2 (CFSV2) [6], and the European Centre for Medium-Range Weather Forecasts (ECMWF) [7]. Recently, the NMME was introduced, and the ensemble system was found to reduce forecast uncertainty and to increase the seasonal forecast skill [8]. The preliminary validation of forecast skill in the NMME model exhibited better confidence globally [9] [10] [11] [12]. However, some significant forecast errors were detected at regional or local scales [13]. Generally, the predictability of the seasonal climate signal in the NMME is highest in the tropical latitudes and decreases towards the extra-tropical latitudes; less than 30% of global drought onsets could be detected by the NMME [14] [15]. In spite of this, the NMME has been utilized with some success in global and some regional contexts, in the outlook of precipitation [16], soil moisture [17] and hydro-meteorological variables [18]. The seasonal forecasting systems and techniques used by Ethiopia's National Meteorological Agency (NMA) have been documented in [19].
The NMA uses indices of sea surface temperatures (SSTs) over the tropical Pacific Ocean, the Southern Oscillation Index (SOI) and North Atlantic Oscillations (NAO) the Multivariate ENSO Index (MEI as described by [20] and the ENSO (El Niño-La Niña) outlook obtained from NOAA/CPC. Historical and current Niño 3.4 SSTs (the Niño 3.4 region is located in the central equatorial tropical Pacific Ocean) are used to select years with ENSO evolution similar to the forecast prediction season/year. Rainfall prediction for the current year is then based on rainfall observed in these analog years. Monthly SSTs are compared for several months in advance of the season to be predicted. Finally, the seasonal forecast is presented in terms of the probability of rainfall being below, near, and above the climatological normal [21] evaluated the skill of February-May (FMAM) and June-September (JJAS) rainy seasons during the period 1999-2011, and the results revealed that the forecasting system is biased toward the near-normal category over Ethiopia. Moreover, there are limitations relating to accuracy, forecast methodology, geographic scale, and interpretation of lead-time.
Some studies demonstrate that combining statistical and dynamical methodologies has the potential to improve the seasonal forecast skill [22].
The focus of this paper is to examine the seasonal prediction skill and predictability of North American Multi-Model Ensemble Models (NMME) In the June-September rainy season (JJAS). We explore the general circulation models of NMME, employing canonical correlation analysis (CCA) as a post-processing methodology, and assess the skill of the forecasts with respect to JJAS precipitation.

Study Area
Ethiopia is located in the Horn of Africa within 3˚N -15˚N and 33˚E -48˚E, bordered with Eritrea to the north, Djibouti to the east, Sudan to the west, Kenya to the south, and Somalia to the south and east. It covers an area of about 1.14 million square kilometers (944,000 square miles), the country's topography is characterized by high and rugged plateaus and peripheral lowlands. Elevation in the country range from 160 meters below sea level (northern exit of the Rift Valley) to over 4600 meters above sea level (of northern mountainous regions) ( Figure 1). The highest mountains are concentrated on the northern and southern plateaus of the country [23].
Ethiopia has three climatological rainy seasons, namely February-May (FMAM, known as Belg in Ethiopia), June-September (JJAS, known as Kiremt), and October-January (ONDJ, known as Bega) [24]- [31]. Figure 1 indicates that JJAS is the main rainy season, accounting for 50% -80% of annual rainfall totals and the largest seasonal rainfall for many portions of the country [19]. The seasonal rainfall is highest over northwestern, western, southwestern, and central portions of Ethiopia ( Figure 1). As Cleary document by [19], the National Meteorological Agency (NMA) has issued seasonal forecasts three times a year since 1987 by as-

Description of NMME Models
The NMME is an ensemble of ensemble predictions across several coupled models from US and Canadian institutions [8]. The NMME multi-model ensemble forecast system has been shown to exhibit better performance than any single model ensemble, motivating the NMME undertaking [31] [32] [33].
The NMME forecast variables include SST, precipitation, and T2m, with real time and archived forecast products available online (http://www.cpc.ncep.noaa.gov/products/NMME). The NMME data represents one of the most extensive archives of seasonal predictions made using active seasonal forecast models currently available [8], and are of tremendous interest and value to both the research and forecasting communities. These represent only some of the most recent studies making use of this extensive and unique dataset [8]. The hindcast and forecast data are also archived at the International Research Institute for Climate and Society (IRI) https://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/).
For this study, we used a total of seven individual NMME models (90 ensemble members). Table 1 indicates the models included in NMME (including their expanded names). The first column includes the institution where each model was produced and the name of the model. The models have various ensemble sizes, ranging from 10 members to 24. Further details about the individual models can be found in their reference papers, listed in Table 1. The hindcasts for the NMME models are available from 1982-2010 and all model outputs have 1.0˚ latitude by 1.0˚ longitude horizontal resolution and forecast leads of at least 0.5 -11.5 months. In this study, we analyzed the rainy season in Ethiopia for the JJAS season using one-month lead time, which means a forecast made from initial conditions at the beginning of one month for the next. For example, the one-month-lead forecast from May's initial conditions is the forecast for June.

Observation Datasets
The NMME models' rainfall forecasts are evaluated based on Climate Hazards Infrared Precipitation with Stations (CHIRPS) precipitation datasets. CHIRPS is

Canonical Correlation Analysis (CCA) Approaches
In this study, forecasts are derived from the various dynamical model outputs by applying post-processing based on Canonical Correlation Analysis (CCA) [39].
The predictor (i.e. dynamical model) spatial domain is usually designed to be larger than the predictand (forecast) domain, so that relevant features outside of the targeted domain can be used for better model error correction, for example, to correct for systematic errors in model spatial structure of circulation and related variables (Figure 1). In the CCA method used here, preorthogonalization, using empirical orthogonal function (EOF) analysis, is done separately on the model hindcasts (the X variable, or predictor) and on the corresponding observations (the Y variable, or predictand), and a truncated set of the principal component time series from these EOFs is used as input to the CCA. The main purpose of the EOF/PCA analyses is to reduce the potential for over-fitting, through data compaction and noise filtering [40].

Deterministic and Probabilistic Skill Assessment
This section provides both deterministic and probabilistic skill assessment of the NMME system. We utilize both deterministic and probabilistic methods of assessment as there is evidence that the two approaches together provide a more complete representation of skill [41]. The forecast skill of NMME models is evaluated by using several deterministic skill measures: two alternative forced choice (2AFC), Spearman, Pearson correlation, and ROC [42]. The Spearman rank and Pearson correlation is computed for the NMME mean precipitation forecast at one-month lead-time before the start of JJAS season. The NMME mean represents the ensemble mean of the post-processed individual NMME model forecasts. Both correlations are computed separately for the JJAS season and at one-month lead-time, for the hindcast period of 1982-2010. According to [43] reported that the sign of the correlation coefficient determines whether the correlation is positive or negative and the magnitude of the correlation coefficient determines the strength of the correlation. Moreover [43] suggests the correlation categories 0.00 -0.19: "very weak", 0.20 -0.39: "weak", 0.40 -0.59: "moderate", 0.60 -0.79: "strong" and 0.80 -1.0: "very strong". We applied root mean square error (RMSE) (also called the root mean square deviation [RMSD]) to measure the difference between modeled and observed variables.

Taylor Diagram
A Taylor diagram summarizes the statistical relationship between model and the observed/reference field [44]. The diagram is useful for evaluating the accuracy

Climatology of JJAS Season
We first analyzed the spatial seasonal rainfall distribution of rainfall climatology for Ethiopia using CHIRPS data. Rainfall over the tropical semi-arid and arid areas was characterized by high inter-annual variability [45]. Therefore, understating the seasonal rainfall performance is important for agriculture, water, energy and for other socio-economic activities. In Ethiopia, June-September is the main rainy season, and the magnitude of rainfall is higher over most parts of the country. The June-September rainy season is very important for its water resources management and agriculture production. Figure 2 shows that northwestern, western, and central parts of the country exhibit the highest rainfall during this season, and the spatial averaged daily rainfall amount varies between 0 to 10 mm/day and exceeds 10 mm over the western portions of Ethiopia. The daily mean spatial rainfall pattern obtained from ensemble NMME models hindcasts shows similarity with CHIRPS over most parts of the country, but the models slightly underestimate rainfall over northern parts of Ethiopia.

Deterministic and Probabilistic Skill of NMME Models
In this section we summarize the deterministic verifications methods utilized.
The accuracy of the predictions can be indicated by their correlation coefficient, standard deviation and root mean square error with respect to the observations, over the 1982-2010 period. The root mean square error is defined as the average, for predictions for a given target month and lead time, of the squared differences between the predictions and their corresponding observations. The skill of the NMME models, in terms of spatially averaged correlation coefficient, root mean square error, and standard deviation metrics, are presented in Figure 3. The   western and southeastern portion of Ethiopia. Figure 5 and Figure 6 shows the ROC skill scores for above and lower tercile forecasts for Jun-September seasonal rainfall. During June-September most of the models show a forecast skill better than the climatological forecast. The relative (or receiver) operating characteristic (ROC) is a representation of the skill of a forecasting system in which the hit rate and the false-alarm rate are compared [38] [46] [47]. The spatial plot of ROC above and ROC below is presented in Figure 5 and Figure 6 for each individual model and for the NMME ensemble.

Verification Metrics
In particular, the GFDL-CM2p5-A06, GFDL-CM2p5-B01, CMC2-CanCM4, and COLA-CCSM4 models exhibit ROC above scores exceeding 0.6 over northwestern,   that the event does not occur) is more desirable than a hit. With a possible range of 0% -100%, a 50% rate of correct discrimination (appearing as a ROC area of 0.5) is expected by chance and reflects 0 forecast skill [46]. The spatial ROC result shows promising skill of NMME models in predicting the JJAS rainfall over portions of Ethiopia. And the skill of the multimodel ensemble is comparable or slightly better than any single model. ROC below skill is higher than ROC above skill.

Correlation Skill of NMME Models
This section compares the cross-validated skill of NMME models at a onemonth lead-time based on Pearson and Spearman correlations. Rainfall generally has lower prediction skill than other variables, due to relatively higher spatial variability in rainfall patterns [16]. For the Pearson correlation, an absolute value of 1 indicates that a perfect linear relationship and a correlation close to 0 depicts that no linear relationship between variables.
Our result of Pearson correlation for the models of GFDL-CM2p5-FLOR-A06, CMC2-CanCM4, and GFDL-CM2p5-FLOR-B01 shows promising skill to predict and capturing the JJAS seasonal rainfall over Ethiopia (Figure 7 and Table   2). These models capture most parts of northern, northeastern, central, and south-western portions of Ethiopia with promising Pearson correlation of >0.6 ( Figure 7). However, the Pearson correlation skill indicates negative correlation −0.6 over western and southeastern and the models are very limited in capturing the June-September seasonal rainfall ( Figure 7). Moreover, the NASA-GEOSS2S, COLA-RSMAS-CCSM4, CMC1-CanCM3, and NCEP-CFSv2 models reveal more limited skill in predicting the JJAS seasonal rainfall over most parts of Ethiopia with the skill of <0.4 ( Figure 8 and Table 2). Noticeable skill differences are obtained between individual NMME models based on Pearson correlation. However, the skill of ensemble NMME models of Pearson correlation shows promising improvement compared to individual NMME models (Figure 7).

Extreme Event Diagnosis in the Case of El Niño 2015
The variability of seasonal rainfall over Ethiopia leads to the recurrence of extreme drought and flood, and the intensity of extreme events affects various socio-economic activities of the country [48]. Ethiopia is one of the largest humanitarian aid beneficiaries in the world and experiences significant climate induced drought and water related stresses on crop and livestock productivity [49]. Every year approximately three million Ethiopians are affected by crop production shortfalls adding to the 7.6 million supported every year by the Table 2. Spatially averaged Pearson correlation coefficient of individual NMME models. properties [48].

Model
Accurate and reliable prediction of the seasonal rainfall prior few months in advance can assist in anticipating extreme events such as drought. We have chosen to focus on the extreme drought year of 2015 in order to examine the skill of the NMME system in capturing the dry extreme event. Figure 9 indicates that  the NMME models predict a below normal probability category rather than above normal. The models capture more than 85% of the area that experienced below normal rainfall over northern, northeastern, central, and eastern parts of

Conclusions and Recommendations
In 3) This assessment of NMME models also shows that the models are more promising for forecasting dry conditions (including extreme drought events) relative to wet conditions over Ethiopia.
Generally, the NMME is a promising tool for seasonal forecasting over Ethiopia during the June-September season. However, both individual and ensemble NMME model skill of rainfall forecasts is limited over the western and southeastern parts of Ethiopia. A more comprehensive skill assessment is a necessary next step to understand the skill of NMME models to predict the seasonal rainfall in different seasons with different lead times over Ethiopia. The results provide promise for significant advancement in operational rainfall forecasting over Ethiopia.

Acknowledgments
This work is undertaken as part of the Columbia World Project, ACToday, Co-

Data Availability
The NMME model data used to support the findings of this study are available from International Research Institute Data Library (IRDL) at https://iridl.ldeo.columbia.edu/SOURCES/Models/NMME/).