An Independent Assessment of the Monthly PRISM Gridded Precipitation Product in Central Oklahoma

Accurate, long-term records of precipitation are required for the development of climate-informed decision support tools for agriculture. But rain gauges are too sparsely located to meet this need, and radar-derived precipitation measurements are too recent in duration. Using all readily available station records, spatiotemporally continuous estimates of precipitation were created by the PRISM Climate Group to address this problem. As with all interpolated data, the validity of the gridded PRISM product requires validation, and given the extreme spatiotemporal variability of precipitation, such validation is essential. Previous work comparing the monthly precipitation product against contributing rain gauge data revealed inconsistencies that prompted the analysis reported herein. As a basis for checking the accuracy of the PRISM product, independent precipitation data gathered at a USDA research laboratory in central Oklahoma were quality controlled, including comparison to a co-located automated rain gauge operated by the Oklahoma Mesonet. Results indicate that the independent USDA gauge data are of sufficient quality to use in the evaluation of the PRISM product. The area average of the independent USDA data over a matching size area was then used to validate colocated gridded PRISM estimates. The validation results indicate that the monthly gridded PRISM precipitation estimates are close to the independent observed data in terms of means (smaller by 3% to 4.5%) and cumulative probability distributions (within ~4%), but with variances too small by 7% to 11%. From the point of view of agricultural decision support, these results indicate that PRISM estimates might be useful for probabilistic applications, such as downscaling climate forecasts or driving weather generators, assuming appropriate corrections to the higher-order statistics were applied. However, the number of months with potentially significant differences precludes the use of PRISM estimates for any retrospective month-by-month analyses of possible interactions between climate, crop management, and productivity.


Introduction
There is a pair of well-known challenges relative to developing climatologies of precipitation: 1) the amount of water reaching the earth's surface varies significantly on all space and time scales; and 2) precipitation is very difficult to measure accurately.Simply put, the first challenge means that knowing how much rain, snow, or sleet fell during a period of time at one point may tell you nothing about how much precipitation reached the surface even a few meters away.The problem is especially acute in deserts, semi-arid plains, and other locations where convectively organized precipitation is the dominate delivery mechanism.Hydrologists and agronomists have been dealing with the spatiotemporal variability problem primarily by using the closest observations available to the location of interest, a practice which is clearly insufficient.Only recently have space-filling, frequent observational estimates of precipitation based on radar data become available, so we actually know very little about the long-term statistical characteristics of precipitation for most locations.This is the reason that any new source of long term, space-filling precipitation data attracts so much attention in the agricultural and hydrologic modeling communities.
As to the second problem of measurement accuracy, all rain gauges suffer from catchment problems, especially in high wind conditions, or when the precipitation is frozen (snow, rain, sleet) or freezing on contact.These problems can be mitigated to some degree by using wind shields around the gauges, and heating the gauges.But it has been understood for decades that all gauges underestimate precipitation during these conditions, with the size of the error depending on the details of the storm, location, and gauge.Automated measurements on short time scales (minutes to hours) are usually more accurate than daily manual readings of a glass or plastic gauge, both because frequent readings minimize overflow problems, and because automation avoids many of the "human errors" associated with taking and recording measurements.
All estimates of precipitation based on remotely sensed data (such as the radar data mentioned above) depend critically on in situ (gauge) data for validation of the algorithms used to compute estimates.The bottom line is that rain gauge data, imperfect as it is, continues to constitute "ground truth" for all methods of measuring or estimating precipitation.Bieekly, monthly, seasonal, and annual variations in precipitation have significant impacts on crops and forages, so developing climate-informed decision support for agriculture that incorporates forecasts of those variations is clearly desirable.The availability of biweekly to seasonal climate forecasts from the National Oceanic and Atmospheric Administration (NOAA) appear to present such an opportunity, but climate forecasts are statements of probability for large areas, necessitating downscaling to locations, with the potential impacts of the forecasts interpreted through the use of crop models (e.g., [1,2]).There are many techniques employed for downscaling forecasts, but they all require long-duration (e.g., several decades) location-specific precipitation climatologies.Precipitation climatologies are also used in agricultural decision support to develop key statistics for daily weather generation at locations lacking historic records, as part of efforts seeking to predict the impacts of climate variability or climate change on agriculture.Another use of precipitation climatologies in agricultural decision support is in historical analyses attempting to elucidate complex and poorly understood interactions of climate, crop management, and productivity, again with intent to produce either improved crop forecasts or long-term climate-change impact forecasts.
Rain gauge data from many sources continue to be used extensively for the three types of climate-based agricultural research listed above, including the Cooperative Summary of the Day data (COOP hereafter, [3]), despite known problems with COOP precipitation data quality (e.g., [4]) and sometimes severe problems with temporal continuity of archives at many locations.But the fundamental problem from the point of view of agri-cultural decision support is the low spatial density of COOP and other observing stations; there are simply not enough to gauges to cover all agricultural locations at any point in time.
Recently, a suite of spatial climate products created by the PRISM Climate Group at Oregon State University have become available via internet and are described online as the USDA's official climatological data [5][6][7][8].The PRISM products were created specifically to address this issue of sparse and incomplete location climatologies by algorithmically generating spatially and temporally continuous climate data covering the contiguous US.The PRISM climate data is defined on a 2.5-arcminute grid, so the values represent estimates averaged over quadrants approximately 4 km on a side.The quality control and grid-filling algorithms reported for the PRISM grid data generation appear to be a rational approach to a difficult problem, using existing station data available from many sources and accounting for most of the known terrain and coastal factors that impact climate on spatial scales of a few kilometers, including altitude.Among the PRISM spatial products, the long term (1895-present) monthly precipitation products [9] are of particular interest as a possible substitute for station data in the development of climate-informed decision support tools for agriculture.However, the representativeness or accuracy of the products for specific locations across the US have not been well established and published, either on the PRISM web site or in the refereed literature.Additionally, most users in agricultural decision support are unfamiliar with the data quality issues associated with estimation of precipitation, so may be employing the PRISM products without question, and with unknown impact on their conclusions and agronomic advice.
A preliminary analysis of PRISM monthly precipitation time series in Oklahoma versus contributing COOP station data (results not shown here) suggested that the monthly means were quite similar, but that there were occasionally problems with probability distributions, variance, and skewness of the PRISM estimates.Given that the PRISM estimates are largely based on COOP data (a very "friendly" test), the differences were of concern.Accurate precipitation means are necessary but not sufficient for the development of agricultural decision support, so a comparison against independent precipitation data is in order.However, the use of all readily available data in the creation of PRISM monthly estimates makes it nearly impossible to validate the estimates in any straightforward or rigorous sense.The ideal test would use very high quality, continuous observations covering the complete period of the PRISM estimates, values that were not used in the PRISM computation, for many locations across the US.
In particular, comparison of PRISM precipitation es-timates to an average calculated over data from multiple rain gauges within a (4 km) 2 area would be strongly preferred.Such independent, dense-network precipitation data are rare and frequently not available electronically due to a number of practical factors.Fortunately, biological science technicians at the USDA/ARS Grazinglands Research Laboratory, west of El Reno, OK, had been gathering and saving such records for several decades in support of agricultural research.These climatologies needed to be digitized, reformatted, and quality controlled, but the resulting precipitation climatology provides an opportunity to evaluate the PRISM precipitation time series at one location in central Oklahoma.

Independent Data from Fort Reno
The Grazinglands Research Laboratory is located west of El Reno, OK on more than 2700 hectacres of open, gently rolling terrain, with cultivated fields and pastures, a few tree-lined creeks and ponds, and widely scattered clusters of buildings.Standard 4" diameter rain gauges have been located on fence posts at the side of fields and pastures as part of ongoing agronomic research for more than 30 years (example installation in Figure 1).While all gauge locations are well sited (removed from buildings or masses of trees or large shrubs), the support posts are substantial with flat tops, so splashing of large drops may contribute slightly to the collected readings.The lack of wind shields means that the gauges will under-report rainfall and especially snowfall in windy conditions compared to shielded gauges, as mentioned previously.Overall, the gauges and mounting are similar to those of many observing networks.Precipitation data has been manually collected from all gauges on working day mornings (no weekend or holiday collections) and manually recorded, with collection methods very similar to those recommended for the Community Collaborative Rain, Hail, & Snow Network volunteers (aka CoCoRaHS, [10]).During winter conditions, the funnel and measuring tube are removed, catching precipitation in the 4" wide outer tube.Any frozen precipitation is melted and measured when collected, using the technician's truck interior heating vents directed at the outside of the cylinder.Rain gauge elevations range from 398 m to 426 m ASL, with higher ground generally to the south.Given the moderate variation in elevation, it appears unlikely that the local terrain has notable impact on the spatial distribution of precipitation.Existing paper and digital archives of precipitation data were gathered, transcribed, and quality controlled.Relatively complete records exist for eight of the gauges from November 1982 through December 2009, all within an area of about (4 km) 2 as shown in Figure 2, and are the basis of this analysis.

Quality Assessment of Fort Reno Precipitation Data
The daily time series from each Fort Reno gauge were examined for data quality, including the two types of reporting problems pointed out by Daly and coauthors [4]: 1) underreporting of daily amounts less than 0.05"; and 2) Daly's "5/10 bias", which is over-reporting of daily precipitation amounts easily divided by 5 or 10,  which are the amounts most easily read on the rain gauges.The daily time series were free of both of these problems.The daily time series were then summed into monthly totals, with information drawn from other sources as necessary to split weekend totals across the two months.There were only 4 months out of the 304 months from November 1982 through December 2009 that required some type of correction to deal with timing issues or a small amount of missing data.Months with fewer than 28 days of data were completely excluded from the analysis.
There were several methods used to assess the Fort Reno gauge data quality.One approach simply compared the means of the individual stations against each other (differences ranging from 0% to 2.5%) and against the mean of the other seven (differences ranging from 0.4% to 1.6%).
Another approach used an initial null hypothesis that all the station time series were samples taken from the same data set.Z-tests and P-scores were calculated for each station against the mean of the other seven, with results indicating a probability of 86% or greater that the time series were all equivalent samples.Autocorrelations for each station and spatial correlations between stations were also computed from the Fort Reno monthly time series.Autocorrelations were highly similar across stations.Spatial correlations of the monthly data between stations were high, with Pearson's coefficients (R) ranging from 0.979 to 0.997.The spatial correlations decrease very slightly with increasing distance between gauges, with the distances between gauges ranging from 0.97 km to 4.64 km.There were no consistent correlations between location and magnitude of the monthly totals, so the expectation of minimal terrain impact on precipitation at this site is supported.Given these tests, the monthly time series appear to be physically reasonable, internally consistent, and good quality despite the less than optimal gauge mountings and manual style of data collection.

Comparison to Oklahoma Mesonet Gauge Data
A separate and more significant data quality assessment of the Fort Reno precipitation data was made against precipitation data of known quality collected at an Oklahoma Mesonet site [11] installed at the Fort (see Figure 2 for the relative location).The common data period for the Fort Reno and Mesonet data is January 1994 into late 2011; for this study, we used the period January 1994 through December 2009, during which there were 174 months with monthly data at all Fort Reno gauges and the Mesonet gauge.The Mesonet daily precipitation data is collected using an unheated tipping bucket rain gauge, surrounded by a wind shield (details available at [12]), with a daily reporting period starting at 0000 UTC.Under-catchment of precipitation during windy conditions should be less than with the Fort Reno gauges, given the wind shielding.Note that the roughly 12-hour difference in collection times and the delayed measurement of frozen precipitation can result in some differences in daily and possibly monthly time series.As a first-order check on the possible impact of the differences, spatial correlations were calculated between the Mesonet data and the Fort Reno data, and compared to those between the Fort Reno gauges; the resulting R values are plotted in Figure 3.
These results are physically reasonable given the spatial variability in precipitation, with correlations decreasing slowly with increasing distance between gauges.In another test, the P values for the Mesonet and Fort Reno data over the common period of record proved to be comparable, ranging from 68% to 98% for the Fort Reno gauge data and 85% for the Mesonet data.As a result, it was judged that the differences were small enough to ignore the timing differences between the Mesonet and Fort Reno daily reports for this analysis of monthly totals.
The comparisons of the Mesonet monthly time series with the Fort Reno time series are summarized statistically in Table 1, as a scatter plot in Figure 4, and as probability of exceedance distributions in Figures 5 and  6.Each Fort Reno gauge, as well as averages over the northern three gauges, the southern 5 gauges, and all 8 gauges, are included in Table 1 and Figures 5 and 6 to address questions concerning the similarity of the PRISM estimates to single gauges or averages over gauges within a (4 km) 2 area.As shown in Table 1, the Fort Reno monthly time series statistics are very similar to the Mesonet monthly time series statistics.The Mesonet mean over the common period of record is about 2% larger than the mean across the five southern Fort Reno stations, and variance is comparable, while skewness and the time series maximum are slightly higher by 8% and 13% respectively.These results are consistent with the expectation that the Mesonet gauge suffers from less under-catch- ment than the Fort Reno gauges.Note that averaging over 3, 5, or 8 Fort Reno gauges does not significantly change the variance (differences 3%) or skewness or maxima (8%) compared to any single station.
The second comparison is a scatter plot of the monthly totals from the Fort Reno gauges against the Mesonet totals (Figure 4).The high correlation of most months is obvious in the clustering along the 1-to-1 line.The few months with significant difference are cases where the timing differences in acquiring the daily data had a notable impact on the monthly totals.There is a hint of possible under-catch by the simpler Fort Reno gauges in the largest event, but overall the agreement is very good.
The third comparison examines plots of probability of exceedance (PoE) distributions for the common months of record, the probability distribution format used for NOAA/CPC seasonal climate forecasts [13] and of primary interest for forecast downscaling techniques used by one the authors [1,14].A probability of exceedance function is a variation on a cumulative probability density function, designed to readily associate odds of occurrence with different ranges of precipitation.This format facilitates a risk-based interpretation of climate forecasts and their associated impacts.Figure 5 compares the PoE distributions for the same time series reported in Table 1 and Figure 4.Because the scale of the plot obscures the subtle differences between the overlapping PoE curves, the portion of the plot centered on 50% probability is shown in Figure 6.The similarity of the curves supports the statistics shown in Table 1 and makes it obvious that any of these individual or averaged time series is in very good agreement with the Mesonet time series from the point of view of probabilities of monthly precipitation.The spread in probability (vertical difference between the curves) near the middle of the distribution is roughly 2% to 4% between the individual Fort Reno time series, and the difference between any of the Fort Reno mean PoEs (N3-avg, S5-avg, All8-avg in Figures 5 and 6) and the Mesonet PoE is only 3% to 4%.
In summary, by comparison to the Mesonet gauge data, the Fort Reno gauge data appears to slightly underestimate monthly precipitation overall.The differences are small enough (on the order of 2%) that the area average over all 8 Fort Reno gauges is deemed appropriate for use in assessing the quality of co-located PRISM grid estimates.

Comparison of PRISM Precipitation Estimates to Fort Reno Data
Time series of estimates of monthly precipitation for quadrants overlying the area of the Fort Reno gauges (see Figure 1) were obtained using the PRISM Data Explorer [15] and confirmed to be for the desired locations by comparison to estimates downloaded from another page on the PRISM web site, the PRISM Products Matrix [16].Months common to both the PRISM estimates and the Fort Reno gauge data cover the period from November 1982 through December 2009 (277 months total).This period of comparison includes all the months in the Fort Reno vs. Mesonet evaluation with the addition of more than 6 years, excluding 1992 and 1993 for which no Fort Reno gauge data was found.
Statistics for the common months in the Fort Reno and PRISM time series were computed and are reported in Table 2.A corresponding scatter plot and PoE curves are shown in Figures 7 and 8.The four PRISM time series are all very similar, as one would expect from an algorithmically generated product.Linear fits to the PRISM estimates have slopes ranging from 0.919 to 0.933, indicating a net tendency for the PRISM estimates to be slightly smaller overall than the Fort Reno average.This is supported by the differences in the means in Table 2, but those differences are small, only 1% to 2.5%.The distribution of this difference in means by month between one of the PRISM estimates (SE) and the Fort  Reno 8-gauge average is shown in Figure 9.This presentation reveals a slight tendency for the colder and drier months to have a wetter PRISM estimate (overestimate), and the warmer and wetter months a drier estimate than the Fort Reno observations (underestimate).Z-tests and P-scores on the 277-month mean indicate probabilities ranging from 72% to 85% that the PRISM estimates could have been drawn from the same population as the Fort Reno 8-gauge average, almost as good as the comparison between Fort Reno and Mesonet data.In summary, recalling that the Fort Reno data appears to be too dry by about 2% compared to the Mesonet data, the PRISM mean may be too dry by 3% to 4.5%.The variances of the PRISM time series are smaller than the Fort Reno 8-gauge variance by 12% to 16%.Given that the Fort Reno gauge variance was larger than the Mesonet variance by almost 5% (averaged over all gauges), this suggests the PRISM variance is too small by 7% to 11%.A test of the variance similar to the Z-test and P-scores calculated for the mean gives a probability that the variances of the PRISM estimates are equivalent with the Fort Reno data of only 58% to 66%.The difference is enough to justify an adjustment before using the variance or standard deviation derived from PRISM data, depending on the application.However, outside of central Oklahoma the required adjustment may be different and is currently unknown.The skewness of the PRISM time series is a bit larger than that of the Fort Reno data, consistent with the longer tail on the PRISM PoE, but this is interpreted as another consequence of the probable underestimate of extreme events by the Fort Reno gauges.There is only one month in common for all three time series (PRISM, Fort Reno, and Mesonet) with a total greater than 10" (June 2007): the Mesonet gauge measured 12.43", the Fort Reno average measurement was 10.81", and the PRISM estimates ranged from 12.06" to 12.14".
Figure 8 shows the PoE curves for all 4 PRISM estimates versus the Fort Reno 8-gauge average, and the curves are similar over most of their range.In the drier 73% of the distribution (precipitation from 0" to about 4", left of the leftmost arrow in Figure 8), PRISM estimates are slightly wetter than the Fort Reno observations.With the exception of the wettest three values (amounts greater than 10"), the PRISM estimates above 4" are systematically a bit drier than the Fort Reno observations.For the three wettest months, the PRISM estimates are higher than Fort Reno gauge observations, but that is consistent with the differences between the Fort Reno and Mesonet maxima.Given that the differences between the PRISM and Fort Reno PoEs even at the widest divergence are only on the order of 4%, the agreement in probabilities is good.
In net, the first order statistical characteristics (means) of the PRISM time series match the Fort Reno independent time series to within a few percent, probably too dry overall in this study period by 3% to 4.5%, given the Mesonet comparison.The variance of the PRISM estimates is too small, possibly by as much as 7% to 11% total, also based on both the comparison of PRISM to Fort Reno, and Fort Reno to Mesonet data.However, the cumulative probability distributions are similar in most respects and agree to within 4% in the center third of the distribution (the portion between 67% to 33%).
The relatively large scatter between the averages calculated from the PRISM estimates and the Fort Reno 8-gauge observations (see Figure 7) prompted an investigation of the specific month-to-month differences between the time series.The SE-PRISM estimate was chosen for this comparison, since it has the closest match statistically to the Fort Reno data ("best" case).SE-PRISM estimates were subtracted from the Fort Reno 8-gauge average time series and the differences tabulated in a histogram (Figure 10).
On the right side of the histogram are the cases where the PRISM data was too small compared to the Fort Reno gauge average (underestimate); the left side shows cases where the PRISM data was too large (overestimate).Bin sizes of 0.3" were chosen as representative of the simple measurement uncertainty of the Fort Reno monthly sums, such that all differences within the two bins bracketing zero are numerically indistinguishable from the Fort Reno values (137 out of 277 cases, or 49% of the data).Most of the months (83%) have differences less than 1.2".But that leaves 17% of the months with differences large enough to be of potential significance for agricultural decision support.(The choice of 1.2" as the break point was made for convenience; most practitioners in agriculture become concerned when errors in monthly precipitation approach 1".) Examination caseby-case shows that most of the negative differences (overestimates) were months with total rainfall less than 4", while most of the positive differences (underestimates) were months with total rainfall between 4" and 10", supporting the differences between the PoE curves in Figure 8. Sorting by month of occurrence (Table 3) shows that these larger differences are occurring primarily in the warmer, wetter months.Sorting by year reveals no systematic pattern, with the number of large differences per year varying from zero (1985) to as many as five (2007).Another facet that emerges from the case-by-case comparison is a strong tendency for an overestimate to be followed by an underestimate, or vice versa, especially in the data before 2007.This leads to good agreement between the means of the time series and the cumulative probability functions but problems in the temporal fidelity of the PRISM time series.As a simple example, the month by month timing of rainfall makes a significant difference in crop productivity, so an error of this magnitude would invalidate any retropsective analysis of crop response to precipitation.
Also of interest, the last three years of the comparison (2007,2008,2009) have larger cumulative differences of one sign or the other, a factor of 2 to 3 larger than for earlier years.It is not clear why this should be the case, although the underlying PRISM algorithms are reported to be different for the years 1997 onward.

Conclusions
Compared against an independent average calculated from eight gauges in a (4 km) 2 area in central Oklahoma, the time series of monthly gridded PRISM precipitation estimates is very close in terms of means (3% to 4.5%)  and cumulative probability distributions (within ~4%) but with variances differing by 7% to 11%.These differences seem reasonable given the reported characteristics of the PRISM algorithm, and in particular the inverse distance weighting of values from surrounding stations that may be tens of kilometers distant from the grid location.The test reported here was relatively "friendly" from the viewpoint of proximity to observing stations, since the nearest COOP station (El Reno) is within 8 km of the Fort Reno network.
From the point of view of developing climate-informed decision support, these results imply that in the absence of long time series of co-located or nearby station data, PRISM estimates might be useful for downscaling large-scale climate forecasts, or for driving weather generators, assuming an appropriate adjustment to all of the statistics based on the results reported herein.However, without similar comparisons to independent data in areas outside the southern Great Plains, and for situations where the nearest PRISM-employed station data is further away, the required adjustments are currently unknown.
From the point of view of retrospective analyses of climate-crop productivity, the results are not as favorable.There are a problematic number of months (47 of 277, or 17% of the tested data) with potentially significant differences between the PRISM precipitation estimates and the Fort Reno 8-gauge average.Based on the analysis presented here, it is impossible to anticipate which months in the PRISM time series will have spuriously large or small monthly precipitation.Because the temporal distribution of precipitation matters at critical points in crop development, the use of PRISM estimates is not advised for any retrospective analysis of possible interactions between climate, crop management, and productivity.

Figure 1 .
Figure 1.Photo of the south (S) Fort Reno gauge installation.

Figure 2 .
Figure 2. A map of the Fort Reno and Mesonet gauge locations relative to overlying PRISM quadrangles.The light box around the gauges is the same size as the PRISM quadrangles, approximately 4 km per side.

Figure 3 .
Figure 3. Spatial correlations (R values) computed between time series from Fort Reno and Mesonet gauges versus the istance between stations.d

Figure 5 .
Figure 5. Probability of exceedance plots of monthly precipitation from each Fort Reno gauge, the Mesonet gauge, and averages over subsets of the Fort Reno gauges; averages are defined in Table1.

Figure 4 .
Figure 4.A scatter plot of Fort Reno gauge data versus Mesonet gauge data; locations are defined in Figure 2. A 1-to-1 line is included on the plot for reference.Slopes of linear fits of each Fort Reno gauge precipitation to the Mesonet precipitation ranged from 0.974 (NW) to 1.01 (NE), with an average of 0.984.

Figure 6 .
Figure 6.Expansion of the indicated area from Figure 5.

Figure 7 .Figure 8 .
Figure 7. Scatter plot of PRISM estimates of monthly precipitation versus the average of observations from all 8 Fort Reno gauges.Linear fits of the four PRISM estimates with the Fort Reno gauge locations are plotted, with the slopes listed in the bottom right corner.

Figure 9 .
Figure 9.Comparison of the monthly mean precipitation averaged over each month in the common record between the SE-PRISM estimates and the Fort Reno 8 gauge average.The number of months in each mean computation varied from 21 to 25.

6 Figure 10 .
Figure 10.Histogram of the differences between Fort Renoverage monthly precipitation and the SE-PRISM estimate.a